You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Sourajit Basak <so...@gmail.com> on 2012/11/25 14:57:18 UTC
mime type text/plain
Some of my target webpages return a mime type of text/plain though they are
htmls. I changed "http.accept" to include text/plain and configured both
tika & parse-html to see if those can be parsed. However, both seem to
produce no content.
I changed parse-plugins.xml & the corresponding plugin.xml's to match this
mime type.
Has anyone encountered this problem ?
Re: mime type text/plain
Posted by Sebastian Nagel <wa...@googlemail.com>.
> Did you set follow redirects ?
No, I didn't try to follow the redirects.
If you follow them (nytimes is sending you around, 10 redirects or more),
finally the page gets fetched and parsed succussfully. Can you try to do this
using parsechecker? If there are still problems, can you sent the output?
...
% nutch parsechecker
"http://www-nc.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?=pagewanted=2&_r=6&ref=global-home&"
fetching:
http://www-nc.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?=pagewanted=2&_r=6&ref=global-home&
parsing:
http://www-nc.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?=pagewanted=2&_r=6&ref=global-home&
contentType: text/html
signature: 45086ebacedc40083339c8458c1065cc
---------
Url
---------------
http://www-nc.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?=pagewanted=2&_r=6&ref=global-home&
---------
ParseData
---------
Version: 5
Status: success(1,0)
Title: Chinese Hackers Infiltrate New York Times Computers - NYTimes.com
Outlinks: 142
...
Content Metadata: cache-control=no-cache expires=Thu, 01 Dec 1994 16:00:00 GMT Date=Mon, 04 Feb 2013
21:22:58 GMT Content-Length=13731 Content-Encoding=gzip Set-cookie=adxcs=s*2814d=0:1; path=/;
domain=.nytimes.com Set-Cookie=RMID=007f010044b3511026b20004; Expires=Tue, 04 Feb 2014 21:22:58 GMT;
Path=/; Domain=.nytimes.com; Content-Type=text/html Connection=close Cteonnt-Length=49721
Server=Apache pragma=no-cache
Parse Metadata: CharEncodingForConversion=utf-8 caching.forbidden=content OriginalCharEncoding=utf-8
On 02/03/2013 06:31 AM, Sourajit Basak wrote:
> Did you set follow redirects ?
>
> On Sat, Feb 2, 2013 at 8:43 PM, Sebastian Nagel
> <wa...@googlemail.com>wrote:
>
>> Hi,
>>
>> the given URL is a redirect (HTTP 303, at least, when I try) with no
>> content (only the HTTP header).
>> Tried with curl and Nutch's parsechecker tool:
>>
>> % bin/nutch parsechecker
>> "
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>> "
>> fetching: http://www.nytimes.com/...
>> ...
>> Content Metadata: Vary=Host Date=Sat, 02 Feb 2013 15:01:18 GMT
>> Content-Length=0
>> Location=
>> http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=548bb88dQ2FRtezRXVzRQ3DQ3DQ3DRfzDQ2AR(tgrHttzEREQ26Q271RQ26Q27R1Q27Rz5gfXtQ2At.VRgf_X5r5hfQ51gG5Hrh_X!_Q2AzHQ51z5hX5Q3DhVtHGhz_D5rhgtDeQ2Bz5HrQ5EfzDQ2A
>> Set-Cookie=RMID=007f0100777d510d2a3e0045; Expires=Sun, 02 Feb 2014
>> 15:01:18 GMT; Path=/;
>> Domain=.nytimes.com; Content-Type=text/plain Connection=close
>> Server=Apache
>> Parse Metadata: Content-Encoding=UTF-8 Content-Type=text/plain;
>> charset=UTF-8
>>
>> % curl -v
>> "
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>> "
>>> /dev/null
>> ...
>>> GET
>>
>> /2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>> HTTP/1.1
>> ...
>>>
>> < HTTP/1.1 303 See Other
>> < Date: Sat, 02 Feb 2013 14:59:03 GMT
>> < Server: Apache
>> < Set-Cookie: RMID=007f01000e9f510d29b70033; Expires=Sun, 02 Feb 2014
>> 14:59:03 GMT; Path=/;
>> Domain=.nytimes.com;
>> < Vary: Host
>> < Location:
>>
>> http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=f39d9b3aQ2FQ2AmQ51dQ2AKSdQ2A(((Q2AQ7Ddg_Q2ANm46JmmdUQ2AUCVMQ2ACVQ2AMVQ2AdQ274Q7DKm_mrSQ2A4Q7DtKQ276Q27!Q7DQ7E42Q27J6!tKyt_dJQ7EdQ27!KQ27(!SmJ2!dtgQ276!4mgQ51ndQ27J6GQ7Ddg_
>> < Content-Length: 0
>> < Connection: close
>> < Content-Type: text/plain
>>
>> Sebastian
>>
>>
>> On 02/01/2013 05:47 AM, Sourajit Basak wrote:
>>> Here it goes.
>>>
>>> Try to dump the content from this url with the following settings.
>>>
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>>>
>>> <property>
>>> <name>http.content.limit</name>
>>> <value>-1</value>
>>> </property>
>>>
>>> This page is gzip encoded. You will see that the fetcher is unable to
>>> download any content. Check by inspecting the content-length.
>>> Initially I was thinking it to be a problem with the parse-html plugin
>> but
>>> now it seems that the fetcher returns null content.
>>>
>>> This seemed related to NUTCH-374
>>>
>>> Let me know if you need further info.
>>>
>>> On Fri, Feb 1, 2013 at 1:54 AM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> Can you briefly describe the problem here Sourajit?
>>>>
>>>> On Thu, Jan 31, 2013 at 9:01 AM, Sourajit Basak
>>>> <so...@gmail.com> wrote:
>>>>> Seems to be related to NUTCH-374 but that shows as fixed.
>>>>>
>>>>> I have set Nutch to accept unlimited content size & this page is gzip
>>>>> encoded.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <
>>>> sourajit.basac@gmail.com>wrote:
>>>>>
>>>>>> Re-opening this thread.
>>>>>>
>>>>>> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
>>>>>> parse-html)
>>>>>>
>>>>>>
>>>>
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>>>>>>
>>>>>> I do not get any content from the fetcher. This is my fetcher accept
>>>>>> params.
>>>>>> <property>
>>>>>> <name>http.accept</name>
>>>>>>
>>>>>>
>>>>
>> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <
>>>> sourajit.basac@gmail.com
>>>>>>> wrote:
>>>>>>
>>>>>>> Ignore my last post. Tika isn't slowing down, neither is this
>> property.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
>>>>>>> sourajit.basac@gmail.com> wrote:
>>>>>>>
>>>>>>>> Enabling this property slows down the parse phase drastically when
>>>>>>>> encountered with mime-type image/jpeg.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
>>>>>>>> sourajit.basac@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Julien.
>>>>>>>>>
>>>>>>>>> I can get the outlinks now, let me check if I can get the raw
>>>> content.
>>>>>>>>> I will update this thread.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>>>>>>>>> lists.digitalpebble@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> The parameter
>>>>>>>>>>
>>>>>>>>>> <property>
>>>>>>>>>> <name>mime.type.magic</name>
>>>>>>>>>> <value>true</value>
>>>>>>>>>> <description>Defines if the mime content type detector uses
>> magic
>>>>>>>>>> resolution.
>>>>>>>>>> </description>
>>>>>>>>>> </property>
>>>>>>>>>>
>>>>>>>>>> should trigger the mime type detection based on the content and
>> not
>>>> on
>>>>>>>>>> what
>>>>>>>>>> the server returns. It is not a Tika issue as such as the
>> selection
>>>> of
>>>>>>>>>> what
>>>>>>>>>> parser to use is based on the mimetype that Nutch uses.
>>>>>>>>>>
>>>>>>>>>> The param above should be set to true by default. I thought we had
>>>> more
>>>>>>>>>> options but am probably confusing with the language identification
>>>>>>>>>>
>>>>>>>>>> Julien
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 25 November 2012 14:16, Sourajit Basak <
>> sourajit.basac@gmail.com
>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> DEBUG tika.TikaParser - Using Tika parser
>>>>>>>>>>> org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>>>>>>>>>>
>>>>>>>>>>> The above indicates Tika is fired. But somehow I need to tell
>> Tika
>>>>>>>>>> to use
>>>>>>>>>>> HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>>>>>>>>>>
>>>>>>>>>>> Is it possible to do anything in Nutch ?
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>>>>>>>>> sourajit.basac@gmail.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Some of my target webpages return a mime type of text/plain
>>>> though
>>>>>>>>>> they
>>>>>>>>>>>> are htmls. I changed "http.accept" to include text/plain and
>>>>>>>>>> configured
>>>>>>>>>>>> both tika & parse-html to see if those can be parsed. However,
>>>>>>>>>> both seem
>>>>>>>>>>> to
>>>>>>>>>>>> produce no content.
>>>>>>>>>>>>
>>>>>>>>>>>> I changed parse-plugins.xml & the corresponding plugin.xml's to
>>>>>>>>>> match
>>>>>>>>>>> this
>>>>>>>>>>>> mime type.
>>>>>>>>>>>>
>>>>>>>>>>>> Has anyone encountered this problem ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *
>>>>>>>>>> *Open Source Solutions for Text Engineering
>>>>>>>>>>
>>>>>>>>>> http://digitalpebble.blogspot.com/
>>>>>>>>>> http://www.digitalpebble.com
>>>>>>>>>> http://twitter.com/digitalpebble
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lewis
>>>>
>>>
>>
>>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Did you set follow redirects ?
On Sat, Feb 2, 2013 at 8:43 PM, Sebastian Nagel
<wa...@googlemail.com>wrote:
> Hi,
>
> the given URL is a redirect (HTTP 303, at least, when I try) with no
> content (only the HTTP header).
> Tried with curl and Nutch's parsechecker tool:
>
> % bin/nutch parsechecker
> "
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> "
> fetching: http://www.nytimes.com/...
> ...
> Content Metadata: Vary=Host Date=Sat, 02 Feb 2013 15:01:18 GMT
> Content-Length=0
> Location=
> http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=548bb88dQ2FRtezRXVzRQ3DQ3DQ3DRfzDQ2AR(tgrHttzEREQ26Q271RQ26Q27R1Q27Rz5gfXtQ2At.VRgf_X5r5hfQ51gG5Hrh_X!_Q2AzHQ51z5hX5Q3DhVtHGhz_D5rhgtDeQ2Bz5HrQ5EfzDQ2A
> Set-Cookie=RMID=007f0100777d510d2a3e0045; Expires=Sun, 02 Feb 2014
> 15:01:18 GMT; Path=/;
> Domain=.nytimes.com; Content-Type=text/plain Connection=close
> Server=Apache
> Parse Metadata: Content-Encoding=UTF-8 Content-Type=text/plain;
> charset=UTF-8
>
> % curl -v
> "
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> "
> >/dev/null
> ...
> > GET
>
> /2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> HTTP/1.1
> ...
> >
> < HTTP/1.1 303 See Other
> < Date: Sat, 02 Feb 2013 14:59:03 GMT
> < Server: Apache
> < Set-Cookie: RMID=007f01000e9f510d29b70033; Expires=Sun, 02 Feb 2014
> 14:59:03 GMT; Path=/;
> Domain=.nytimes.com;
> < Vary: Host
> < Location:
>
> http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=f39d9b3aQ2FQ2AmQ51dQ2AKSdQ2A(((Q2AQ7Ddg_Q2ANm46JmmdUQ2AUCVMQ2ACVQ2AMVQ2AdQ274Q7DKm_mrSQ2A4Q7DtKQ276Q27!Q7DQ7E42Q27J6!tKyt_dJQ7EdQ27!KQ27(!SmJ2!dtgQ276!4mgQ51ndQ27J6GQ7Ddg_
> < Content-Length: 0
> < Connection: close
> < Content-Type: text/plain
>
> Sebastian
>
>
> On 02/01/2013 05:47 AM, Sourajit Basak wrote:
> > Here it goes.
> >
> > Try to dump the content from this url with the following settings.
> >
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> >
> > <property>
> > <name>http.content.limit</name>
> > <value>-1</value>
> > </property>
> >
> > This page is gzip encoded. You will see that the fetcher is unable to
> > download any content. Check by inspecting the content-length.
> > Initially I was thinking it to be a problem with the parse-html plugin
> but
> > now it seems that the fetcher returns null content.
> >
> > This seemed related to NUTCH-374
> >
> > Let me know if you need further info.
> >
> > On Fri, Feb 1, 2013 at 1:54 AM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> >> Can you briefly describe the problem here Sourajit?
> >>
> >> On Thu, Jan 31, 2013 at 9:01 AM, Sourajit Basak
> >> <so...@gmail.com> wrote:
> >>> Seems to be related to NUTCH-374 but that shows as fixed.
> >>>
> >>> I have set Nutch to accept unlimited content size & this page is gzip
> >>> encoded.
> >>>
> >>>
> >>>
> >>> On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <
> >> sourajit.basac@gmail.com>wrote:
> >>>
> >>>> Re-opening this thread.
> >>>>
> >>>> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
> >>>> parse-html)
> >>>>
> >>>>
> >>
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> >>>>
> >>>> I do not get any content from the fetcher. This is my fetcher accept
> >>>> params.
> >>>> <property>
> >>>> <name>http.accept</name>
> >>>>
> >>>>
> >>
> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
> >>>> </property>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <
> >> sourajit.basac@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> Ignore my last post. Tika isn't slowing down, neither is this
> property.
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
> >>>>> sourajit.basac@gmail.com> wrote:
> >>>>>
> >>>>>> Enabling this property slows down the parse phase drastically when
> >>>>>> encountered with mime-type image/jpeg.
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
> >>>>>> sourajit.basac@gmail.com> wrote:
> >>>>>>
> >>>>>>> Thanks Julien.
> >>>>>>>
> >>>>>>> I can get the outlinks now, let me check if I can get the raw
> >> content.
> >>>>>>> I will update this thread.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
> >>>>>>> lists.digitalpebble@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> The parameter
> >>>>>>>>
> >>>>>>>> <property>
> >>>>>>>> <name>mime.type.magic</name>
> >>>>>>>> <value>true</value>
> >>>>>>>> <description>Defines if the mime content type detector uses
> magic
> >>>>>>>> resolution.
> >>>>>>>> </description>
> >>>>>>>> </property>
> >>>>>>>>
> >>>>>>>> should trigger the mime type detection based on the content and
> not
> >> on
> >>>>>>>> what
> >>>>>>>> the server returns. It is not a Tika issue as such as the
> selection
> >> of
> >>>>>>>> what
> >>>>>>>> parser to use is based on the mimetype that Nutch uses.
> >>>>>>>>
> >>>>>>>> The param above should be set to true by default. I thought we had
> >> more
> >>>>>>>> options but am probably confusing with the language identification
> >>>>>>>>
> >>>>>>>> Julien
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 25 November 2012 14:16, Sourajit Basak <
> sourajit.basac@gmail.com
> >>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> DEBUG tika.TikaParser - Using Tika parser
> >>>>>>>>> org.apache.tika.parser.txt.TXTParser for mime-type text/plain
> >>>>>>>>>
> >>>>>>>>> The above indicates Tika is fired. But somehow I need to tell
> Tika
> >>>>>>>> to use
> >>>>>>>>> HtmlParser for mime-type text/plain. Have to dig into Tika docs.
> >>>>>>>>>
> >>>>>>>>> Is it possible to do anything in Nutch ?
> >>>>>>>>>
> >>>>>>>>> On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
> >>>>>>>> sourajit.basac@gmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Some of my target webpages return a mime type of text/plain
> >> though
> >>>>>>>> they
> >>>>>>>>>> are htmls. I changed "http.accept" to include text/plain and
> >>>>>>>> configured
> >>>>>>>>>> both tika & parse-html to see if those can be parsed. However,
> >>>>>>>> both seem
> >>>>>>>>> to
> >>>>>>>>>> produce no content.
> >>>>>>>>>>
> >>>>>>>>>> I changed parse-plugins.xml & the corresponding plugin.xml's to
> >>>>>>>> match
> >>>>>>>>> this
> >>>>>>>>>> mime type.
> >>>>>>>>>>
> >>>>>>>>>> Has anyone encountered this problem ?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> *
> >>>>>>>> *Open Source Solutions for Text Engineering
> >>>>>>>>
> >>>>>>>> http://digitalpebble.blogspot.com/
> >>>>>>>> http://www.digitalpebble.com
> >>>>>>>> http://twitter.com/digitalpebble
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Lewis
> >>
> >
>
>
Re: mime type text/plain
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,
the given URL is a redirect (HTTP 303, at least, when I try) with no content (only the HTTP header).
Tried with curl and Nutch's parsechecker tool:
% bin/nutch parsechecker
"http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home"
fetching: http://www.nytimes.com/...
...
Content Metadata: Vary=Host Date=Sat, 02 Feb 2013 15:01:18 GMT Content-Length=0
Location=http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=548bb88dQ2FRtezRXVzRQ3DQ3DQ3DRfzDQ2AR(tgrHttzEREQ26Q271RQ26Q27R1Q27Rz5gfXtQ2At.VRgf_X5r5hfQ51gG5Hrh_X!_Q2AzHQ51z5hX5Q3DhVtHGhz_D5rhgtDeQ2Bz5HrQ5EfzDQ2A
Set-Cookie=RMID=007f0100777d510d2a3e0045; Expires=Sun, 02 Feb 2014 15:01:18 GMT; Path=/;
Domain=.nytimes.com; Content-Type=text/plain Connection=close Server=Apache
Parse Metadata: Content-Encoding=UTF-8 Content-Type=text/plain; charset=UTF-8
% curl -v
"http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home"
>/dev/null
...
> GET
/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
HTTP/1.1
...
>
< HTTP/1.1 303 See Other
< Date: Sat, 02 Feb 2013 14:59:03 GMT
< Server: Apache
< Set-Cookie: RMID=007f01000e9f510d29b70033; Expires=Sun, 02 Feb 2014 14:59:03 GMT; Path=/;
Domain=.nytimes.com;
< Vary: Host
< Location:
http://www.nytimes.com/glogin?URI=http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html&OQ=pagewantedQ3D2Q26_rQ3D1Q26refQ3Dglobal-homeQ26&OP=f39d9b3aQ2FQ2AmQ51dQ2AKSdQ2A(((Q2AQ7Ddg_Q2ANm46JmmdUQ2AUCVMQ2ACVQ2AMVQ2AdQ274Q7DKm_mrSQ2A4Q7DtKQ276Q27!Q7DQ7E42Q27J6!tKyt_dJQ7EdQ27!KQ27(!SmJ2!dtgQ276!4mgQ51ndQ27J6GQ7Ddg_
< Content-Length: 0
< Connection: close
< Content-Type: text/plain
Sebastian
On 02/01/2013 05:47 AM, Sourajit Basak wrote:
> Here it goes.
>
> Try to dump the content from this url with the following settings.
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>
> <property>
> <name>http.content.limit</name>
> <value>-1</value>
> </property>
>
> This page is gzip encoded. You will see that the fetcher is unable to
> download any content. Check by inspecting the content-length.
> Initially I was thinking it to be a problem with the parse-html plugin but
> now it seems that the fetcher returns null content.
>
> This seemed related to NUTCH-374
>
> Let me know if you need further info.
>
> On Fri, Feb 1, 2013 at 1:54 AM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Can you briefly describe the problem here Sourajit?
>>
>> On Thu, Jan 31, 2013 at 9:01 AM, Sourajit Basak
>> <so...@gmail.com> wrote:
>>> Seems to be related to NUTCH-374 but that shows as fixed.
>>>
>>> I have set Nutch to accept unlimited content size & this page is gzip
>>> encoded.
>>>
>>>
>>>
>>> On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <
>> sourajit.basac@gmail.com>wrote:
>>>
>>>> Re-opening this thread.
>>>>
>>>> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
>>>> parse-html)
>>>>
>>>>
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>>>>
>>>> I do not get any content from the fetcher. This is my fetcher accept
>>>> params.
>>>> <property>
>>>> <name>http.accept</name>
>>>>
>>>>
>> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <
>> sourajit.basac@gmail.com
>>>>> wrote:
>>>>
>>>>> Ignore my last post. Tika isn't slowing down, neither is this property.
>>>>>
>>>>>
>>>>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
>>>>> sourajit.basac@gmail.com> wrote:
>>>>>
>>>>>> Enabling this property slows down the parse phase drastically when
>>>>>> encountered with mime-type image/jpeg.
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
>>>>>> sourajit.basac@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Julien.
>>>>>>>
>>>>>>> I can get the outlinks now, let me check if I can get the raw
>> content.
>>>>>>> I will update this thread.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>>>>>>> lists.digitalpebble@gmail.com> wrote:
>>>>>>>
>>>>>>>> The parameter
>>>>>>>>
>>>>>>>> <property>
>>>>>>>> <name>mime.type.magic</name>
>>>>>>>> <value>true</value>
>>>>>>>> <description>Defines if the mime content type detector uses magic
>>>>>>>> resolution.
>>>>>>>> </description>
>>>>>>>> </property>
>>>>>>>>
>>>>>>>> should trigger the mime type detection based on the content and not
>> on
>>>>>>>> what
>>>>>>>> the server returns. It is not a Tika issue as such as the selection
>> of
>>>>>>>> what
>>>>>>>> parser to use is based on the mimetype that Nutch uses.
>>>>>>>>
>>>>>>>> The param above should be set to true by default. I thought we had
>> more
>>>>>>>> options but am probably confusing with the language identification
>>>>>>>>
>>>>>>>> Julien
>>>>>>>>
>>>>>>>>
>>>>>>>> On 25 November 2012 14:16, Sourajit Basak <sourajit.basac@gmail.com
>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> DEBUG tika.TikaParser - Using Tika parser
>>>>>>>>> org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>>>>>>>>
>>>>>>>>> The above indicates Tika is fired. But somehow I need to tell Tika
>>>>>>>> to use
>>>>>>>>> HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>>>>>>>>
>>>>>>>>> Is it possible to do anything in Nutch ?
>>>>>>>>>
>>>>>>>>> On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>>>>>>> sourajit.basac@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Some of my target webpages return a mime type of text/plain
>> though
>>>>>>>> they
>>>>>>>>>> are htmls. I changed "http.accept" to include text/plain and
>>>>>>>> configured
>>>>>>>>>> both tika & parse-html to see if those can be parsed. However,
>>>>>>>> both seem
>>>>>>>>> to
>>>>>>>>>> produce no content.
>>>>>>>>>>
>>>>>>>>>> I changed parse-plugins.xml & the corresponding plugin.xml's to
>>>>>>>> match
>>>>>>>>> this
>>>>>>>>>> mime type.
>>>>>>>>>>
>>>>>>>>>> Has anyone encountered this problem ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *
>>>>>>>> *Open Source Solutions for Text Engineering
>>>>>>>>
>>>>>>>> http://digitalpebble.blogspot.com/
>>>>>>>> http://www.digitalpebble.com
>>>>>>>> http://twitter.com/digitalpebble
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
>>
>> --
>> Lewis
>>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
>> ./nutch plugin urlfilter-regex
org.apache.nutch.urlfilter.regex.RegexURLFilter
http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
+
http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
Here is the relevant part of the regex file.
#-[?*!@=]
+^http://www\.nytimes\.com/([^\*@!])*(\s|$)
On Fri, Feb 1, 2013 at 10:48 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> And your regex rules?
> So is the URL fetched?
>
> On Thu, Jan 31, 2013 at 8:47 PM, Sourajit Basak
> <so...@gmail.com> wrote:
> > Here it goes.
> >
> > Try to dump the content from this url with the following settings.
> >
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> >
> > <property>
> > <name>http.content.limit</name>
> > <value>-1</value>
> > </property>
> >
> > This page is gzip encoded. You will see that the fetcher is unable to
> > download any content. Check by inspecting the content-length.
> > Initially I was thinking it to be a problem with the parse-html plugin
> but
> > now it seems that the fetcher returns null content.
> >
> > This seemed related to NUTCH-374
> >
> > Let me know if you need further info.
>
Re: mime type text/plain
Posted by Lewis John Mcgibbney <le...@gmail.com>.
And your regex rules?
So is the URL fetched?
On Thu, Jan 31, 2013 at 8:47 PM, Sourajit Basak
<so...@gmail.com> wrote:
> Here it goes.
>
> Try to dump the content from this url with the following settings.
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>
> <property>
> <name>http.content.limit</name>
> <value>-1</value>
> </property>
>
> This page is gzip encoded. You will see that the fetcher is unable to
> download any content. Check by inspecting the content-length.
> Initially I was thinking it to be a problem with the parse-html plugin but
> now it seems that the fetcher returns null content.
>
> This seemed related to NUTCH-374
>
> Let me know if you need further info.
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Here it goes.
Try to dump the content from this url with the following settings.
http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
<property>
<name>http.content.limit</name>
<value>-1</value>
</property>
This page is gzip encoded. You will see that the fetcher is unable to
download any content. Check by inspecting the content-length.
Initially I was thinking it to be a problem with the parse-html plugin but
now it seems that the fetcher returns null content.
This seemed related to NUTCH-374
Let me know if you need further info.
On Fri, Feb 1, 2013 at 1:54 AM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Can you briefly describe the problem here Sourajit?
>
> On Thu, Jan 31, 2013 at 9:01 AM, Sourajit Basak
> <so...@gmail.com> wrote:
> > Seems to be related to NUTCH-374 but that shows as fixed.
> >
> > I have set Nutch to accept unlimited content size & this page is gzip
> > encoded.
> >
> >
> >
> > On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <
> sourajit.basac@gmail.com>wrote:
> >
> >> Re-opening this thread.
> >>
> >> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
> >> parse-html)
> >>
> >>
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
> >>
> >> I do not get any content from the fetcher. This is my fetcher accept
> >> params.
> >> <property>
> >> <name>http.accept</name>
> >>
> >>
> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
> >> </property>
> >>
> >>
> >>
> >>
> >> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <
> sourajit.basac@gmail.com
> >> > wrote:
> >>
> >>> Ignore my last post. Tika isn't slowing down, neither is this property.
> >>>
> >>>
> >>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
> >>> sourajit.basac@gmail.com> wrote:
> >>>
> >>>> Enabling this property slows down the parse phase drastically when
> >>>> encountered with mime-type image/jpeg.
> >>>>
> >>>>
> >>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
> >>>> sourajit.basac@gmail.com> wrote:
> >>>>
> >>>>> Thanks Julien.
> >>>>>
> >>>>> I can get the outlinks now, let me check if I can get the raw
> content.
> >>>>> I will update this thread.
> >>>>>
> >>>>>
> >>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
> >>>>> lists.digitalpebble@gmail.com> wrote:
> >>>>>
> >>>>>> The parameter
> >>>>>>
> >>>>>> <property>
> >>>>>> <name>mime.type.magic</name>
> >>>>>> <value>true</value>
> >>>>>> <description>Defines if the mime content type detector uses magic
> >>>>>> resolution.
> >>>>>> </description>
> >>>>>> </property>
> >>>>>>
> >>>>>> should trigger the mime type detection based on the content and not
> on
> >>>>>> what
> >>>>>> the server returns. It is not a Tika issue as such as the selection
> of
> >>>>>> what
> >>>>>> parser to use is based on the mimetype that Nutch uses.
> >>>>>>
> >>>>>> The param above should be set to true by default. I thought we had
> more
> >>>>>> options but am probably confusing with the language identification
> >>>>>>
> >>>>>> Julien
> >>>>>>
> >>>>>>
> >>>>>> On 25 November 2012 14:16, Sourajit Basak <sourajit.basac@gmail.com
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>> > DEBUG tika.TikaParser - Using Tika parser
> >>>>>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
> >>>>>> >
> >>>>>> > The above indicates Tika is fired. But somehow I need to tell Tika
> >>>>>> to use
> >>>>>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
> >>>>>> >
> >>>>>> > Is it possible to do anything in Nutch ?
> >>>>>> >
> >>>>>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
> >>>>>> sourajit.basac@gmail.com
> >>>>>> > >wrote:
> >>>>>> >
> >>>>>> > > Some of my target webpages return a mime type of text/plain
> though
> >>>>>> they
> >>>>>> > > are htmls. I changed "http.accept" to include text/plain and
> >>>>>> configured
> >>>>>> > > both tika & parse-html to see if those can be parsed. However,
> >>>>>> both seem
> >>>>>> > to
> >>>>>> > > produce no content.
> >>>>>> > >
> >>>>>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to
> >>>>>> match
> >>>>>> > this
> >>>>>> > > mime type.
> >>>>>> > >
> >>>>>> > > Has anyone encountered this problem ?
> >>>>>> > >
> >>>>>> > >
> >>>>>> > >
> >>>>>> >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *
> >>>>>> *Open Source Solutions for Text Engineering
> >>>>>>
> >>>>>> http://digitalpebble.blogspot.com/
> >>>>>> http://www.digitalpebble.com
> >>>>>> http://twitter.com/digitalpebble
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
>
> --
> Lewis
>
Re: mime type text/plain
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Can you briefly describe the problem here Sourajit?
On Thu, Jan 31, 2013 at 9:01 AM, Sourajit Basak
<so...@gmail.com> wrote:
> Seems to be related to NUTCH-374 but that shows as fixed.
>
> I have set Nutch to accept unlimited content size & this page is gzip
> encoded.
>
>
>
> On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <so...@gmail.com>wrote:
>
>> Re-opening this thread.
>>
>> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
>> parse-html)
>>
>> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>>
>> I do not get any content from the fetcher. This is my fetcher accept
>> params.
>> <property>
>> <name>http.accept</name>
>>
>> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
>> </property>
>>
>>
>>
>>
>> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <sourajit.basac@gmail.com
>> > wrote:
>>
>>> Ignore my last post. Tika isn't slowing down, neither is this property.
>>>
>>>
>>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
>>> sourajit.basac@gmail.com> wrote:
>>>
>>>> Enabling this property slows down the parse phase drastically when
>>>> encountered with mime-type image/jpeg.
>>>>
>>>>
>>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
>>>> sourajit.basac@gmail.com> wrote:
>>>>
>>>>> Thanks Julien.
>>>>>
>>>>> I can get the outlinks now, let me check if I can get the raw content.
>>>>> I will update this thread.
>>>>>
>>>>>
>>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>>>>> lists.digitalpebble@gmail.com> wrote:
>>>>>
>>>>>> The parameter
>>>>>>
>>>>>> <property>
>>>>>> <name>mime.type.magic</name>
>>>>>> <value>true</value>
>>>>>> <description>Defines if the mime content type detector uses magic
>>>>>> resolution.
>>>>>> </description>
>>>>>> </property>
>>>>>>
>>>>>> should trigger the mime type detection based on the content and not on
>>>>>> what
>>>>>> the server returns. It is not a Tika issue as such as the selection of
>>>>>> what
>>>>>> parser to use is based on the mimetype that Nutch uses.
>>>>>>
>>>>>> The param above should be set to true by default. I thought we had more
>>>>>> options but am probably confusing with the language identification
>>>>>>
>>>>>> Julien
>>>>>>
>>>>>>
>>>>>> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > DEBUG tika.TikaParser - Using Tika parser
>>>>>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>>>>> >
>>>>>> > The above indicates Tika is fired. But somehow I need to tell Tika
>>>>>> to use
>>>>>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>>>>> >
>>>>>> > Is it possible to do anything in Nutch ?
>>>>>> >
>>>>>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>>>>> sourajit.basac@gmail.com
>>>>>> > >wrote:
>>>>>> >
>>>>>> > > Some of my target webpages return a mime type of text/plain though
>>>>>> they
>>>>>> > > are htmls. I changed "http.accept" to include text/plain and
>>>>>> configured
>>>>>> > > both tika & parse-html to see if those can be parsed. However,
>>>>>> both seem
>>>>>> > to
>>>>>> > > produce no content.
>>>>>> > >
>>>>>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to
>>>>>> match
>>>>>> > this
>>>>>> > > mime type.
>>>>>> > >
>>>>>> > > Has anyone encountered this problem ?
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *
>>>>>> *Open Source Solutions for Text Engineering
>>>>>>
>>>>>> http://digitalpebble.blogspot.com/
>>>>>> http://www.digitalpebble.com
>>>>>> http://twitter.com/digitalpebble
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
--
Lewis
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Seems to be related to NUTCH-374 but that shows as fixed.
I have set Nutch to accept unlimited content size & this page is gzip
encoded.
On Thu, Jan 31, 2013 at 9:38 PM, Sourajit Basak <so...@gmail.com>wrote:
> Re-opening this thread.
>
> Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
> parse-html)
>
> http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
>
> I do not get any content from the fetcher. This is my fetcher accept
> params.
> <property>
> <name>http.accept</name>
>
> <value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
> </property>
>
>
>
>
> On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak <sourajit.basac@gmail.com
> > wrote:
>
>> Ignore my last post. Tika isn't slowing down, neither is this property.
>>
>>
>> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <
>> sourajit.basac@gmail.com> wrote:
>>
>>> Enabling this property slows down the parse phase drastically when
>>> encountered with mime-type image/jpeg.
>>>
>>>
>>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <
>>> sourajit.basac@gmail.com> wrote:
>>>
>>>> Thanks Julien.
>>>>
>>>> I can get the outlinks now, let me check if I can get the raw content.
>>>> I will update this thread.
>>>>
>>>>
>>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>>>> lists.digitalpebble@gmail.com> wrote:
>>>>
>>>>> The parameter
>>>>>
>>>>> <property>
>>>>> <name>mime.type.magic</name>
>>>>> <value>true</value>
>>>>> <description>Defines if the mime content type detector uses magic
>>>>> resolution.
>>>>> </description>
>>>>> </property>
>>>>>
>>>>> should trigger the mime type detection based on the content and not on
>>>>> what
>>>>> the server returns. It is not a Tika issue as such as the selection of
>>>>> what
>>>>> parser to use is based on the mimetype that Nutch uses.
>>>>>
>>>>> The param above should be set to true by default. I thought we had more
>>>>> options but am probably confusing with the language identification
>>>>>
>>>>> Julien
>>>>>
>>>>>
>>>>> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > DEBUG tika.TikaParser - Using Tika parser
>>>>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>>>> >
>>>>> > The above indicates Tika is fired. But somehow I need to tell Tika
>>>>> to use
>>>>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>>>> >
>>>>> > Is it possible to do anything in Nutch ?
>>>>> >
>>>>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>>>> sourajit.basac@gmail.com
>>>>> > >wrote:
>>>>> >
>>>>> > > Some of my target webpages return a mime type of text/plain though
>>>>> they
>>>>> > > are htmls. I changed "http.accept" to include text/plain and
>>>>> configured
>>>>> > > both tika & parse-html to see if those can be parsed. However,
>>>>> both seem
>>>>> > to
>>>>> > > produce no content.
>>>>> > >
>>>>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to
>>>>> match
>>>>> > this
>>>>> > > mime type.
>>>>> > >
>>>>> > > Has anyone encountered this problem ?
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *
>>>>> *Open Source Solutions for Text Engineering
>>>>>
>>>>> http://digitalpebble.blogspot.com/
>>>>> http://www.digitalpebble.com
>>>>> http://twitter.com/digitalpebble
>>>>>
>>>>
>>>>
>>>
>>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Re-opening this thread.
Using Nutch v1.5 try to get the parseText from this NYTimes url (Use
parse-html)
http://www.nytimes.com/2013/01/31/technology/chinese-hackers-infiltrate-new-york-times-computers.html?pagewanted=2&_r=0&ref=global-home
I do not get any content from the fetcher. This is my fetcher accept params.
<property>
<name>http.accept</name>
<value>text/plain,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</value>
</property>
On Mon, Nov 26, 2012 at 11:03 PM, Sourajit Basak
<so...@gmail.com>wrote:
> Ignore my last post. Tika isn't slowing down, neither is this property.
>
>
> On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak <sourajit.basac@gmail.com
> > wrote:
>
>> Enabling this property slows down the parse phase drastically when
>> encountered with mime-type image/jpeg.
>>
>>
>> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <sourajit.basac@gmail.com
>> > wrote:
>>
>>> Thanks Julien.
>>>
>>> I can get the outlinks now, let me check if I can get the raw content. I
>>> will update this thread.
>>>
>>>
>>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>>> lists.digitalpebble@gmail.com> wrote:
>>>
>>>> The parameter
>>>>
>>>> <property>
>>>> <name>mime.type.magic</name>
>>>> <value>true</value>
>>>> <description>Defines if the mime content type detector uses magic
>>>> resolution.
>>>> </description>
>>>> </property>
>>>>
>>>> should trigger the mime type detection based on the content and not on
>>>> what
>>>> the server returns. It is not a Tika issue as such as the selection of
>>>> what
>>>> parser to use is based on the mimetype that Nutch uses.
>>>>
>>>> The param above should be set to true by default. I thought we had more
>>>> options but am probably confusing with the language identification
>>>>
>>>> Julien
>>>>
>>>>
>>>> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
>>>> wrote:
>>>>
>>>> > DEBUG tika.TikaParser - Using Tika parser
>>>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>>> >
>>>> > The above indicates Tika is fired. But somehow I need to tell Tika to
>>>> use
>>>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>>> >
>>>> > Is it possible to do anything in Nutch ?
>>>> >
>>>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>>> sourajit.basac@gmail.com
>>>> > >wrote:
>>>> >
>>>> > > Some of my target webpages return a mime type of text/plain though
>>>> they
>>>> > > are htmls. I changed "http.accept" to include text/plain and
>>>> configured
>>>> > > both tika & parse-html to see if those can be parsed. However, both
>>>> seem
>>>> > to
>>>> > > produce no content.
>>>> > >
>>>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to
>>>> match
>>>> > this
>>>> > > mime type.
>>>> > >
>>>> > > Has anyone encountered this problem ?
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> *
>>>> *Open Source Solutions for Text Engineering
>>>>
>>>> http://digitalpebble.blogspot.com/
>>>> http://www.digitalpebble.com
>>>> http://twitter.com/digitalpebble
>>>>
>>>
>>>
>>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Ignore my last post. Tika isn't slowing down, neither is this property.
On Mon, Nov 26, 2012 at 10:50 PM, Sourajit Basak
<so...@gmail.com>wrote:
> Enabling this property slows down the parse phase drastically when
> encountered with mime-type image/jpeg.
>
>
> On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <so...@gmail.com>wrote:
>
>> Thanks Julien.
>>
>> I can get the outlinks now, let me check if I can get the raw content. I
>> will update this thread.
>>
>>
>> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
>> lists.digitalpebble@gmail.com> wrote:
>>
>>> The parameter
>>>
>>> <property>
>>> <name>mime.type.magic</name>
>>> <value>true</value>
>>> <description>Defines if the mime content type detector uses magic
>>> resolution.
>>> </description>
>>> </property>
>>>
>>> should trigger the mime type detection based on the content and not on
>>> what
>>> the server returns. It is not a Tika issue as such as the selection of
>>> what
>>> parser to use is based on the mimetype that Nutch uses.
>>>
>>> The param above should be set to true by default. I thought we had more
>>> options but am probably confusing with the language identification
>>>
>>> Julien
>>>
>>>
>>> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
>>> wrote:
>>>
>>> > DEBUG tika.TikaParser - Using Tika parser
>>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>>> >
>>> > The above indicates Tika is fired. But somehow I need to tell Tika to
>>> use
>>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>>> >
>>> > Is it possible to do anything in Nutch ?
>>> >
>>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>>> sourajit.basac@gmail.com
>>> > >wrote:
>>> >
>>> > > Some of my target webpages return a mime type of text/plain though
>>> they
>>> > > are htmls. I changed "http.accept" to include text/plain and
>>> configured
>>> > > both tika & parse-html to see if those can be parsed. However, both
>>> seem
>>> > to
>>> > > produce no content.
>>> > >
>>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to match
>>> > this
>>> > > mime type.
>>> > >
>>> > > Has anyone encountered this problem ?
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> *
>>> *Open Source Solutions for Text Engineering
>>>
>>> http://digitalpebble.blogspot.com/
>>> http://www.digitalpebble.com
>>> http://twitter.com/digitalpebble
>>>
>>
>>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Enabling this property slows down the parse phase drastically when
encountered with mime-type image/jpeg.
On Mon, Nov 26, 2012 at 8:07 PM, Sourajit Basak <so...@gmail.com>wrote:
> Thanks Julien.
>
> I can get the outlinks now, let me check if I can get the raw content. I
> will update this thread.
>
>
> On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
>> The parameter
>>
>> <property>
>> <name>mime.type.magic</name>
>> <value>true</value>
>> <description>Defines if the mime content type detector uses magic
>> resolution.
>> </description>
>> </property>
>>
>> should trigger the mime type detection based on the content and not on
>> what
>> the server returns. It is not a Tika issue as such as the selection of
>> what
>> parser to use is based on the mimetype that Nutch uses.
>>
>> The param above should be set to true by default. I thought we had more
>> options but am probably confusing with the language identification
>>
>> Julien
>>
>>
>> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
>> wrote:
>>
>> > DEBUG tika.TikaParser - Using Tika parser
>> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>> >
>> > The above indicates Tika is fired. But somehow I need to tell Tika to
>> use
>> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>> >
>> > Is it possible to do anything in Nutch ?
>> >
>> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
>> sourajit.basac@gmail.com
>> > >wrote:
>> >
>> > > Some of my target webpages return a mime type of text/plain though
>> they
>> > > are htmls. I changed "http.accept" to include text/plain and
>> configured
>> > > both tika & parse-html to see if those can be parsed. However, both
>> seem
>> > to
>> > > produce no content.
>> > >
>> > > I changed parse-plugins.xml & the corresponding plugin.xml's to match
>> > this
>> > > mime type.
>> > >
>> > > Has anyone encountered this problem ?
>> > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> *
>> *Open Source Solutions for Text Engineering
>>
>> http://digitalpebble.blogspot.com/
>> http://www.digitalpebble.com
>> http://twitter.com/digitalpebble
>>
>
>
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
Thanks Julien.
I can get the outlinks now, let me check if I can get the raw content. I
will update this thread.
On Mon, Nov 26, 2012 at 2:37 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
> The parameter
>
> <property>
> <name>mime.type.magic</name>
> <value>true</value>
> <description>Defines if the mime content type detector uses magic
> resolution.
> </description>
> </property>
>
> should trigger the mime type detection based on the content and not on what
> the server returns. It is not a Tika issue as such as the selection of what
> parser to use is based on the mimetype that Nutch uses.
>
> The param above should be set to true by default. I thought we had more
> options but am probably confusing with the language identification
>
> Julien
>
>
> On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com>
> wrote:
>
> > DEBUG tika.TikaParser - Using Tika parser
> > org.apache.tika.parser.txt.TXTParser for mime-type text/plain
> >
> > The above indicates Tika is fired. But somehow I need to tell Tika to use
> > HtmlParser for mime-type text/plain. Have to dig into Tika docs.
> >
> > Is it possible to do anything in Nutch ?
> >
> > On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <
> sourajit.basac@gmail.com
> > >wrote:
> >
> > > Some of my target webpages return a mime type of text/plain though they
> > > are htmls. I changed "http.accept" to include text/plain and configured
> > > both tika & parse-html to see if those can be parsed. However, both
> seem
> > to
> > > produce no content.
> > >
> > > I changed parse-plugins.xml & the corresponding plugin.xml's to match
> > this
> > > mime type.
> > >
> > > Has anyone encountered this problem ?
> > >
> > >
> > >
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
Re: mime type text/plain
Posted by Julien Nioche <li...@gmail.com>.
The parameter
<property>
<name>mime.type.magic</name>
<value>true</value>
<description>Defines if the mime content type detector uses magic
resolution.
</description>
</property>
should trigger the mime type detection based on the content and not on what
the server returns. It is not a Tika issue as such as the selection of what
parser to use is based on the mimetype that Nutch uses.
The param above should be set to true by default. I thought we had more
options but am probably confusing with the language identification
Julien
On 25 November 2012 14:16, Sourajit Basak <so...@gmail.com> wrote:
> DEBUG tika.TikaParser - Using Tika parser
> org.apache.tika.parser.txt.TXTParser for mime-type text/plain
>
> The above indicates Tika is fired. But somehow I need to tell Tika to use
> HtmlParser for mime-type text/plain. Have to dig into Tika docs.
>
> Is it possible to do anything in Nutch ?
>
> On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <sourajit.basac@gmail.com
> >wrote:
>
> > Some of my target webpages return a mime type of text/plain though they
> > are htmls. I changed "http.accept" to include text/plain and configured
> > both tika & parse-html to see if those can be parsed. However, both seem
> to
> > produce no content.
> >
> > I changed parse-plugins.xml & the corresponding plugin.xml's to match
> this
> > mime type.
> >
> > Has anyone encountered this problem ?
> >
> >
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: mime type text/plain
Posted by Sourajit Basak <so...@gmail.com>.
DEBUG tika.TikaParser - Using Tika parser
org.apache.tika.parser.txt.TXTParser for mime-type text/plain
The above indicates Tika is fired. But somehow I need to tell Tika to use
HtmlParser for mime-type text/plain. Have to dig into Tika docs.
Is it possible to do anything in Nutch ?
On Sun, Nov 25, 2012 at 7:27 PM, Sourajit Basak <so...@gmail.com>wrote:
> Some of my target webpages return a mime type of text/plain though they
> are htmls. I changed "http.accept" to include text/plain and configured
> both tika & parse-html to see if those can be parsed. However, both seem to
> produce no content.
>
> I changed parse-plugins.xml & the corresponding plugin.xml's to match this
> mime type.
>
> Has anyone encountered this problem ?
>
>
>