You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Eyeris Rodriguez Rueda <er...@uci.cu> on 2012/11/25 00:42:11 UTC
problem with text/html content type of documents appears application/xhtml+xml in solr index
Hi.
I have changed my nutch version from 1.4 to 1.5.1 and I have detected a problem with content type of some document, some pages with text/html appears in solr index with application/xhtml+xml , when I check the links the navegator tell me that efectively is text/html.
Any body can help me to fix this problem, I think change this content type manually in solr index to text/html but is not a good way for me.
Please any suggestion or advice will be accepted.
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
RE: problem with text/html content type of documents appears
application/xhtml+xml in solr index
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - you need to enable mime-type mapping in Nutch config and define your mappings. Enable it with:
<property>
<name>moreIndexingFilter.mapMimeTypes</name>
<value>true</value>
</property>
and add the following to your mapping config:
cat conf/contenttype-mapping.txt
# Target content type <TAB> type1 [<TAB> type2 ...]
text/html application/xhtml+xml
This will map application/xhtml+xml to text/html when indexing documents to Solr. You can configure any arbitrary target such as `web page` or `document` for various similar content types.
Trunk has this feature. You can either patch your version or check out from trunk and compile Nutch yourself. Patching is very simple:
$ cd trunk ; patch -p0 < file.patch
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Sun 25-Nov-2012 20:42
> To: user@nutch.apache.org
> Subject: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Thanks a lot Markus for your answer. My English is not so good.
> I was reading but i don’t know how to fix the problems yet. Could you explain me in details the solution please. I was looking in conf directory but I can't find how to map one mime types to another. I need to replace index-more plugin ?
> I was looking in the link that you suggest me and a saw a NUTCH-1262-1.5-1.patch but I don’t know how to use that patch.
> Please tell me if I need to delete the index completely or there is a way to replace an application/xhtml+xml to text/html in solr index.
>
>
>
>
> -----Mensaje original-----
> De: Markus Jelsma [mailto:markus.jelsma@openindex.io]
> Enviado el: domingo, 25 de noviembre de 2012 4:33 AM
> Para: user@nutch.apache.org
> Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi - trunk's more indexing filter can map mime types to any target. With it you can map both (x)html mimes to text/html or to `web page`.
>
> https://issues.apache.org/jira/browse/NUTCH-1262
>
>
> -----Original message-----
> > From:Eyeris Rodriguez Rueda <er...@uci.cu>
> > Sent: Sun 25-Nov-2012 00:48
> > To: user@nutch.apache.org
> > Subject: problem with text/html content type of documents appears application/xhtml+xml in solr index
> >
> > Hi.
> >
> > I have changed my nutch version from 1.4 to 1.5.1 and I have detected a problem with content type of some document, some pages with text/html appears in solr index with application/xhtml+xml , when I check the links the navegator tell me that efectively is text/html.
> > Any body can help me to fix this problem, I think change this content type manually in solr index to text/html but is not a good way for me.
> > Please any suggestion or advice will be accepted.
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci
>
Re: problem with text/html content type of documents appears application/xhtml+xml in solr index
Posted by Eyeris Rodriguez Rueda <er...@uci.cu>.
Thanks a lot for your time markus, could you give a link to download index-more.jar fixed
----- Mensaje original -----
De: "Markus Jelsma" <ma...@openindex.io>
Para: user@nutch.apache.org
Enviados: Martes, 27 de Noviembre 2012 16:26:19
Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
You indeed need to replace the jar if you come from an older version and if it hasn't been updated.
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Tue 27-Nov-2012 22:23
> To: user@nutch.apache.org
> Subject: Re: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi.
> I have checked the separation with tab in contenttype-mapping.txt but is ok.
> This is my console out with index checker tool
>
>
> bin/nutch indexchecker http://comunidades.uci.cu/
> fetching: http://comunidades.uci.cu/
> parsing: http://comunidades.uci.cu/
> contentType: application/xhtml+xml
> host : comunidades.uci.cu
> custom-h1 : Comunidades UCI
> tstamp : Tue Nov 27 16:10:47 CST 2012
> custom-strong : GESPRO
> custom-strong : Personalice su página según sus necesidades y preferencias
> date : Tue Nov 27 16:10:47 CST 2012
> type : application/xhtml+xml
> type : application
> type : xhtml+xml
> lang : gl
> url : http://comunidades.uci.cu/
> content : Comunidades UCI Sign in Register Home Projects Help Search : Comunidades UCI DevGirls BlackHat coden
> title : Comunidades UCI
> inlinks : #Ayuda
> inlinks : #Cómoen-Comunidades-UCI
> inlinks : #Documéntese
> inlinks : /
> inlinks : /account/register
> inlinks : /gespro_help
> inlinks : /links
> inlinks : /login
> inlinks : /news
> inlinks : /news/76
> inlinks : /news/77
> inlinks : /news/78
> inlinks : /news/79
> inlinks : /news/80
> inlinks : /projects
> inlinks : /projects/coj
> inlinks : /projects/comunidad
> inlinks : /projects/kgetmms
> inlinks : /projects/konoha
> inlinks : /projects/lua
> inlinks : /projects/lxkblayoutsetter
> inlinks : /projects/lycan-ide
> inlinks : /projects/progcomp
> inlinks : /projects/sigest
> inlinks : /projects/xfce4-globalmenu
> inlinks : /search
> inlinks : /tab/system_show
> inlinks : /users/132
> inlinks : /users/440
> inlinks : /users/629
> inlinks : /users/976
> inlinks : http://comunidades.uci.cu
> inlinks : http://comunidades.uci.cu/gespro-help/
> inlinks : http://comunidades.uci.cu/git/lxkblayoutsetter.git
> inlinks : http://comunidades.uci.cu/projects
> inlinks : http://comunidades.uci.cu/projects/cised
> inlinks : http://comunidades.uci.cu/projects/erlang
> inlinks : http://comunidades.uci.cu/projects/java
> inlinks : http://comunidades.uci.cu/projects/porycomgobelectronic
> inlinks : http://comunidades.uci.cu/projects/progcomp
> inlinks : http://comunidades.uci.cu/projects/proycaxtor
> inlinks : http://comunidades.uci.cu/projects/proycombd
> inlinks : http://comunidades.uci.cu/projects/proycombpmsoa
> inlinks : http://comunidades.uci.cu/projects/proycomcedin
> inlinks : http://comunidades.uci.cu/projects/proycomcolaborativo
> inlinks : http://comunidades.uci.cu/projects/proycommultimedia
> inlinks : http://comunidades.uci.cu/projects/proycomprogramacion
> inlinks : http://comunidades.uci.cu/projects/proycomsoftlibre
> inlinks : http://comunidades.uci.cu/projects/proyphp
> inlinks : http://comunidades.uci.cu/projects/ucicom
> inlinks : http://comunidades.uci.cu/projects/ucicom/documents
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Acerca_de_Comunidades_UCI
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Autenticación
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/C%C3%B3digo_de_%C3%A9tica
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Contacto
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Cr%C3%A9ditos_y_bonificaciones
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Gu%C3%ADa_de_proyectos_exitosos
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Lineamientos_m%C3%ADnimos_de_calidad
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Manual_del_usuario
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Preguntas_Frecuentes
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Registro
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Reporte_incidencias
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Revisi%C3%B3n_de_componentes
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nueva_comunidad
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nuevo_proyecto
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/T%C3%A9rminos_y_Condiciones
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Unirse_a_un_proyecto
> outlinks : http://blogs.prod.uci.cu/BlackHat
> outlinks : http://blogs.prod.uci.cu/DevGirls
> outlinks : http://blogs.prod.uci.cu/codeninja
> outlinks : http://blogs.prod.uci.cu/desarrolloSOA/
> outlinks : http://blogs.prod.uci.cu/echo
> outlinks : http://blogs.prod.uci.cu/gestionproyecto/
> outlinks : http://blogs.prod.uci.cu/humanOS
> outlinks : http://blogs.prod.uci.cu/micro
> outlinks : http://blogs.prod.uci.cu/nova/
> outlinks : http://coj.uci.cu/general/about.xhtml
> outlinks : http://pgs.soporte.uci.cu
> outlinks : http://portal.albet.prod.uci.cu
> outlinks : http://portal.calisoft.prod.uci.cu
> outlinks : http://portal.cdae.prod.uci.cu
> outlinks : http://portal.cedin.prod.uci.cu
> outlinks : http://portal.cegel.prod.uci.cu
> outlinks : http://portal.ceige.prod.uci.cu
> outlinks : http://portal.cenia.prod.uci.cu
> outlinks : http://portal.cesim.prod.uci.cu
> outlinks : http://portal.cice.prod.uci.cu
> outlinks : http://portal.cidi.prod.uci.cu
> outlinks : http://portal.cised.prod.uci.cu
> outlinks : http://portal.datec.prod.uci.cu
> outlinks : http://portal.dgp.prod.uci.cu
> outlinks : http://portal.dt.prod.uci.cu
> outlinks : http://portal.fortes.prod.uci.cu
> outlinks : http://portal.frcav.cav.uci.cu
> outlinks : http://portal.frgrm.grm.uci.cu
> outlinks : http://portal.frhab.hab.uci.cu
> outlinks : http://portal.geitel.prod.uci.cu
> outlinks : http://portal.geysed.prod.uci.cu
> outlinks : http://portal.hlg.uci.cu
> outlinks : http://portal.isec.prod.uci.cu
> outlinks : http://portal.tlm.prod.uci.cu
> outlinks : http://portal.vcl.uci.cu/
> outlinks : http://postgresql.uci.cu
> outlinks : http://www.redmine.org/
> outlinks : http://www.redmine.org/guide
> contentLength : 5280
>
> and this is the page code that i check with firefox.
>
> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
> <head>
> <meta http-equiv="content-type" content="text/html; charset=utf-8" />
> <title>Comunidades UCI</title>
> ....continue
>
>
>
> I need to replace index-more.jar plugin ?
>
>
>
>
> ----- Mensaje original -----
> De: "Markus Jelsma" <ma...@openindex.io>
> Para: user@nutch.apache.org
> Enviados: Martes, 27 de Noviembre 2012 15:33:20
> Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi - are you sure you have tabs separating the target and the mapped mimes? Use the nutch indexchecker tool to quickly test if it works.
>
> -----Original message-----
> > From:Eyeris Rodriguez Rueda <er...@uci.cu>
> > Sent: Tue 27-Nov-2012 21:18
> > To: user@nutch.apache.org
> > Subject: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
> >
> > Hi. Markus.
> > I was doing your recommendations but, my problem persist, some documents still with application/xhtml+xml instead of text/html.
> > I add the property to nutch-site.xml and make the conf/contenttype-mapping.txt file
> > <property>
> > <name>moreIndexingFilter.mapMimeTypes</name>
> > <value>true</value>
> > </property>
> >
> > I'm using nutch 1.5.1. Tell me if I need to replace index-more.jar in plugin directory with any fixed version ?
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
RE: problem with text/html content type of documents appears
application/xhtml+xml in solr index
Posted by Markus Jelsma <ma...@openindex.io>.
You indeed need to replace the jar if you come from an older version and if it hasn't been updated.
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Tue 27-Nov-2012 22:23
> To: user@nutch.apache.org
> Subject: Re: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi.
> I have checked the separation with tab in contenttype-mapping.txt but is ok.
> This is my console out with index checker tool
>
>
> bin/nutch indexchecker http://comunidades.uci.cu/
> fetching: http://comunidades.uci.cu/
> parsing: http://comunidades.uci.cu/
> contentType: application/xhtml+xml
> host : comunidades.uci.cu
> custom-h1 : Comunidades UCI
> tstamp : Tue Nov 27 16:10:47 CST 2012
> custom-strong : GESPRO
> custom-strong : Personalice su página según sus necesidades y preferencias
> date : Tue Nov 27 16:10:47 CST 2012
> type : application/xhtml+xml
> type : application
> type : xhtml+xml
> lang : gl
> url : http://comunidades.uci.cu/
> content : Comunidades UCI Sign in Register Home Projects Help Search : Comunidades UCI DevGirls BlackHat coden
> title : Comunidades UCI
> inlinks : #Ayuda
> inlinks : #Cómoen-Comunidades-UCI
> inlinks : #Documéntese
> inlinks : /
> inlinks : /account/register
> inlinks : /gespro_help
> inlinks : /links
> inlinks : /login
> inlinks : /news
> inlinks : /news/76
> inlinks : /news/77
> inlinks : /news/78
> inlinks : /news/79
> inlinks : /news/80
> inlinks : /projects
> inlinks : /projects/coj
> inlinks : /projects/comunidad
> inlinks : /projects/kgetmms
> inlinks : /projects/konoha
> inlinks : /projects/lua
> inlinks : /projects/lxkblayoutsetter
> inlinks : /projects/lycan-ide
> inlinks : /projects/progcomp
> inlinks : /projects/sigest
> inlinks : /projects/xfce4-globalmenu
> inlinks : /search
> inlinks : /tab/system_show
> inlinks : /users/132
> inlinks : /users/440
> inlinks : /users/629
> inlinks : /users/976
> inlinks : http://comunidades.uci.cu
> inlinks : http://comunidades.uci.cu/gespro-help/
> inlinks : http://comunidades.uci.cu/git/lxkblayoutsetter.git
> inlinks : http://comunidades.uci.cu/projects
> inlinks : http://comunidades.uci.cu/projects/cised
> inlinks : http://comunidades.uci.cu/projects/erlang
> inlinks : http://comunidades.uci.cu/projects/java
> inlinks : http://comunidades.uci.cu/projects/porycomgobelectronic
> inlinks : http://comunidades.uci.cu/projects/progcomp
> inlinks : http://comunidades.uci.cu/projects/proycaxtor
> inlinks : http://comunidades.uci.cu/projects/proycombd
> inlinks : http://comunidades.uci.cu/projects/proycombpmsoa
> inlinks : http://comunidades.uci.cu/projects/proycomcedin
> inlinks : http://comunidades.uci.cu/projects/proycomcolaborativo
> inlinks : http://comunidades.uci.cu/projects/proycommultimedia
> inlinks : http://comunidades.uci.cu/projects/proycomprogramacion
> inlinks : http://comunidades.uci.cu/projects/proycomsoftlibre
> inlinks : http://comunidades.uci.cu/projects/proyphp
> inlinks : http://comunidades.uci.cu/projects/ucicom
> inlinks : http://comunidades.uci.cu/projects/ucicom/documents
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Acerca_de_Comunidades_UCI
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Autenticación
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/C%C3%B3digo_de_%C3%A9tica
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Contacto
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Cr%C3%A9ditos_y_bonificaciones
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Gu%C3%ADa_de_proyectos_exitosos
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Lineamientos_m%C3%ADnimos_de_calidad
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Manual_del_usuario
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Preguntas_Frecuentes
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Registro
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Reporte_incidencias
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Revisi%C3%B3n_de_componentes
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nueva_comunidad
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nuevo_proyecto
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/T%C3%A9rminos_y_Condiciones
> inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Unirse_a_un_proyecto
> outlinks : http://blogs.prod.uci.cu/BlackHat
> outlinks : http://blogs.prod.uci.cu/DevGirls
> outlinks : http://blogs.prod.uci.cu/codeninja
> outlinks : http://blogs.prod.uci.cu/desarrolloSOA/
> outlinks : http://blogs.prod.uci.cu/echo
> outlinks : http://blogs.prod.uci.cu/gestionproyecto/
> outlinks : http://blogs.prod.uci.cu/humanOS
> outlinks : http://blogs.prod.uci.cu/micro
> outlinks : http://blogs.prod.uci.cu/nova/
> outlinks : http://coj.uci.cu/general/about.xhtml
> outlinks : http://pgs.soporte.uci.cu
> outlinks : http://portal.albet.prod.uci.cu
> outlinks : http://portal.calisoft.prod.uci.cu
> outlinks : http://portal.cdae.prod.uci.cu
> outlinks : http://portal.cedin.prod.uci.cu
> outlinks : http://portal.cegel.prod.uci.cu
> outlinks : http://portal.ceige.prod.uci.cu
> outlinks : http://portal.cenia.prod.uci.cu
> outlinks : http://portal.cesim.prod.uci.cu
> outlinks : http://portal.cice.prod.uci.cu
> outlinks : http://portal.cidi.prod.uci.cu
> outlinks : http://portal.cised.prod.uci.cu
> outlinks : http://portal.datec.prod.uci.cu
> outlinks : http://portal.dgp.prod.uci.cu
> outlinks : http://portal.dt.prod.uci.cu
> outlinks : http://portal.fortes.prod.uci.cu
> outlinks : http://portal.frcav.cav.uci.cu
> outlinks : http://portal.frgrm.grm.uci.cu
> outlinks : http://portal.frhab.hab.uci.cu
> outlinks : http://portal.geitel.prod.uci.cu
> outlinks : http://portal.geysed.prod.uci.cu
> outlinks : http://portal.hlg.uci.cu
> outlinks : http://portal.isec.prod.uci.cu
> outlinks : http://portal.tlm.prod.uci.cu
> outlinks : http://portal.vcl.uci.cu/
> outlinks : http://postgresql.uci.cu
> outlinks : http://www.redmine.org/
> outlinks : http://www.redmine.org/guide
> contentLength : 5280
>
> and this is the page code that i check with firefox.
>
> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
> <head>
> <meta http-equiv="content-type" content="text/html; charset=utf-8" />
> <title>Comunidades UCI</title>
> ....continue
>
>
>
> I need to replace index-more.jar plugin ?
>
>
>
>
> ----- Mensaje original -----
> De: "Markus Jelsma" <ma...@openindex.io>
> Para: user@nutch.apache.org
> Enviados: Martes, 27 de Noviembre 2012 15:33:20
> Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi - are you sure you have tabs separating the target and the mapped mimes? Use the nutch indexchecker tool to quickly test if it works.
>
> -----Original message-----
> > From:Eyeris Rodriguez Rueda <er...@uci.cu>
> > Sent: Tue 27-Nov-2012 21:18
> > To: user@nutch.apache.org
> > Subject: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
> >
> > Hi. Markus.
> > I was doing your recommendations but, my problem persist, some documents still with application/xhtml+xml instead of text/html.
> > I add the property to nutch-site.xml and make the conf/contenttype-mapping.txt file
> > <property>
> > <name>moreIndexingFilter.mapMimeTypes</name>
> > <value>true</value>
> > </property>
> >
> > I'm using nutch 1.5.1. Tell me if I need to replace index-more.jar in plugin directory with any fixed version ?
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci
>
Re: problem with text/html content type of documents appears application/xhtml+xml in solr index
Posted by Eyeris Rodriguez Rueda <er...@uci.cu>.
Hi.
I have checked the separation with tab in contenttype-mapping.txt but is ok.
This is my console out with index checker tool
bin/nutch indexchecker http://comunidades.uci.cu/
fetching: http://comunidades.uci.cu/
parsing: http://comunidades.uci.cu/
contentType: application/xhtml+xml
host : comunidades.uci.cu
custom-h1 : Comunidades UCI
tstamp : Tue Nov 27 16:10:47 CST 2012
custom-strong : GESPRO
custom-strong : Personalice su página según sus necesidades y preferencias
date : Tue Nov 27 16:10:47 CST 2012
type : application/xhtml+xml
type : application
type : xhtml+xml
lang : gl
url : http://comunidades.uci.cu/
content : Comunidades UCI Sign in Register Home Projects Help Search : Comunidades UCI DevGirls BlackHat coden
title : Comunidades UCI
inlinks : #Ayuda
inlinks : #Cómoen-Comunidades-UCI
inlinks : #Documéntese
inlinks : /
inlinks : /account/register
inlinks : /gespro_help
inlinks : /links
inlinks : /login
inlinks : /news
inlinks : /news/76
inlinks : /news/77
inlinks : /news/78
inlinks : /news/79
inlinks : /news/80
inlinks : /projects
inlinks : /projects/coj
inlinks : /projects/comunidad
inlinks : /projects/kgetmms
inlinks : /projects/konoha
inlinks : /projects/lua
inlinks : /projects/lxkblayoutsetter
inlinks : /projects/lycan-ide
inlinks : /projects/progcomp
inlinks : /projects/sigest
inlinks : /projects/xfce4-globalmenu
inlinks : /search
inlinks : /tab/system_show
inlinks : /users/132
inlinks : /users/440
inlinks : /users/629
inlinks : /users/976
inlinks : http://comunidades.uci.cu
inlinks : http://comunidades.uci.cu/gespro-help/
inlinks : http://comunidades.uci.cu/git/lxkblayoutsetter.git
inlinks : http://comunidades.uci.cu/projects
inlinks : http://comunidades.uci.cu/projects/cised
inlinks : http://comunidades.uci.cu/projects/erlang
inlinks : http://comunidades.uci.cu/projects/java
inlinks : http://comunidades.uci.cu/projects/porycomgobelectronic
inlinks : http://comunidades.uci.cu/projects/progcomp
inlinks : http://comunidades.uci.cu/projects/proycaxtor
inlinks : http://comunidades.uci.cu/projects/proycombd
inlinks : http://comunidades.uci.cu/projects/proycombpmsoa
inlinks : http://comunidades.uci.cu/projects/proycomcedin
inlinks : http://comunidades.uci.cu/projects/proycomcolaborativo
inlinks : http://comunidades.uci.cu/projects/proycommultimedia
inlinks : http://comunidades.uci.cu/projects/proycomprogramacion
inlinks : http://comunidades.uci.cu/projects/proycomsoftlibre
inlinks : http://comunidades.uci.cu/projects/proyphp
inlinks : http://comunidades.uci.cu/projects/ucicom
inlinks : http://comunidades.uci.cu/projects/ucicom/documents
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Acerca_de_Comunidades_UCI
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Autenticación
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/C%C3%B3digo_de_%C3%A9tica
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Contacto
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Cr%C3%A9ditos_y_bonificaciones
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Gu%C3%ADa_de_proyectos_exitosos
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Lineamientos_m%C3%ADnimos_de_calidad
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Manual_del_usuario
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Preguntas_Frecuentes
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Registro
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Reporte_incidencias
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Revisi%C3%B3n_de_componentes
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nueva_comunidad
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Solicitar_nuevo_proyecto
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/T%C3%A9rminos_y_Condiciones
inlinks : http://comunidades.uci.cu/projects/ucicom/wiki/Unirse_a_un_proyecto
outlinks : http://blogs.prod.uci.cu/BlackHat
outlinks : http://blogs.prod.uci.cu/DevGirls
outlinks : http://blogs.prod.uci.cu/codeninja
outlinks : http://blogs.prod.uci.cu/desarrolloSOA/
outlinks : http://blogs.prod.uci.cu/echo
outlinks : http://blogs.prod.uci.cu/gestionproyecto/
outlinks : http://blogs.prod.uci.cu/humanOS
outlinks : http://blogs.prod.uci.cu/micro
outlinks : http://blogs.prod.uci.cu/nova/
outlinks : http://coj.uci.cu/general/about.xhtml
outlinks : http://pgs.soporte.uci.cu
outlinks : http://portal.albet.prod.uci.cu
outlinks : http://portal.calisoft.prod.uci.cu
outlinks : http://portal.cdae.prod.uci.cu
outlinks : http://portal.cedin.prod.uci.cu
outlinks : http://portal.cegel.prod.uci.cu
outlinks : http://portal.ceige.prod.uci.cu
outlinks : http://portal.cenia.prod.uci.cu
outlinks : http://portal.cesim.prod.uci.cu
outlinks : http://portal.cice.prod.uci.cu
outlinks : http://portal.cidi.prod.uci.cu
outlinks : http://portal.cised.prod.uci.cu
outlinks : http://portal.datec.prod.uci.cu
outlinks : http://portal.dgp.prod.uci.cu
outlinks : http://portal.dt.prod.uci.cu
outlinks : http://portal.fortes.prod.uci.cu
outlinks : http://portal.frcav.cav.uci.cu
outlinks : http://portal.frgrm.grm.uci.cu
outlinks : http://portal.frhab.hab.uci.cu
outlinks : http://portal.geitel.prod.uci.cu
outlinks : http://portal.geysed.prod.uci.cu
outlinks : http://portal.hlg.uci.cu
outlinks : http://portal.isec.prod.uci.cu
outlinks : http://portal.tlm.prod.uci.cu
outlinks : http://portal.vcl.uci.cu/
outlinks : http://postgresql.uci.cu
outlinks : http://www.redmine.org/
outlinks : http://www.redmine.org/guide
contentLength : 5280
and this is the page code that i check with firefox.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>Comunidades UCI</title>
....continue
I need to replace index-more.jar plugin ?
----- Mensaje original -----
De: "Markus Jelsma" <ma...@openindex.io>
Para: user@nutch.apache.org
Enviados: Martes, 27 de Noviembre 2012 15:33:20
Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
Hi - are you sure you have tabs separating the target and the mapped mimes? Use the nutch indexchecker tool to quickly test if it works.
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Tue 27-Nov-2012 21:18
> To: user@nutch.apache.org
> Subject: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi. Markus.
> I was doing your recommendations but, my problem persist, some documents still with application/xhtml+xml instead of text/html.
> I add the property to nutch-site.xml and make the conf/contenttype-mapping.txt file
> <property>
> <name>moreIndexingFilter.mapMimeTypes</name>
> <value>true</value>
> </property>
>
> I'm using nutch 1.5.1. Tell me if I need to replace index-more.jar in plugin directory with any fixed version ?
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
RE: problem with text/html content type of documents appears
application/xhtml+xml in solr index
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - are you sure you have tabs separating the target and the mapped mimes? Use the nutch indexchecker tool to quickly test if it works.
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Tue 27-Nov-2012 21:18
> To: user@nutch.apache.org
> Subject: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi. Markus.
> I was doing your recommendations but, my problem persist, some documents still with application/xhtml+xml instead of text/html.
> I add the property to nutch-site.xml and make the conf/contenttype-mapping.txt file
> <property>
> <name>moreIndexingFilter.mapMimeTypes</name>
> <value>true</value>
> </property>
>
> I'm using nutch 1.5.1. Tell me if I need to replace index-more.jar in plugin directory with any fixed version ?
>
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci
>
RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
Posted by Eyeris Rodriguez Rueda <er...@uci.cu>.
Hi. Markus.
I was doing your recommendations but, my problem persist, some documents still with application/xhtml+xml instead of text/html.
I add the property to nutch-site.xml and make the conf/contenttype-mapping.txt file
<property>
<name>moreIndexingFilter.mapMimeTypes</name>
<value>true</value>
</property>
I'm using nutch 1.5.1. Tell me if I need to replace index-more.jar in plugin directory with any fixed version ?
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
Posted by Eyeris Rodriguez Rueda <er...@uci.cu>.
Thanks a lot Markus for your answer. My English is not so good.
I was reading but i don’t know how to fix the problems yet. Could you explain me in details the solution please. I was looking in conf directory but I can't find how to map one mime types to another. I need to replace index-more plugin ?
I was looking in the link that you suggest me and a saw a NUTCH-1262-1.5-1.patch but I don’t know how to use that patch.
Please tell me if I need to delete the index completely or there is a way to replace an application/xhtml+xml to text/html in solr index.
-----Mensaje original-----
De: Markus Jelsma [mailto:markus.jelsma@openindex.io]
Enviado el: domingo, 25 de noviembre de 2012 4:33 AM
Para: user@nutch.apache.org
Asunto: RE: problem with text/html content type of documents appears application/xhtml+xml in solr index
Hi - trunk's more indexing filter can map mime types to any target. With it you can map both (x)html mimes to text/html or to `web page`.
https://issues.apache.org/jira/browse/NUTCH-1262
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Sun 25-Nov-2012 00:48
> To: user@nutch.apache.org
> Subject: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi.
>
> I have changed my nutch version from 1.4 to 1.5.1 and I have detected a problem with content type of some document, some pages with text/html appears in solr index with application/xhtml+xml , when I check the links the navegator tell me that efectively is text/html.
> Any body can help me to fix this problem, I think change this content type manually in solr index to text/html but is not a good way for me.
> Please any suggestion or advice will be accepted.
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
RE: problem with text/html content type of documents appears
application/xhtml+xml in solr index
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - trunk's more indexing filter can map mime types to any target. With it you can map both (x)html mimes to text/html or to `web page`.
https://issues.apache.org/jira/browse/NUTCH-1262
-----Original message-----
> From:Eyeris Rodriguez Rueda <er...@uci.cu>
> Sent: Sun 25-Nov-2012 00:48
> To: user@nutch.apache.org
> Subject: problem with text/html content type of documents appears application/xhtml+xml in solr index
>
> Hi.
>
> I have changed my nutch version from 1.4 to 1.5.1 and I have detected a problem with content type of some document, some pages with text/html appears in solr index with application/xhtml+xml , when I check the links the navegator tell me that efectively is text/html.
> Any body can help me to fix this problem, I think change this content type manually in solr index to text/html but is not a good way for me.
> Please any suggestion or advice will be accepted.
>
>
> 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
> CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION
>
> http://www.uci.cu
> http://www.facebook.com/universidad.uci
> http://www.flickr.com/photos/universidad_uci
>