You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2011/02/02 18:03:35 UTC
[jira] Created: (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Root-relative paths without leading / do not resolve properly
-------------------------------------------------------------
Key: CONNECTORS-157
URL: https://issues.apache.org/jira/browse/CONNECTORS-157
Project: ManifoldCF
Issue Type: Bug
Components: Web connector
Affects Versions: ManifoldCF 0.1
Reporter: Karl Wright
If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright updated CONNECTORS-157:
-----------------------------------
Component/s: RSS connector
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: RSS connector, Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2, ManifoldCF 0.3
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "David Broadfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087000#comment-13087000 ]
David Broadfoot commented on CONNECTORS-157:
--------------------------------------------
Works like a charm. Many thanks!
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: RSS connector, Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2, ManifoldCF 0.3
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086058#comment-13086058 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
r1158475 to resolve the latest problem.
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2, ManifoldCF 0.3
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-157.
------------------------------------
Resolution: Fixed
Fix Version/s: ManifoldCF 0.3
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.3, ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990061#comment-12990061 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
After consultation, updated the fix to more narrowly target the problem case. r1066781.
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085171#comment-13085171 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
What is the url of the page with the link? Is it "http://mysare.sare.org/MySareF?do=searchProj&q=*&searchmethod=and®ion=&state=&projType=0&sortby=1&page=1"? Because, if so, I don't see where /ProjectReport.aspx is supposed to be coming from.
The relative URL composition rules in Java in general adhere to the w3c specifications. The problem is often that browsers do somewhat different things than the w3c spec. So let's work with your specific case and figure out what's happening. The only two inputs are: (a) the url of the page, and (b) the relative url of the reference. Since I don't see /ProjectReport.aspx in either one, it must either be in the page URL, or there must be a redirection taking place.
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "David Broadfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085949#comment-13085949 ]
David Broadfoot commented on CONNECTORS-157:
--------------------------------------------
My bad - I miss-pasted the seed url:
http://mysare.sare.org/MySare/ProjectReport.aspx?do=searchProj&q=*&searchmethod=and®ion=&state=&projType=0&sortby=1&page=1
That should make more sense :)
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086059#comment-13086059 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
r1158476 to port the same fixes to the RSS connector.
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: RSS connector, Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2, ManifoldCF 0.3
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright resolved CONNECTORS-157.
------------------------------------
Resolution: Fixed
Fix Version/s: ManifoldCF next
r1066559.
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "David Broadfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085161#comment-13085161 ]
David Broadfoot commented on CONNECTORS-157:
--------------------------------------------
Hi Karl - I've recently run into a very similar problem
I have a job with this page as a seed:
http://mysare.sare.org/MySareF?do=searchProj&q=*&searchmethod=and®ion=&state=&projType=0&sortby=1&page=1
and i want to crawl / ingest pages with
do=viewRept
in the urls. The links in the html are like this:
<a href="?do=viewRept&pn=LNE03-182&y=2004&t=0">2004 Annual Report
When I check the simple history, I see lines like the following in the identifier column :
http://mysare.sare.org/MySare/?do=viewRept&pn=LNC04-240&t=0&y=2006
ie the /ProjectReport.aspx part is being omited from the crawled url.
Any idea what is going on here?
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright reassigned CONNECTORS-157:
--------------------------------------
Assignee: Karl Wright
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF next
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085959#comment-13085959 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
I can see what the problem is right away. The relative urls are not legit according to w3c. It looks like I will need to replace java's implementation of url with my own in order to make such a thing work. I will experiment and get back to you.
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright updated CONNECTORS-157:
-----------------------------------
Description:
If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
Another similar case is the following:
"http://foo.com/bar/xyz.asmx?q=hello"
... with a relative URL of "q=there"
... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
This is apparently a bug, but we need to find a way to work around it properly.
was:If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
Summary: Some relative paths without leading / do not resolve properly (was: Root-relative paths without leading / do not resolve properly)
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-157) Some relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086062#comment-13086062 ]
Karl Wright commented on CONNECTORS-157:
----------------------------------------
r1158480 to add similar tests to the RSS connector.
> Some relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: RSS connector, Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2, ManifoldCF 0.3
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf".
> Another similar case is the following:
> "http://foo.com/bar/xyz.asmx?q=hello"
> ... with a relative URL of "q=there"
> ... produces: "http://foo.com/bar/?q=there", incorrectly losing the last part of the path.
> This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CONNECTORS-157) Root-relative paths without
leading / do not resolve properly
Posted by "Karl Wright (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CONNECTORS-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karl Wright reopened CONNECTORS-157:
------------------------------------
Going to make this ticket more broad
> Root-relative paths without leading / do not resolve properly
> -------------------------------------------------------------
>
> Key: CONNECTORS-157
> URL: https://issues.apache.org/jira/browse/CONNECTORS-157
> Project: ManifoldCF
> Issue Type: Bug
> Components: Web connector
> Affects Versions: ManifoldCF 0.1
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.2
>
>
> If a document has a URL which is just the domain, e.g. "http://foo.com", the java.net.URI class fails to resolve URLs in that document which have no starting "/", e.g. "document.pdf". The resolved URI has no path part, e.g. "http://foo.comdocument.pdf". This is apparently a bug, but we need to find a way to work around it properly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira