You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by Ferdinand Soethe <sa...@soethe.net> on 2005/06/04 09:48:56 UTC
Add support for Googles sitemap protocol?
This seems to makes a lot of sense in terms of better indexing and
doesn't sound too complex to implement if we have a url-encoder
somewhere.
Although we might want to use this
http://www.openarchives.org/OAI/openarchivesprotocol.html protocol
which is an open standard and will be accepted by Google as well.
The tricky part - though more of a nice to have - is to automate
reporting of changes to the google sitemap.
wdyt
--
Ferdinand Soethe
Re: Add support for Googles sitemap protocol?
Posted by Ross Gardler <rg...@apache.org>.
Ross Gardler wrote:
> Ferdinand Soethe wrote:
>
>>
>>
>>
>>> The only requried information in the sitemap is the URL. This means we
>>> can create the Google sitemap now, with minimal effort. Over time we can
>>> enhance this by adding further meta-data once it becomes available.
>>
>>
>>
>> I'm happy to go for the Google format, I just thought that our
>> commitment to standards would tip the balance towards OAI. Does OAI
>> not allow for a minimal form like Googles'?
>
>
> Good point. However, I don't think OAI has a "minimal" form, I did some
> preliminary research into it a few months ago. Let me check it out, I'll
> report back.
OK, I've rechecked the schema and the OAI-PMH schema requires that a
date be provided in headers for each record (which coresponds to a
documet in our case). This means we need the last modified date
otherwise our data will be inaccurate, that is using last-published will
make harvesters think that the file was modified on that date.
see
http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm#OAIPMHschema
Furthermore, as I mentioned in a previous mail, OAI is not about
indexing documents (like the Google sitemap) it is about indexing
meta-data. To use OAI-PMH for anything worthwdile requires quite a bit
of additional functionality like complete meta-data etc. I *do* want to
see this included but do not believe it is appropriate to the vast
majority of our present users.
I think we should proceed with the simpler case of Google and return to
OAI wen we have a meta-data system in place.
Ross
Re: Add support for Googles sitemap protocol?
Posted by Ross Gardler <rg...@apache.org>.
Ferdinand Soethe wrote:
>
>
>
>>The only requried information in the sitemap is the URL. This means we
>>can create the Google sitemap now, with minimal effort. Over time we can
>>enhance this by adding further meta-data once it becomes available.
>
>
> I'm happy to go for the Google format, I just thought that our
> commitment to standards would tip the balance towards OAI. Does OAI
> not allow for a minimal form like Googles'?
Good point. However, I don't think OAI has a "minimal" form, I did some
preliminary research into it a few months ago. Let me check it out, I'll
report back.
However, I'd still like to see support for Google sitemaps since we can
do it very quickly and it is more "approachable" than OAI since everyone
knows Google.
> If we go for the Google format, I'd like to suggest to use slightly
> more than the minimum format in this form (as documented in
> https://www.google.com/webmasters/sitemaps/docs/en/protocol.html)
>
> <?xml version="1.0" encoding="UTF-8"?>
> <urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
> <url>
> <loc>http://www.yoursite.com/catalog?item=83&desc=vacation_usa</loc>
> <lastmod>2004-11-23</lastmod>
> </url>
> </urlset>
>
> and include the 'lastmod' right away as that would be the key to speedy
> updates. Can we do that?
No. At least at present, unless you experiments with the cache-key
thingy show otherwise. At present all files are rebuilt regardless of
whether they have changed. The need for keeping track of last modified
files is something that comes up fairly regularly. We all want it, but
nowone has had a sufficient itch yet.
I'd recomend getting the minimal done, then looking at a way of getting
the lastmod as well.
I'd like to see us being able to create meta-data with last-mod, likely
change frequency etc. the itch is getting stronger, but it is not stong
enough (for me personally) just yet.
> Did you see that Google wants the urls to be url encoded? Does our
> XSLT-engine have a function for that?
http://www.exslt.org/str/functions/encode-uri/index.html
NOTE the code on the above side is in the public domain we *cannot*
relicense it as ASF, so the best thing to do is to write our own version
of the function using replace(string,pattern,replace), the above is
useful for quick testing, but please don't commit it to our CVS.
The replace function returns a string that is created by replacing the
given pattern with the replace argument
Example: replace("Bella Italia", "l", "*")
Result: 'Be**a Ita*ia'
Example: replace("Bella Italia", "l", "")
Result: 'Bea Itaia'
Ross
Re: Add support for Googles sitemap protocol?
Posted by Ferdinand Soethe <sa...@soethe.net>.
> The only requried information in the sitemap is the URL. This means we
> can create the Google sitemap now, with minimal effort. Over time we can
> enhance this by adding further meta-data once it becomes available.
I'm happy to go for the Google format, I just thought that our
commitment to standards would tip the balance towards OAI. Does OAI
not allow for a minimal form like Googles'?
If we go for the Google format, I'd like to suggest to use slightly
more than the minimum format in this form (as documented in
https://www.google.com/webmasters/sitemaps/docs/en/protocol.html)
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">
<url>
<loc>http://www.yoursite.com/catalog?item=83&desc=vacation_usa</loc>
<lastmod>2004-11-23</lastmod>
</url>
</urlset>
and include the 'lastmod' right away as that would be the key to speedy
updates. Can we do that?
Did you see that Google wants the urls to be url encoded? Does our
XSLT-engine have a function for that?
--
Ferdinand Soethe
Re: Add support for Googles sitemap protocol?
Posted by Ross Gardler <rg...@apache.org>.
Gregor J. Rothfuss wrote:
> Ross Gardler wrote:
>
>> Ferdinand Soethe wrote:
>>
>>> This seems to makes a lot of sense in terms of better indexing and
>>> doesn't sound too complex to implement if we have a url-encoder
>>> somewhere.
>
>
> this makes a ton of sense.
Absolutely, now I have read what they are wonder why I didn't know about
them before.
>> Can you provide an overview of what is required with a link to more
>> info. I have no idea what this is.
>
>
> https://www.google.com/webmasters/sitemaps/stats
>
> basically a way to alert google and other search engines to all the
> pages in your site, as originally proposed in 1997!
The only requried information in the sitemap is the URL. This means we
can create the Google sitemap now, with minimal effort. Over time we can
enhance this by adding further meta-data once it becomes available.
The submitting of the sitemap should not be a part of Forrest itself.
Although the Forrestbot would benefit from being able to submit it.
+1 for creating a Google sitemap, I guess the easiest method will be to
adapt the existing linkmap.html file.
Ross
Re: Add support for Googles sitemap protocol?
Posted by "Gregor J. Rothfuss" <gr...@apache.org>.
Ross Gardler wrote:
> Ferdinand Soethe wrote:
>
>> This seems to makes a lot of sense in terms of better indexing and
>> doesn't sound too complex to implement if we have a url-encoder
>> somewhere.
this makes a ton of sense.
> Can you provide an overview of what is required with a link to more
> info. I have no idea what this is.
https://www.google.com/webmasters/sitemaps/stats
basically a way to alert google and other search engines to all the
pages in your site, as originally proposed in 1997!
http://davenet.scripting.com/1997/12/15/realWorldXml
(this is the old dave winer, at his best, without all the ranting :)
>> The tricky part - though more of a nice to have - is to automate
>> reporting of changes to the google sitemap.
you don't have to do that. the crawler will visit this special sitemap
quite frequently ;)
-gregor
Re: Add support for Googles sitemap protocol?
Posted by Ross Gardler <rg...@apache.org>.
Ferdinand Soethe wrote:
> This seems to makes a lot of sense in terms of better indexing and
> doesn't sound too complex to implement if we have a url-encoder
> somewhere.
Can you provide an overview of what is required with a link to more
info. I have no idea what this is.
> Although we might want to use this
> http://www.openarchives.org/OAI/openarchivesprotocol.html protocol
> which is an open standard and will be accepted by Google as well.
This is on the todo list over at Burrokeet. OAI is very important within
the research community and is growing in importance elsewhere. However,
to provide meaningful data via OAI we need extensive meta-data about the
documents available. OAI is not really about indexing web sites, it is
more about indexing and things like white-papers, research papers and
the like.
I'm not sure Forrest is ready for this yet.
> The tricky part - though more of a nice to have - is to automate
> reporting of changes to the google sitemap.
That depends on what you mean "automate reporting of changes". Do you
mean notification that a file has changed?
Without knowing anything about what the "Google sitemap" is, I am
guessing that the notification of changed files wouldn't be too hard, at
least for files in a standard file system (as opposed to a repository)
as we could use the directory generator to create a list of recently
modified files.
It would be even better if we had some way of collecting meta data.
Something that Forrest doesn't do yet (although there is the start of
some infrastructure for that).
If "reporting of changes" means providing diff that is not the job of
Forrest. That is the job of the repository. We will get this kind of
feature with the development of plugins for CMS/repositories.
Ross