You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stormcrawler.apache.org by jn...@apache.org on 2024/05/17 09:19:25 UTC
(incubator-stormcrawler-site) branch 29 created (now 30090f2)
This is an automated email from the ASF dual-hosted git repository.
jnioche pushed a change to branch 29
in repository https://gitbox.apache.org/repos/asf/incubator-stormcrawler-site.git
at 30090f2 Fix link to Javadocs and content of getting started, fix #29
This branch includes the following new commits:
new 30090f2 Fix link to Javadocs and content of getting started, fix #29
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(incubator-stormcrawler-site) 01/01: Fix link to Javadocs and content of getting started, fix #29
Posted by jn...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jnioche pushed a commit to branch 29
in repository https://gitbox.apache.org/repos/asf/incubator-stormcrawler-site.git
commit 30090f20b9ff5492e15af3960bf4b68a7ef8d3a0
Author: Julien Nioche <ju...@digitalpebble.com>
AuthorDate: Fri May 17 10:19:14 2024 +0100
Fix link to Javadocs and content of getting started, fix #29
Signed-off-by: Julien Nioche <ju...@digitalpebble.com>
---
_includes/header.html | 2 +-
getting-started/index.html | 17 ++++++-----------
2 files changed, 7 insertions(+), 12 deletions(-)
diff --git a/_includes/header.html b/_includes/header.html
index 1440c4b..a8dfe16 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -11,7 +11,7 @@
<li><a href="{{ site.baseurl }}/download/index.html">Download</a>
<li><a href="https://github.com/apache/incubator-stormcrawler">Source Code</a></li>
<li><a href="{{ site.baseurl }}/getting-started/">Getting Started</a></li>
- <li><a href="https://javadoc.io/doc/com.digitalpebble.stormcrawler/storm-crawler-core/latest/index.html">JavaDocs</a>
+ <li><a href="https://javadoc.io/doc/org.apache.stormcrawler/stormcrawler-core/3.0/index.html">JavaDocs</a>
<li><a href="{{ site.baseurl }}/faq/">FAQ</a></li>
<li><a href="{{ site.baseurl }}/support/">Support</a></li>
</ul>
diff --git a/getting-started/index.html b/getting-started/index.html
index fe328d4..8fbf831 100644
--- a/getting-started/index.html
+++ b/getting-started/index.html
@@ -8,23 +8,23 @@ title: Getting started with Apache StormCrawler (Incubating)
<h1>Quickstart</h1>
<br>
<p>NOTE: These instructions assume that you have <a href="https://maven.apache.org/install.html">Apache Maven®</a> installed.
- You will also need to install <a href="https://storm.apache.org/">Apache Storm®</a> to run the crawler. The version of Storm to use must match the one defined in the pom.xml file of your topology. The major version of Apache StormCrawler (Incubating) mirrors the one from Apache Storm®, i.e whereas StormCrawler 1.x used Storm 1.2.3, the current version now requires Storm 2.6.0. Our <a href="https://github.com/DigitalPebble/ansible-storm">Ansible-Storm</a> repository contains resources t [...]
+ You will also need to install <a href="https://storm.apache.org/">Apache Storm® 2.6.2</a> to run the crawler.</p>
<p>Once Apache Storm® is installed, the easiest way to get started is to generate a brand new Apache StormCrawler (Incubating) project using:</p>
- <p><i>mvn archetype:generate -DarchetypeGroupId=com.digitalpebble.stormcrawler -DarchetypeArtifactId=storm-crawler-archetype -DarchetypeVersion=2.11</i></p>
+ <p><i>mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=3.0</i></p>
<p>You'll be asked to enter a groupId (e.g. <i>com.mycompany.crawler</i>), an artefactId (e.g. <i>stormcrawler</i>), a version, a package name and details about the user agent to use.</p>
<p>This will not only create a fully formed project containing a POM with the dependency above but also the default resource files, a default CrawlTopology class and a configuration file. Enter the directory you just created (should be the same as the artefactId you specified earlier e.g. stormcrawler) and follow the instructions on the README file.</p>
- <p>Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from <a href="https://github.com/DigitalPebble/storm-crawler/tree/master/archetype/src/main/resources/archetype-resources">archetype-resources</a>.</p>
+ <p>Alternatively if you can't or don't want to use the Maven archetype above, you can simply copy the files from <a href="https://github.com/apache/incubator-stormcrawler/tree/master/archetype/src/main/resources/archetype-resources">archetype-resources</a>.</p>
- <p>Have a look at the code of the <a href="https://github.com/DigitalPebble/storm-crawler/blob/master/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java">CrawlTopology class</a>, the crawler-conf.yaml file as well as the files in 'src/main/resources/', they are all that is needed to run a crawl topology : all the other components come from the core module.</p>
+ <p>Have a look at the code of the <a href="https://github.com/apache/incubator-stormcrawler/blob/master/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java">CrawlTopology class</a>, the <a href="https://github.com/apache/incubator-stormcrawler/blob/master/archetype/src/main/resources/archetype-resources/crawler-conf.yaml">crawler-conf.yaml</a> file as well as the files in <a href="https://github.com/apache/incubator-stormcrawler/tree/master/archetype/src/m [...]
<p>What this CrawlTopology does is very simple : it gets URLs to crawl from a <a href="https://urlfrontier.net">URLFrontier</a> instance and emits them on the topology. These URLs are then partitioned by hostname to enfore the politeness and then fetched. The next bolt (SiteMapParserBolt) checks whether they are sitemap files and if not passes them on to a HTML parser. The parser extracts the text from the document and passes it to a dummy indexer which simply prints a representation of [...]
- <p>Of course this topology is very primitive and its purpose is merely to give you an idea of how Apache StormCrawler (Incubating) works. In reality, you'd use a different spout and index the documents to a proper backend. Look at the <a href="https://github.com/DigitalPebble/storm-crawler/tree/master/external">external modules</a> to see what's already available. Another limitation of this topology is that it will work in local mode only or on a single worker.</p>
+ <p>Of course this topology is very primitive and its purpose is merely to give you an idea of how Apache StormCrawler (Incubating) works. In reality, you'd use a different spout and index the documents to a proper backend. Look at the <a href="https://github.com/apache/incubator-stormcrawler/blob/master/external">external modules</a> to see what's already available. Another limitation of this topology is that it will work in local mode only or on a single worker.</p>
<p>You can run the topology in local mode with :</p>
@@ -32,12 +32,7 @@ title: Getting started with Apache StormCrawler (Incubating)
<br>
- <p>The <a href="https://github.com/DigitalPebble/storm-crawler/wiki">WIKI</a> pages contain useful information on the components and configuration and should help you going further.</p>
-
- <br>
-
- <p>If you want to use Apache StormCrawler (Incubating) with Elasticsearch, the tutorial below should be a good starting point.</p>
- <iframe width="840" height="472" src="https://www.youtube.com/embed/8kpJLPdhvLw" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+ <p>The <a href="https://github.com/apache/incubator-stormcrawler/wiki">WIKI</a> pages contain useful information on the components and configuration and should help you going further.</p>
<br>