You are viewing a plain text version of this content. The canonical link for it is here.
Posted to droids-dev@incubator.apache.org by Tobias Rübner <to...@apache.org> on 2012/12/18 09:43:44 UTC
How to get started
Hi Antonio,
you are right, we definitely need to improve our documentation.
I can see that you still use the code provided in the droids-examples
module.
This module is an example, to show you how to use the project.
Take a look in the
org.apache.droids.examples.TestSimpleDroid#testBasicCrawling test.
This is an example for creating an droids crawler.
You can see a call to
DroidsFactory.createSimpleReportCrawlingDroid(targetURI) which creates a
simple droid.
In this method you can see the basic steps required to set up a droid.
But keep in mind, these are only examples.
To set up your own crawler, for your specific use case, you need to define
your specific components.
We have common components for basic crawling stuff, e.g. a TikaParser for
extracting the content or
a SolrHandler to send the extracted data to a handler.
If these components do not match your requiremnents you can implements your
own, using the provided interfaces.
So for you questions:
1. How to make it work Standalone.
Create your own droid in your own project and start it from within a main
method.
2. How to capture the output.
Use a Handler that fits your requirements.
When you want to search your data try the SolrHandler, when you want to
write your data to log or xml files
take a look at the droids-core module handlers
(org.apache.droids.handle.*).
These are simple handlers which implement the Handler interface, but I
think for your use case
you need to implement it on your own.
PS: Please send mails also to the droids mailing list, so that everyone can
share their opinions.
I also added this email to the droids mailing list.
Hope this helps,
Tobias
On Sun, Dec 16, 2012 at 12:04 PM, Antonio Moreno wrote:
Hi Tobias,
I'm testing with localhost with a simple HTML.
<!DOCTYPE html>
<html>
<body>
<h1>This is heading 1</h1>
<h2>This is heading 2</h2>
<h3>This is heading 3</h3>
<h4>This is heading 4</h4>
<h5>This is heading 5</h5>
<h6>This is heading 6</h6>
</body>
</html>
Apparently the test run.
public void execute_linkIsParsed() throws DroidsException, IOException,
URISyntaxException {
// Arrange
final Link link = new LinkTask(null, new URI("
http://localhost/droids"),
1);
// Act
this.instance.execute(link);
// Assert
Mockito.verify(htmlParser).parse(Matchers.any(ContentEntity.class),
Matchers.any(Link.class));
}
How I can make it work without @ test? and how I can capture a labels in
file XML or log?
documentation is not there?.
thanks Tobias.
Antonio
El 14/12/12 15:08, Tobias Rübner escribió:
Hi Antonio,
the droids-examples module runs some test cases.
For example the TestCrawlingWorker parses the google site
and the TestSimpleDroid creates a local webserver with some webpages to
crawl.
So go to the droids-examples module and run
$ mvn clean test
Then you should see a lot of output, that is the response from the crawls.
You should also have a look into the Droids Implementations in
org.apache.droids.examples and
how they are called in the unit tests.
One last point, you can also have a look into the cleanup branch.
I tried to refactor the API to make it much easier to use.
Maybe you can share your opinion about this on the mailing list.
Tobias
On Fri, Dec 14, 2012 at 2:54 PM, Antonio Moreno wrote:
ok, sorry, i'm testing
https://svn.apache.org/repos/asf/incubator/droids/trunk/
I downloaded it again and apparently installs it.
BUILD SUCCESS
Where I can see the result?
the following is to understand the code, to make crawler a other web, my
intention is to contribute to a future code droids.
Thank.
El 14/12/12 14:29, Tobias Rübner escribió:
Hi Antonio,
Sorry, but I'm not sure what you want to do.
So I have some questions:
1. What version of droids have you used?
the trunk (https://svn.apache.org/repos/asf/incubator/droids/trunk/)
or the cleanup branch (
https://svn.apache.org/repos/asf/incubator/droids/branches/0.2.x-cleanup/)
Currently the examples are only working in trunk!
2. What are you trying to do?
When implementing an application with droids you typically set up your
own application.
Do not change the code in droids, unless you want to contribute.
So please start with checking out the trunk and then build droids:
$ svn co https://svn.apache.org/repos/asf/incubator/droids/trunk/apache-droids
$ cd apache-droids
$ mvn clean install
The cleanup branch is currently in development and
maybe some parts of it will be used in the next droids release.
Tobias
On Fri, Dec 14, 2012 at 2:05 PM, Antonio Moreno wrote:
Hello again Tobias,
I made the changes and it shows me this error:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile
(default-compile) on project droids-examples: Compilation failure:
Compilation failure:
[ERROR]
/home/name/src/droids/droids-examples/src/main/java/org/apache/droids/examples
I do really, speaking is the droid-example folder:
$ mvn clean install
These are the units I have in the pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.droids</groupId>
<artifactId>droids-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.droids</groupId>
<artifactId>droids-tika</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.8.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.droids</groupId>
<artifactId>droids-core</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.6.4</version>
</dependency>
<dependency>
<groupId>org.apache.droids</groupId>
<artifactId>droids-core</artifactId>
<version>0.2-incubating</version>
</dependency>
</dependencies>
Thank for the help.
El 14/12/12 13:31, Tobias Rübner escribió:
Hi Antonio,
that is exactly the problem why I created this branch.
Using the current trunk, it is really hard to get started with droids.
The documentation is currently a little bit outdated, but we are working on
it.
A simple example for using droids can be found here:
http://people.apache.org/~tobr/droids/getting-started.html
If you want to get a little bit deeper check out the project from svn
svn co https://svn.apache.org/repos/asf/incubator/droids/trunk
and take a look into the droids-examples module.
For developing applications with droids, I would recommend you to use the
deployed maven artifacts.
Start a new maven project and add the droids dependencies.
<dependency>
<groupId>org.apache.droids</groupId>
<artifactId>droids-core</artifactId>
<version>0.2-incubating</version>
</dependency>
Hope this helps,
Tobias
On Fri, Dec 14, 2012 at 1:08 PM, Antonio Moreno wrote:
hi Tobias,
I am a beginner in programming, and I'm interested in trying Droids, but
not how to start a simple example, there is little documentation the droid.
As I can boot your example with MVN?. Thank you.