You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by thammegowda <gi...@git.apache.org> on 2016/04/14 11:44:51 UTC
[GitHub] nutch pull request: NUTCH-2250 : CommonCrawlDumper : Invalid forma...
GitHub user thammegowda opened a pull request:
https://github.com/apache/nutch/pull/103
NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
### Summary
+ Reads all parts of segments
+ writes single document per dump file
### What was the fix?
1. Lists all files insides segments and selects all the parts matching to pattern part-[0-9]{5} under the content directory.
2. CommonCrawlFormat implementation for "JACKSON" type does not clear state, instead it keeps appending documents. This fix uses a new instance for each document (actually it was hard to resolve the state issue because of too many options such as compressing and archiving).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thammegowda/nutch NUTCH-2250
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #103
----
commit 47cc4e27a41a78753ed431685010da1987bca269
Author: Thamme Gowda <tg...@gmail.com>
Date: 2016-04-14T09:38:30Z
NUTCH-2250 : CommonCrawlDumper : Invalid format and skipped parts
+ Reads all parts of segements
+ FIX : writes only one document to dump file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request: NUTCH-2250 : CommonCrawlDumper : Invalid forma...
Posted by thammegowda <gi...@git.apache.org>.
Github user thammegowda commented on a diff in the pull request:
https://github.com/apache/nutch/pull/103#discussion_r59827773
--- Diff: src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java ---
@@ -368,7 +346,11 @@ public boolean accept(File file) {
String mimeType = new Tika().detect(content.getContent());
// Maps file to JSON-based structure
- jsonData = format.getJsonData(url, content, metadata);
+ //TODO: Make this Jackson Format implementation reusable
--- End diff --
Yes, [NUTCH-2251](https://issues.apache.org/jira/browse/NUTCH-2251)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request: NUTCH-2250 : CommonCrawlDumper : Invalid forma...
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/nutch/pull/103#discussion_r59822916
--- Diff: src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java ---
@@ -368,7 +346,11 @@ public boolean accept(File file) {
String mimeType = new Tika().detect(content.getContent());
// Maps file to JSON-based structure
- jsonData = format.getJsonData(url, content, metadata);
+ //TODO: Make this Jackson Format implementation reusable
--- End diff --
Can you log a Jira issue for this please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nutch pull request: NUTCH-2250 : CommonCrawlDumper : Invalid forma...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nutch/pull/103
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---