You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by Apache Wiki <wi...@apache.org> on 2017/02/21 01:51:09 UTC
[Tika Wiki] Update of "ApacheTikaHtmlEncodingStudy" by TimothyAllison
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Tika Wiki" for change notification.
The "ApacheTikaHtmlEncodingStudy" page has been changed by TimothyAllison:
https://wiki.apache.org/tika/ApacheTikaHtmlEncodingStudy
New page:
= Apache Tika's Html Encoding Study =
In support of [https://issues.apache.org/jira/browse/TIKA-2038|TIKA-2038], we gathered a new subset of html pages from CC-MAIN-2017-04.
This page offers a first rough draft of the process.
1. Determined which top level domains (TLDs) were of interest
2. Downloaded the 300 index files from Common Crawl
3. Counted the number of pages per TLD that had "html/text" in the http Content-Type header
4. Created sampling frequencies per TLD, with a target of 50k per TLD, with the exception of 100k for ".com"
5. Randomly sampled according to the sampling frequencies per TLD from the 300 index files
Map:
{{{
java -cp cc-extractor-0.0.1.jar org.tallison.cc.index.CCIndexBatchReader
10 /data1/commoncrawl_indices/CC-MAIN-2017-04/ DownSample
tld_mimes.txt tld_mimes_down_sampled
}}}
Reduce:
{{{
java -cp cc-extractor-0.0.1.jar org.tallison.cc.index.reducers.ConcatReducer
tld_mimes_down_sampled tld_mimes_down_sampled_index
}}}
6.#6 Pulled the data from Common Crawl
{{{
java -cp cc-extractor-0.0.1.jar org.tallison.cc.CCGetter
tld_mimes_down_sampled_index /data4/docs/commoncrawl_html_study
cc_html_study_crawl_status.txt
}}}