You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by Apache Wiki <wi...@apache.org> on 2017/02/21 01:51:09 UTC

[Tika Wiki] Update of "ApacheTikaHtmlEncodingStudy" by TimothyAllison

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Tika Wiki" for change notification.

The "ApacheTikaHtmlEncodingStudy" page has been changed by TimothyAllison:
https://wiki.apache.org/tika/ApacheTikaHtmlEncodingStudy

New page:
= Apache Tika's Html Encoding Study =
In support of [https://issues.apache.org/jira/browse/TIKA-2038|TIKA-2038], we gathered a new subset of html pages from CC-MAIN-2017-04.

This page offers a first rough draft of the process.

 1. Determined which top level domains (TLDs) were of interest
 2. Downloaded the 300 index files from Common Crawl
 3. Counted the number of pages per TLD that had "html/text" in the http Content-Type header
 4. Created sampling frequencies per TLD, with a target of 50k per TLD, with the exception of 100k for ".com"  
 5. Randomly sampled according to the sampling frequencies per TLD from the 300 index files

    Map:
    {{{
       java -cp cc-extractor-0.0.1.jar org.tallison.cc.index.CCIndexBatchReader 
           10 /data1/commoncrawl_indices/CC-MAIN-2017-04/ DownSample 
           tld_mimes.txt tld_mimes_down_sampled
    }}}
    
    Reduce:
    {{{
       java -cp cc-extractor-0.0.1.jar org.tallison.cc.index.reducers.ConcatReducer 
           tld_mimes_down_sampled tld_mimes_down_sampled_index
    }}}

 6.#6 Pulled the data from Common Crawl
    {{{
       java -cp cc-extractor-0.0.1.jar org.tallison.cc.CCGetter 
           tld_mimes_down_sampled_index /data4/docs/commoncrawl_html_study 
           cc_html_study_crawl_status.txt
    }}}