You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by le...@apache.org on 2020/02/20 18:42:03 UTC
[incubator-datasketches-website] 03/03: Interim update,
Fix broken links, some reorganization.
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-datasketches-website.git
commit c1b4bbd01803972ed9f97d328c6c36d774daf316
Author: Lee Rhodes <le...@users.noreply.github.com>
AuthorDate: Thu Feb 20 10:41:43 2020 -0800
Interim update, Fix broken links, some reorganization.
---
README.md | 21 +---
_includes/page_header.html | 4 +-
_includes/toc.html | 25 +++-
_layouts/doc_page.html | 2 +-
_layouts/table_page.html | 4 +-
docs/Adaptors.md | 36 ------
docs/Architecture.md | 149 -----------------------
docs/Architecture/Components.md | 4 +-
docs/{ => Architecture}/Concurrency.md | 0
docs/{ => Architecture}/KeyFeatures.md | 16 ++-
docs/{ => Architecture}/OrderSensitivity.md | 0
docs/{ => Architecture}/SketchCriteria.md | 0
docs/Architecture/SketchesByComponent.md | 2 +-
docs/{downloads.md => Community/Downloads.md} | 0
docs/{ => Community}/Research.md | 2 +-
{community => docs/Community}/index.md | 7 +-
docs/Frequency/FrequentItemsOverview.md | 2 +-
docs/HLL/HllSketchVsDruidHyperLogLogCollector.md | 12 +-
docs/LargeScale.md | 6 +-
docs/Quantiles/KllSketchVsTDigest.md | 2 +-
docs/Sampling/ReservoirSamplingPerformance.md | 2 +-
docs/Sampling/VarOptSampling.md | 4 +-
docs/SketchElements.md | 2 +-
docs/Theta/ThetaUpdateSpeed.md | 2 +-
index.md | 29 +----
src/main/resources/docgen/toc.json | 18 ++-
26 files changed, 83 insertions(+), 268 deletions(-)
diff --git a/README.md b/README.md
index 31cf469..d29d24b 100644
--- a/README.md
+++ b/README.md
@@ -24,22 +24,11 @@ Apache DataSketches is an open source, high-performance library of stochastic st
After 8 years of development and 5 years as in Open Source, we have begun the important migration from a stand-alone GitHub site to being a member of the Apache Software Foundation community. While we undergo this migration, we beg your patience.
-* How to contact us:
- * We have two ASF [the-ASF.slack.com](http://the-ASF.slack.com) slack channels:
- * datasketches -- general user questions
- * datasketches-dev -- similar to our Apache [Developers Mail list](https://lists.apache.org/list.html?dev@datasketches.apache.org), except more interactive, but not as easily searchable.
- * For general questions about using the library
- * [Users Mail List](https://lists.apache.org/list.html?users@datasketches.apache.org)
- * For issues about a specific component repository, please open issues on the appropriate GitHub repository listed below.
- * If you wish to contribute to our sketch development please contact us on our developers email list
- * [Developers Mail list](https://lists.apache.org/list.html?dev@datasketches.apache.org)
-
-* Finding our code releases:
- * Apache releases (versions 1.X.X):
- * Zip source: [dist.apache.org/repos/dist/release/incubator/datasketches/](https://dist.apache.org/repos/dist/release/incubator/datasketches/)
- * Jar Artifacts (Java only): [repository.apache.org/#nexus-search;quick~datasketches](https://repository.apache.org/#nexus-search;quick~datasketches)
- * Pre-Apache releases (versions 0.X.X)
- * Jar Artifacts (Java only): [search.maven.org/classic/#search%7Cga%7C1%7Ccom.yahoo.datasketches](https://search.maven.org/classic/#search%7Cga%7C1%7Ccom.yahoo.datasketches)
+* [How to contact us]({{site.docs_dir}}/Community/index.html)
+
+* [Finding our code releases]({{site.docs_dir}}/Community/index.html)
+
+* [Contributing]({{site.docs_dir}}/Community/index.html)
* As the repositories under GitHub.io/DataSketches migrate they will disapear from the the GitHub.com/DataSketches organization page. Please refer to this list be directed to the new locations.
diff --git a/_includes/page_header.html b/_includes/page_header.html
index aa90542..6795f5d 100644
--- a/_includes/page_header.html
+++ b/_includes/page_header.html
@@ -19,7 +19,7 @@
<span class="fa fa-info-circle"></span> OVERVIEW</a>
</li>
<li>
- <a href="/docs/downloads.html">
+ <a href="/docs/Community/Downloads.html">
<span class="fa fa-download"></span> DOWNLOAD</a>
</li>
<li>
@@ -27,7 +27,7 @@
<span class="fa fa-github"></span> GITHUB</a>
</li>
<li>
- <a href="/docs/Research.html">
+ <a href="/docs/Community/Research.html">
<span class="fa fa-paper-plane"></span> RESEARCH</a>
</li>
<li>
diff --git a/_includes/toc.html b/_includes/toc.html
index 2f360ae..6bc3b8d 100644
--- a/_includes/toc.html
+++ b/_includes/toc.html
@@ -10,19 +10,34 @@
<li><a href="{{site.docs_dir}}/MajorSketchFamilies.html">The Major Sketch Families</a></li>
<li><a href="{{site.docs_dir}}/SketchOrigins.html">Sketch Origins</a></li>
<li><a href="{{site.docs_dir}}/SketchElements.html">Sketch Elements</a></li>
- <li><a href="{{site.docs_dir}}/KeyFeatures.html">Key Features</a></li>
<li><a href="{{site.docs_dir}}/LargeScale.html">Large Scale Computing</a></li>
- <li><a href="{{site.docs_dir}}/Architecture.html">Architecture</a></li>
- <li><a href="{{site.docs_dir}}/OrderSensitivity.html">Notes on Order Sensitivity</a></li>
- <li><a href="{{site.docs_dir}}/Concurrency.html">Notes on Concurrency</a></li>
+
+ <p id="architecture">
+ <a data-toggle="collapse" class="menu collapsed" href="#collapse_architecture">Architecture</a>
+ </p>
+ <div class="collapse" id="collapse_architecture">
+ <li><a href="{{site.docs_dir}}/Architecture/KeyFeatures.html">Key Features</a></li>
+ <li><a href="{{site.docs_dir}}/Architecture/Components.html">Components</a></li>
+ <li><a href="{{site.docs_dir}}/Architecture/SketchesByComponent.html">Sketches by Component</a></li>
+ <li><a href="{{site.docs_dir}}/Architecture/SketchesCriteria.html">Sketch Criteria</a></li>
+ <li><a href="{{site.docs_dir}}/Architecture/OrderSensitivity.html">Notes on Order Sensitivity</a></li>
+ <li><a href="{{site.docs_dir}}/Architecture/Concurrency.html">Notes on Concurrency</a></li>
+ </div>
<li><a href="{{site.docs_pdf_dir}}/DataSketches_deck.pdf">Overview Slide Deck</a></li>
</div>
+ <p id="community">
+ <a data-toggle="collapse" class="menu collapsed" href="#collapse_community">Community</a>
+ </p>
+ <div class="collapse" id="collapse_community">
+ <li><a href="{{site.docs_dir}}/Community/index.html">Community</a></li>
+ </div>
+
<p id="research">
<a data-toggle="collapse" class="menu collapsed" href="#collapse_research">Research</a>
</p>
<div class="collapse" id="collapse_research">
- <li><a href="{{site.docs_dir}}/Research.html">Research</a></li>
+ <li><a href="{{site.docs_dir}}/Community/Research.html">Research</a></li>
</div>
<p id="distinct-counting">
diff --git a/_layouts/doc_page.html b/_layouts/doc_page.html
index 80a0eea..3387090 100644
--- a/_layouts/doc_page.html
+++ b/_layouts/doc_page.html
@@ -29,7 +29,7 @@
{% include toc.html %}
</div>
<div class="col-md-9 doc-content">
- <a class="btn btn-default btn-xs visible-xs-inline-block visible-sm-inline-block" href="/docs/toc.html">Table of Contents</a>
+ <a class="btn btn-default btn-xs visible-xs-inline-block visible-sm-inline-block" href="/_includes/toc.html">Table of Contents</a>
{{ content }}
</div>
</div>
diff --git a/_layouts/table_page.html b/_layouts/table_page.html
index 7691fe6..5696abf 100644
--- a/_layouts/table_page.html
+++ b/_layouts/table_page.html
@@ -14,10 +14,10 @@
<h1>{{ page.title }}</h1>
<h3>{{ page.subtitle }}</h3>
<h4>API Snapshots:
- <a href="/api/core/snapshot/apidocs/index.html">Core</a>,
+ <a href="/api/java/snapshot/apidocs/index.html">Java Core</a>,
+ <a href="/api/memory/snapshot/apidocs/index.html">Memory</a>,
<a href="/api/pig/snapshot/apidocs/index.html">Pig</a>,
<a href="/api/hive/snapshot/apidocs/index.html">Hive</a>,
- <a href="/api/misc/snapshot/apidocs/index.html">Misc</a>
</h4>
</div>
</div>
diff --git a/docs/Adaptors.md b/docs/Adaptors.md
deleted file mode 100644
index d2803c2..0000000
--- a/docs/Adaptors.md
+++ /dev/null
@@ -1,36 +0,0 @@
----
-layout: doc_page
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-## Adaptors for Hadoop Pig, Hive and Druid
-
-### Repositories
-* [Pig Repository](https://github.com/DataSketches/sketches-pig)
-* [Hive Repository](https://github.com/DataSketches/sketches-hive)
-* [Druid Adaptors for DataSketches](https://github.com/druid-io/druid/tree/master/extensions-core/datasketches)
-
-### Adaptor Examples
-* [Pig Examples for Theta Sketches]({{site.docs_dir}}/Theta/ThetaPigUDFs.html)
-* [Pig Examples for Tuple Sketches]({{site.docs_dir}}/Tuple/TuplePigUDFs.html)
-* [Hive Examples for Theta Sketches]({{site.docs_dir}}/Theta/ThetaHiveUDFs.html)
-* [Hive Examples for Tuple Sketches]({{site.docs_dir}}/Tuple/TupleHiveUDFs.html)
-
-
-
diff --git a/docs/Architecture.md b/docs/Architecture.md
deleted file mode 100644
index 4600486..0000000
--- a/docs/Architecture.md
+++ /dev/null
@@ -1,149 +0,0 @@
----
-layout: doc_page
----
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-# Sketches by Component
-
-The DataSketches Library is organized into the following repository groups:
-
-## Java
-
-### incubator-datasketches-java
-This repository has the core-java sketching classes, which are leveraged by some of the other repositories.
-This repository has no external dependencies outside of the DataSketches/memory repository, Java and TestNG for unit tests.
-This code is versioned and the latest release can be obtained from
-<a href="https://www.apache.org/dyn/closer.cgi?path=/incubator/datasketches/java">incubator-datasketches-java<a/>.
-
-<b>High-level Repositories Structure</b>
-
-Sketches-core Packages. | Package Description
--------------------------------|---------------------
-org.apache.datasketches | Common functions and utilities
-org.apache.datasketches.cpc | New Unique Counting Sketch with better accuracy per size than HLL
-org.apache.datasketches.fdt | Frequent Distinct Tuples Sketch.
-org.apache.datasketches.frequencies | Frequent Item Sketches, for both longs and generics
-org.apache.datasketches.hash | The 128-bit MurmurHash3 and adaptors
-org.apache.datasketches.hll | Unique counting HLL sketches for both heap and off-heap.
-org.apache.datasketches.hllmap | The (HLL) Unique Count Map Sketch
-org.apache.datasketches.kll | New quantiles sketch with better accuracy per size than the standard quantiles sketch.
-org.apache.datasketches.quantiles | Sketches for quantiles, PMF and CDF functions, both doubles and generics and for heap and off-heap.
-org.apache.datasketches.sampling | Weighted and uniform reservoir sampling with generics
-org.apache.datasketches.theta | Unique counting Theta Sketches for both heap and off-heap
-org.apache.datasketches.tuple | Tuple sketches for both primitives and generics
-org.apache.datasketches.tuple.adouble | A Tuple sketch with a Summary of a single double
-org.apache.datasketches.tuple.aninteger | A Tuple sketch with a Summary of a single integer
-org.apache.datasketches.tuple.Strings | A Tuple sketch with a Summary of an array of Strings
-
-### incubator-datasketches-memory
-This code is versioned and the latest release can be obtained from
-<a href="https://www.apache.org/dyn/closer.cgi?path=/incubator/datasketches/memory">incubator-datasketches-memory<a/>.
-
-Memory Packages | Package Description
--------------------------------|---------------------
-org.apache.datasketches.memory | Low level, high-performance Memory data-structure management primarily for off-heap.
-
-
-### incubator-datasketches-hive
-This repository contains Hive UDFs and UDAFs for use within Hadoop grid enviornments.
-This code has dependencies on sketches-core as well as Hadoop and Hive.
-Users of this code are advised to use Maven to bring in all the required dependencies.
-This code is versioned and the latest release can be obtained from
-<a href="https://www.apache.org/dyn/closer.cgi?path=/incubator/datasketches/hive">incubator-datasketches-hive<a/>.
-
-Sketches-hive Packages | Package Description
--------------------------------------|---------------------
-org.apache.datasketches.hive.cpc | Hive UDF and UDAFs for CPC sketches
-org.apache.datasketches.hive.frequencies | Hive UDF and UDAFs for Frequent Items sketches
-org.apache.datasketches.hive.hll | Hive UDF and UDAFs for HLL sketches
-org.apache.datasketches.hive.kll | Hive UDF and UDAFs for KLL sketches
-org.apache.datasketches.hive.quantiles | Hive UDF and UDAFs for Quantiles sketches
-org.apache.datasketches.hive.theta | Hive UDF and UDAFs for Theta sketches
-org.apache.datasketches.hive.tuple | Hive UDF and UDAFs for Tuple sketches
-
-### incubator-datasketches-pig
-This repository contains Pig User Defined Functions (UDF) for use within Hadoop grid environments.
-This code has dependencies on sketches-core as well as Hadoop and Pig.
-Users of this code are advised to use Maven to bring in all the required dependencies.
-This code is versioned and the latest release can be obtained from
-<a href="https://www.apache.org/dyn/closer.cgi?path=/incubator/datasketches/pig">incubator-datasketches-pig<a/>.
-
-Sketches-pig Packages | Package Description
------------------------------------|---------------------
-org.apache.datasketches.pig.cpc | Pig UDFs for CPC sketches
-org.apache.datasketches.pig.frequencies | Pig UDFs for Frequent Items sketches
-org.apache.datasketches.pig.hash | Pig UDFs for MurmerHash3
-org.apache.datasketches.pig.hll | Pig UDFs for HLL sketches
-org.apache.datasketches.pig.kll | Pig UDFs for KLL sketches
-org.apache.datasketches.pig.quantiles | Pig UDFs for Quantiles sketches
-org.apache.datasketches.pig.sampling. | Pig UDFs for Sampling sketches
-org.apache.datasketches.pig.theta | Pig UDFs for Theta sketches
-org.apache.datasketches.pig.tuple | Pig UDFs for Tuple sketches
-
-
-### incubator-datasketches-characterization
-This relatively new repository is for Java and C++ code that we use to characterize the accuracy and speed performance of the sketches in
-the library and is constantly being updated. Examples of the job command files used for various tests can be found in the src/main/resources directory. Some of these tests can run for hours depending on its configuration.
-
-Characterization Packages | Package Description
-------------------------------------------------|---------------------
-org.apache.datasketches.characterization | Common functions and utilities
-org.apache.datasketches.characterization.concurrent | Concurrent Theta Sketch
-org.apache.datasketches.characterization.cpc | Compressed Probabilistic Counting Sketch
-org.apache.datasketches.characterization.fdt | Frequent Distinct Tuples Sketch
-org.apache.datasketches.characterization.frequencies | Frequent Items Sketches
-org.apache.datasketches.characterization.hash | Hash function performance
-org.apache.datasketches.characterization.hll | HyperLogLog Sketcch
-org.apache.datasketches.characterization.memory | Memory performance
-org.apache.datasketches.characterization.quantiles | Quantiles performance
-org.apache.datasketches.characterization.theta | Theta Sketch
-org.apache.datasketches.characterization.uniquecount | Base Profiles for Unique Counting Sketches
-
-#### C++ Characterizations
-* CPC
-* Frequent Items
-* HLL
-* KLL
-* Theta
-
-
-### incubator-datasketches-vector
-This component implements the [Frequent Directions Algorithm](https://datasketches.apache.org/docs/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on the Memory component.
-
-
-## C++ and Python
-
-### incubator-datasketches-cpp
-This is the evolving C++ implementations of the same sketches that are available in Java.
-These implementations are *binary compatible* with their counterparts in Java.
-In other words, a sketch created and stored in C++ can be opened and read in Java and visa-versa.
-
-This site also has our Python adaptors that basically wrap the C++ implementations,
-making the high performance C++ implementations available from Python.
-
-### incubator-datasketches-postgresql
-This site provides the postgres-specific adaptors that wrap the C++ implementations making
-them available to the PostgreSQL database users. PostgreSQL users should download the PostgreSQL extension from [pgxn.org](https://pgxn.org/dist/datasketches/). For examples refer to the README on the component site.
-
-
-
-
-
-
-
diff --git a/docs/Architecture/Components.md b/docs/Architecture/Components.md
index 7154f44..6586e48 100644
--- a/docs/Architecture/Components.md
+++ b/docs/Architecture/Components.md
@@ -24,7 +24,7 @@ Apapters integrate the core components into the aggregation APIs of specific dat
* **[Apache Pig](https://https://github.com/apache/incubator-datasketches-pig)** (Versioned, Apache Released)
* [Theta Sketch Example]({{site.docs_dir}}/Theta/ThetaPigUDFs.html)
* [Tuple Sketch Example]({{site.docs_dir}}/Tuple/TuplePigUDFs.html)
-* **[Apache Druid](https://github.com/druid-io/druid/tree/master/extensions-core/datasketches)** (Apach Released as part of Druid)
+* **[Apache Druid](https://github.com/apache/druid/tree/master/extensions-core/datasketches)** (Apach Released as part of Druid)
### C++ Adaptors
* **[PostgreSQL](https://github.com/apache/incubator-datasketches-postgresql)** (Versioned, Apache Released)
@@ -34,7 +34,7 @@ them available to the PostgreSQL database users. PostgreSQL users should downloa
## Other Components
* **[Memory](https://github.com/apache/incubator-datasketches-memory):** (Versioned, Apache Released) This is a low-level library that enables fast access to off-heap memory for Java.
* **[Characterization](https://github.com/apache/incubator-datasketches-characterization):** This is a collection of Java and C++ code that we use for long-running studies of accuracy and speed performance over many different parameters. Feel free to run these tests to reproduce many of the graphs and charts you see on our website.
-* **[Vector (Experimental)](https://github.com/apache/incubator-datasketches-vector):** This component implements the [Frequent Directions Algorithm](https://datasketches.apache.org/docs/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on [...]
+* **[Vector (Experimental)](https://github.com/apache/incubator-datasketches-vector):** This component implements the [Frequent Directions Algorithm](https://datasketches.apache.org/docs/Community/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dep [...]
* **[Website](https://github.com/apache/incubator-datasketches-website):** This repository is the home of our website and is constantly being updated with new material.
diff --git a/docs/Concurrency.md b/docs/Architecture/Concurrency.md
similarity index 100%
rename from docs/Concurrency.md
rename to docs/Architecture/Concurrency.md
diff --git a/docs/KeyFeatures.md b/docs/Architecture/KeyFeatures.md
similarity index 90%
rename from docs/KeyFeatures.md
rename to docs/Architecture/KeyFeatures.md
index da405ae..00a0c97 100644
--- a/docs/KeyFeatures.md
+++ b/docs/Architecture/KeyFeatures.md
@@ -22,9 +22,8 @@ layout: doc_page
<h2>Key Features</h2>
<h3>Common Sketch Properties</h3>
- * <b>Single-pass, "one-touch"</b> algorithms enable efficient processing in either real-time or batch.
- * <b>Mergeable</b> algorithms enable parallel processing, which is critical for large systems.
- * <b>Space Sub-linear</b> algorithm not only start small but grow very slowly or not at all as the size of the input stream grows.
+
+ * [Sketch Criteria]({{site.docs_dir}}/Architecture/SketchCriteria.html) for all sketches in the library.
* Query results are <b>approximate</b> but within well defined error bounds that are user
configurable by trading off sketch size with accuracy.
* Designed for <a href="{{site.docs_dir}}/LargeScale.html">Large-scale</a> computing environments
@@ -41,6 +40,7 @@ and are heavily used within Yahoo.
* Extensive documentation with the systems developer in mind.
<h3>Built-In, General Purpose Functions</h3>
+
* General purpose <a href="{{site.docs_dir}}/Memory/MemoryPackage.html">Memory Package</a> for managing data off the Java Heap.
This enables systems designers the ability to manage their own large data heaps with
dedicated processor threads that would otherwise put undue pressure on the Java heap and
@@ -49,6 +49,7 @@ its garbage collection.
with a number of useful extensions.
<h3>Robust, High Quality Implementations.</h3>
+
* Extensive test code leveraging <a href="https://testng.org">TestNG</a>.
* Speed and accuracy performance characterization testing code
included in the <a href="https://github.com/DataSketches/sketches-misc">sketches-misc</a> repository.
@@ -56,13 +57,14 @@ its garbage collection.
<a href="{{site.core_readme}}">Core</a>,
<a href="{{site.pig_readme}}">Pig</a>,
<a href="{{site.hive_readme}}">Hive</a>,) as measured by
-<a href="https://eclemma.org/JaCoCo">JaCoCo</a> and published by
+<a href="https://www.eclemma.org/jacoco/">JaCoCo</a> and published by
<a href="https://coveralls.io">Coveralls</a>.
* Comprehensive Javadocs that satisfy
<a href="https://www.oracle.com/technetwork/java/index.html">Java JDK8</a> standards.
* Suitable for production environments.
<h3>Opportunities to Extend</h3>
+
* There is ample opportunity for interested parties to contribute additional algorithms in this exciting area.
@@ -72,6 +74,7 @@ its garbage collection.
<h3>Count Distinct / Count Unique</h3>
<h4>Solves Computational Challenges Associated with Unique Identifiers</h4>
+
* <b>Estimating cardinality</b> of a stream with many duplicates
* Performing <a href="{{site.docs_dir}}/Theta/ThetaSketchSetOps.html">set operations</a> (e.g., Union, Intersection,
and Difference) on sets of unique identifiers
@@ -79,17 +82,20 @@ its garbage collection.
* Two families of Count Unique algorithms:
* <a href="{{site.docs_dir}}/Theta/ThetaSketchFramework.html">The Theta Sketch Framework</a> algorithms that are tuned
for operation on the java heap or off-heap.
- * <a href="{{site.docs_dir}}/HLL/HLL.html">The Hyper-Log Log algorithms<a/> when sketch size is of utmost concern.
+ * <a href="{{site.docs_dir}}/HLL/HLL.html">The Hyper-Log Log algorithms</a> when sketch size is of utmost concern.
<h3>Quantiles</h3>
+
* Get normal or inverse PDFs or CDFs of the distributions of any numeric value from your raw data in a
single pass.
* Well defined error bounds on the result.
<h3>Frequent Items</h3>
+
* Get the most frequent items from a stream of items.
<h3>Tuple Sketch</h3>
+
* Associative sketches that are useful for performing approximate join operations and
extracting other kinds of behavior associated with unique identifiers.
diff --git a/docs/OrderSensitivity.md b/docs/Architecture/OrderSensitivity.md
similarity index 100%
rename from docs/OrderSensitivity.md
rename to docs/Architecture/OrderSensitivity.md
diff --git a/docs/SketchCriteria.md b/docs/Architecture/SketchCriteria.md
similarity index 100%
rename from docs/SketchCriteria.md
rename to docs/Architecture/SketchCriteria.md
diff --git a/docs/Architecture/SketchesByComponent.md b/docs/Architecture/SketchesByComponent.md
index 4600486..b1bc5c8 100644
--- a/docs/Architecture/SketchesByComponent.md
+++ b/docs/Architecture/SketchesByComponent.md
@@ -124,7 +124,7 @@ org.apache.datasketches.characterization.uniquecount | Base Profiles for Unique
### incubator-datasketches-vector
-This component implements the [Frequent Directions Algorithm](https://datasketches.apache.org/docs/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on the Memory component.
+This component implements the [Frequent Directions Algorithm](https://datasketches.apache.org/docs/Community/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on the Memory component.
## C++ and Python
diff --git a/docs/downloads.md b/docs/Community/Downloads.md
similarity index 100%
rename from docs/downloads.md
rename to docs/Community/Downloads.md
diff --git a/docs/Research.md b/docs/Community/Research.md
similarity index 99%
rename from docs/Research.md
rename to docs/Community/Research.md
index 0f61cb6..c3ec0a0 100644
--- a/docs/Research.md
+++ b/docs/Community/Research.md
@@ -149,7 +149,7 @@ Michael Mitzenmacher, Thomas Steinke, and Justin Thaler. Hierarchical heavy hitt
Lee Rhodes, Kevin Lang, Alexander Saydakov, Justin Thaler, Edo Liberty, and Jon Malkin. DataSketches: A Java software library for streaming data algorithms. Apache License, Version 2.0, 2015. <https://datasketches.apahce.org>.
[Tha07]
-Justin Thaler. REU project website: A near-optimal algorithm for computing the entropy of a stream, 2007. <https://reu.dimacs.rutgers.edu/~jthaler/>.
+Justin Thaler. REU project website: A near-optimal algorithm for computing the entropy of a stream, 2007. <https://people.cs.georgetown.edu/jthaler>.
[VSGB05]
Shobha Venkataraman, Dawn Xiaodong Song, Phillip B. Gibbons, and Avrim Blum. New streaming algorithms for fast detection of superspreaders. In *Internet Society NDSS Proceedings*, 2005.
diff --git a/community/index.md b/docs/Community/index.md
similarity index 95%
rename from community/index.md
rename to docs/Community/index.md
index aef3f17..063f5db 100644
--- a/community/index.md
+++ b/docs/Community/index.md
@@ -20,11 +20,14 @@ There are many ways that are available for our community to communicate with eac
<br/><br/>
* **[GitHub Components]({{site.docs_dir}}/Architecture/Components.html):** Our library is made up of components that are partitioned into GitHub repositories. If you have a specific issue or bug report that impacts only one of these components please open an issue on the respective component. If you are a developer and wish to submit a PR, please choose the appropriate repository.
+## Downloads
+* [Downloads]({{site.docs_dir}}/Community/Downloads.html)
+
## Contributing
We are always open to contributions from our community. Contributions can be of many forms: documentation, testing, science as well as bug fixes, code enhancements, code reviews, feature suggestions, usability feedback, etc. Contributions usually take the form of a Pull Request (PR), but if you wish to contribute and not sure how, please contact us on our [dev]({{page.dev}}) list.
-We are also open to the submission of entirely new sketch algorithms. If you have a sketch algorithm (or a significant enhancement of our current algorithms), please read our [Sketch Criteria]({{site.docs_dir}}/SketchCriteria.html) and contact us on our [dev]({{page.dev}}) list.
+We are also open to the submission of entirely new sketch algorithms. If you have a sketch algorithm (or a significant enhancement of our current algorithms), please read our [Sketch Criteria]({{site.docs_dir}}/Architecture/SketchCriteria.html) and contact us on our [dev]({{page.dev}}) list.
### What to work on
We are in the process of developing a comprehensive TODO list that will be referenced here.
@@ -39,8 +42,6 @@ Proposed changes to the code or documentation are usually done through GitHub Pu
### Becoming a committer
We welcome anyone who is eager to continue to contribute to the DataSketches mission of providing open source, production quality sketch algorithms and become part of our team. Please send us a message on [dev]({{page.dev}}) where we can give you some guidance. After you have made some successful contributions, the current committers will discuss your candidacy for becoming a committer. You can also review the [Apache policies on becoming a committer](https://community.apache.org/cont [...]
-
-
## Governance
The [Project Management Committee](https://www.apache.org/foundation/how-it-works.html) (PMC) is responsible for the administrative aspects of the DataSketches project.
diff --git a/docs/Frequency/FrequentItemsOverview.md b/docs/Frequency/FrequentItemsOverview.md
index 3a640c1..242c5c4 100644
--- a/docs/Frequency/FrequentItemsOverview.md
+++ b/docs/Frequency/FrequentItemsOverview.md
@@ -105,7 +105,7 @@ all item counts, and <i>epsilon = 3.5/M</i>, where <i>M</i> is the <i>maxMapSize
This is a worst case guarantee that applies to arbitrary inputs.<sup>1</sup>
For inputs typically seen in practice <i>(UB-LB)</i> is usually much smaller.
-The [Frequent Items Error Table]({{site.docs_dir}}/FrequentItems/FrequentItemsErrorTable.html) can serve as a guide for selecting an
+The [Frequent Items Error Table]({{site.docs_dir}}/Frequency/FrequentItemsErrorTable.html) can serve as a guide for selecting an
appropriate sized sketch for your application.
### Returned Results
diff --git a/docs/HLL/HllSketchVsDruidHyperLogLogCollector.md b/docs/HLL/HllSketchVsDruidHyperLogLogCollector.md
index a23db5b..e7fce06 100644
--- a/docs/HLL/HllSketchVsDruidHyperLogLogCollector.md
+++ b/docs/HLL/HllSketchVsDruidHyperLogLogCollector.md
@@ -21,12 +21,12 @@ layout: doc_page
-->
# HLL sketch vs Druid HyperLogLogCollector
-The goal of this article is to compare the HLL sketch implemented in this library to the <a href="https://github.com/druid-io/druid/tree/master/hll/src/main/java/io/druid/hll">Druid HyperLogLogCollector</a>.
+The goal of this article is to compare the HLL sketch implemented in this library to the <a href="https://github.com/apache/druid/tree/master/hlll">Druid HyperLogLogCollector</a>.
## Versions
-* HLL sketch form <a href="https://github.com/DataSketches/sketches-core/releases/tag/sketches-core-0.11.1">sketches-core-0.11.1</a> (April 20, 2018)
-* Druid HyperLogLogCollector from <a href="https://github.com/druid-io/druid/releases/tag/druid-0.12.0">druid-0.12.0</a> (March 8, 2018)
+* HLL sketch form <a href="https://github.com/apache/incubator-datasketches-java">Java Version 0.11.1, April 20, 2018</a>
+* Druid HyperLogLogCollector from <a href="https://github.com/apache/druid">Apache Druid Version 0.12.0, March 8, 2018</a>
## Size
@@ -85,8 +85,8 @@ This slightly larger error is still within the RSE Specification of error define
## Source code
-The code to reproduce these measurements is available in the <a href="https://github.com/DataSketches/characterization/tree/druid-hyperloglogcollector">Datasketches/characterization repository</a>
+The code to reproduce these measurements is available in the <a href="https://github.com/apache/incubator-datasketches-characterization">Datasketches/characterization repository</a>
-## HLL sketch Druid module
+## DataSketches HLL Sketch Druid module
-The <a href="http://druid.io/docs/latest/development/extensions-core/datasketches-hll.html">Hll sketch module</a> for Druid is available as a part of the <a href="http://druid.io/docs/latest/development/extensions-core/datasketches-extension.html">DataSketches extension</a>.
+The DataSketches Hll sketch module for Druid is available as a part of the <a href="https://github.com/apache/druid/tree/master/extensions-core/datasketches">druid/extensions-core</a>.
diff --git a/docs/LargeScale.md b/docs/LargeScale.md
index 249017c..ddc23d1 100644
--- a/docs/LargeScale.md
+++ b/docs/LargeScale.md
@@ -41,11 +41,11 @@ layout: doc_page
### Integration for Hive, Pig, Druid and Spark
-* <a href="https://github.com/DataSketches/sketches-hive/tree/master/src/main/java/org/apache/datasketches/hive">Hadoop / Hive Adaptors</a>.
+* <a href="https://github.com/apache/incubator-datasketches-hive">Hadoop / Hive Adaptors</a>.
-* <a href="https://github.com/DataSketches/sketches-pig/tree/master/src/main/java/org/apache/datasketches/pig">Hadoop / Pig Adaptors</a>.
+* <a href="https://github.com/apache.incubator-datasketches-pig">Hadoop / Pig Adaptors</a>.
-* <a href="https://github.com/druid-io/druid/tree/master/extensions-core/datasketches/src/main/java/io/druid/query/aggregation/datasketches">Druid Adaptors</a>.
+* <a href="https://github.com/apache/druid/tree/master/extensions-core/datasketches">Druid Adaptors</a>.
* For documentation navigate to <i>druid.io/docs/latest/development/extensions-core/datasketches-aggregators.html</i>
* <a href="{{site.docs_dir}}/Theta/ThetaSparkExample.html">Spark Examples</a>
diff --git a/docs/Quantiles/KllSketchVsTDigest.md b/docs/Quantiles/KllSketchVsTDigest.md
index 92e4173..54beda8 100644
--- a/docs/Quantiles/KllSketchVsTDigest.md
+++ b/docs/Quantiles/KllSketchVsTDigest.md
@@ -119,4 +119,4 @@ The input for the following speed measurements was generated using uniform rando
## Source code
-The code to reproduce these measurements is available in the <a href="https://github.com/DataSketches/characterization/tree/tdigest">Datasketches/characterization repository</a>
+The code to reproduce these measurements is available in the <a href="https://github.com/apache/incubator-datasketches-characterization">Datasketches/characterization repository</a>
diff --git a/docs/Sampling/ReservoirSamplingPerformance.md b/docs/Sampling/ReservoirSamplingPerformance.md
index fca4663..7b3b1a6 100644
--- a/docs/Sampling/ReservoirSamplingPerformance.md
+++ b/docs/Sampling/ReservoirSamplingPerformance.md
@@ -56,7 +56,7 @@ Consistent with reservoir updates, we can see that union speed has a nearly line
### Comparison with Apache DataFu
-[DataFu](datafu.apache.org) is an Apache Incubator project that includes a set of Pig UDFs for various data processing purposes.
+[DataFu](https://datafu.apache.org) is an Apache Incubator project that includes a set of Pig UDFs for various data processing purposes.
Because the focus of the Sketches library is on sublinear or bounded-size algorithms, compared the performance of our reservoir
sampling Pig UDF against DataFu's basic reservoir sampling, which uses a pre-specified maximum size.
diff --git a/docs/Sampling/VarOptSampling.md b/docs/Sampling/VarOptSampling.md
index b060dd9..4bbff40 100644
--- a/docs/Sampling/VarOptSampling.md
+++ b/docs/Sampling/VarOptSampling.md
@@ -59,7 +59,7 @@ duplicate items if the input stream contains duplicates.
### More on VarOpt Sampling
-The basic VarOpt algorithm was first presented by Cohen et al[1]. We have modified and extended that work to allow for unions of VarOpt samples, producing a result that remains a valid VarOpt sketch.
+The basic VarOpt algorithm was first presented by Cohen et al<sup>1</sup>. We have modified and extended that work to allow for unions of VarOpt samples, producing a result that remains a valid VarOpt sketch.
The underlying goal of VarOpt sampling is to provide the best possible estimate of subset sums of items in the sample. As an example, we might select a sample o size <tt>k</tt> from the ~3200 counties (a political administrative region below the level of a state) in the United States, using the county population as the weight. We could then apply a predicate to our sample -- for instance, counties in the state of California -- and sum the resulting weights. That sum is our estimate of th [...]
@@ -78,4 +78,4 @@ When extracting items from a sketch, the heavy items retain their original weigh
When using VarOpt with uniform input weights, for instance giving all items a weight of 1.0, the sketch will perform standard reservoir sampling. Similarly, if operating over an infinite stream of items with finite weights, there will also be no heavy items and the sketch will return a standard weighted sample of items.
-[1]: E. Cohen, N. G. Guffield, H. Kapla, C. Lund, M. Thorup, <em>Efficient Stream Sampling for Variance-Optimal Estimation of Subset Sums</em>, SIAM J. Comput. 40(5): 1402-1431, 2011.
+1. E. Cohen, N. G. Guffield, H. Kapla, C. Lund, M. Thorup, <em>Efficient Stream Sampling for Variance-Optimal Estimation of Subset Sums</em>, SIAM J. Comput. 40(5): 1402-1431, 2011.
diff --git a/docs/SketchElements.md b/docs/SketchElements.md
index c66a119..2bdec1c 100644
--- a/docs/SketchElements.md
+++ b/docs/SketchElements.md
@@ -62,4 +62,4 @@ be merged without losing accuracy.
are a function of the configured size of the sketch.
With this background, let's examine some of the
-<a href="{{site.docs_dir}}/KeyFeatures.html">Key Features</a> of the DataSketches library.
+<a href="{{site.docs_dir}}/Architecture/KeyFeatures.html">Key Features</a> of the DataSketches library.
diff --git a/docs/Theta/ThetaUpdateSpeed.md b/docs/Theta/ThetaUpdateSpeed.md
index 1ebd9fd..aa2d917 100644
--- a/docs/Theta/ThetaUpdateSpeed.md
+++ b/docs/Theta/ThetaUpdateSpeed.md
@@ -63,7 +63,7 @@ It needs to be pointed out that these tests were designed to measure the maximum
Very few systems would actually be able to feed a single sketch at this rate so these plots represent an upper bound of performance, and not as realistic update rates in more complex systems environments. Nonetheless, this demonstrates that the sketches would consume very little of an overall system's budget for updating, if there was one, and are quite suitable for real-time streams.
The graphs on this page were generated using the utilities in the
-[Characterization Repository](https://github.com/apache/incubator-datasketches-characterization repository).
+[Characterization Repository](https://github.com/apache/incubator-datasketches-characterization).
There is some more documentation with the code on using these tools if you wish to re-run these characterization tests yourself.
diff --git a/index.md b/index.md
index f0c2441..5d5574f 100644
--- a/index.md
+++ b/index.md
@@ -39,9 +39,9 @@ id: home
<!--<p class="lead" style="font-size: 16px; line-height: 1.0; margin-bottom: 15px"><i>"Excellence in theoretically informed algorithm engineering" -- Graham Cormode</i></p> -->
<p>
<a class="btn btn-lg btn-outline-inverse" href="overview.html"><span class="fa fa-info-circle"></span> Overview</a>
- <a class="btn btn-lg btn-outline-inverse" href="/docs/downloads.html"><span class="fa fa-download"></span> Download</a>
+ <a class="btn btn-lg btn-outline-inverse" href="/docs/Community/Downloads.html"><span class="fa fa-download"></span> Download</a>
<a class="btn btn-lg btn-outline-inverse" href="https://github.com/apache?utf8=%E2%9C%93&q=datasketches"><span class="fa fa-github"></span> GitHub</a>
- <a class="btn btn-lg btn-outline-inverse" href="/docs/Research.html"><span class="fa fa-paper-plane"></span> Research</a>
+ <a class="btn btn-lg btn-outline-inverse" href="/docs/Community/Research.html"><span class="fa fa-paper-plane"></span> Research</a>
<a class="btn btn-lg btn-outline-inverse" href="https://lists.apache.org/list.html?users@datasketches.apache.org"><span class="fa fa-comment"></span> Contact Us</a>
</p>
</div>
@@ -64,7 +64,7 @@ id: home
</div>
<div class="row text-center main-marketing">
<div class="col-md-4">
- <p><a href="/docs/KeyFeatures.html">
+ <p><a href="/docs/Architecture/KeyFeatures.html">
<span class="fa fa-fighter-jet fa-4x"></span><br>
<h2>Fast</h2>
</a></p>
@@ -76,7 +76,7 @@ id: home
</div>
<div class="col-md-4">
- <a href="/docs/KeyFeatures.html">
+ <a href="/docs/Architecture/KeyFeatures.html">
<span class="fa fa-database fa-4x"></span>
<h2>Big Data</h2>
</a>
@@ -87,7 +87,7 @@ id: home
</div>
<div class="col-md-4">
- <p><a href="/docs/KeyFeatures.html">
+ <p><a href="/docs/Architecture/KeyFeatures.html">
<span class="fa fa-bar-chart-o fa-4x"></span><br>
<h2>Analysis</h2>
</a></p>
@@ -101,22 +101,5 @@ id: home
</div>
</div>
-<div class="ds-panel">
- <div class="container">
- <div class="row-fluid text-center">
- <div class="col-sm-4 col-sm-offset-2">
- <h3>Get Started</h3>
- <p><a href="/docs/Architecture.html">Architecture</a></p> <!-- add {{site.stable_version}} -->
- <!-- <p><a href="/docs/tutorials.html">Tutorials</a></p> add {{site.stable_version}} -->
- <!-- ><p style="color:rgba(177,186,198,0.7)">Latest Stable Release</p> add {{site.stable_version}} -->
- </div>
- <div class="col-sm-4">
- <h3>Learn More</h3>
- <p><a href="/docs/KeyFeatures.html">Key Features</a></p>
- <p><a href="/docs/TheChallenge.html">The Challenge</a></p>
- <p><a href="/docs/WhoUses.html">Who uses DataSketches?</a></p>
- </div>
- </div>
- </div>
-</div>
+
<!-- End index.md -->
diff --git a/src/main/resources/docgen/toc.json b/src/main/resources/docgen/toc.json
index 09ab748..dc495f7 100644
--- a/src/main/resources/docgen/toc.json
+++ b/src/main/resources/docgen/toc.json
@@ -6,24 +6,30 @@
{"class":"Doc", "desc" : "The Major Sketch Families", "dir" : "", "file": "MajorSketchFamilies" },
{"class":"Doc", "desc" : "Sketch Origins", "dir" : "", "file": "SketchOrigins" },
{"class":"Doc", "desc" : "Sketch Elements", "dir" : "", "file": "SketchElements" },
- {"class":"Doc", "desc" : "Key Features", "dir" : "", "file": "KeyFeatures" },
+
{"class":"Doc", "desc" : "Large Scale Computing", "dir" : "", "file": "LargeScale" },
- {"class":"Doc", "desc" : "Architecture", "dir" : "", "file": "Architecture" },
+
{ "class":"Dropdown", "desc" : "Architecture", "array":
[
+ {"class":"Doc", "desc" : "Key Features", "dir" : "Architecture", "file": "KeyFeatures" },
{"class":"Doc", "desc" : "Components", "dir" : "Architecture", "file": "Components" },
{"class":"Doc", "desc" : "Sketches by Component", "dir" : "Architecture", "file": "SketchesByComponent" },
+ {"class":"Doc", "desc" : "Sketch Criteria", "dir" : "Architecture", "file": "SketchesCriteria" },
+ {"class":"Doc", "desc" : "Notes on Order Sensitivity", "dir" : "Architecture", "file": "OrderSensitivity" },
+ {"class":"Doc", "desc" : "Notes on Concurrency", "dir" : "Architecture", "file": "Concurrency" },
]
},
- {"class":"Doc", "desc" : "Notes on Order Sensitivity", "dir" : "", "file": "OrderSensitivity" },
- {"class":"Doc", "desc" : "Notes on Concurrency", "dir" : "", "file": "Concurrency" },
{"class":"Doc", "desc" : "Overview Slide Deck", "dir" : "", "file": "DataSketches_deck", "pdf":"true" },
]
},
-
+ { "class":"Dropdown", "desc" : "Community", "array":
+ [
+ {"class":"Doc", "desc" : "Community", "dir" : "Community", "file": "index" }
+ ]
+ },
{ "class":"Dropdown", "desc" : "Research", "array":
[
- {"class":"Doc", "desc" : "Research", "dir" : "", "file": "Research" }
+ {"class":"Doc", "desc" : "Research", "dir" : "Community", "file": "Research" }
]
},
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org