You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by sunjincheng121 <gi...@git.apache.org> on 2017/09/07 00:44:27 UTC
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
GitHub user sunjincheng121 opened a pull request:
https://github.com/apache/flink/pull/4652
[FLINK-7465][table]Add cardinality count for tableAPI and SQL.
## What is the purpose of the change
*In this PR. we want add add CARDINALITY_COUNT for tableAPI and SQL.(Using `HyperLogLog` algorithm).
The implementation of HyperLogLog (HLL) algorithm from this paper: http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
As we know there are still some improved algorithms, such as: HyperLogLog++, HyperBitBit etc.
But `HyperLogLog` is a classic algorithm that has been massively verified, so I chose to use the `HyperLogLog` algorithm as the first version of cardinality to achieve. And we can improve the algorithm at any time If we need.
*
## Brief change log
- *Add Java implementation of `HyperLogLog`(base on stream-lib)*
- *Add MURMURHASH See more: http://murmurhash.googlepages.com/*
- *Add build-in `CardinalityCountAggFunction`*
- *Add some test case for the validation*
- *Add documentation for TableAPI&SQL*
## Verifying this change
This change added tests and can be verified as follows:
- *Added SQL/TableAPI integration tests for `cardinality_count`*
- *Added `CardinalityCountAggFunctionTest` test case for verify the AGG logic.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs / JavaDocs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sunjincheng121/flink FLINK-7465-PR
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4652.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4652
----
commit bc1166ad88538bdcdd6df685c750359aadff3950
Author: 金竹 <ji...@alibaba-inc.com>
Date: 2017-09-05T10:21:10Z
[FLINK-7465][table]Add cardinality count for tableAPI and SQL.
----
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140832701
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/functions/aggfunctions/cardinality/HyperLogLog.java ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.aggfunctions.cardinality;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutput;
+import java.io.Serializable;
+
+/**
+ * Java implementation of HyperLogLog (HLL) algorithm from this paper:
+ * <p/>
+ * http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
+ * <p/>
+ * HLL is an improved version of LogLog that is capable of estimating
+ * the cardinality of a set with accuracy = 1.04/sqrt(m) where
+ * m = 2^b. So we can control accuracy vs space usage by increasing
+ * or decreasing b.
+ * <p/>
+ * The main benefit of using HLL over LL is that it only requires 64%
+ * of the space that LL does to get the same accuracy.
+ * <p/>
+ * <p>
+ * Note that this implementation does not include the long range correction function
+ * defined in the original paper. Empirical evidence shows that the correction
+ * function causes more harm than good.
+ * </p>
+ */
+public class HyperLogLog implements ICardinality, Serializable {
--- End diff --
I see this class is adapted from https://github.com/addthis/stream-lib. I think you should comment that this class was adapted from the link, so people can track differences.
---
[GitHub] flink issue #4652: [FLINK-7465][table]Add cardinality count for tableAPI and...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/4652
@fhueske @Jacob Park I am appreciated if you can review the PR.
Thanks, jincheng
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by sunjincheng121 <gi...@git.apache.org>.
Github user sunjincheng121 closed the pull request at:
https://github.com/apache/flink/pull/4652
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140823213
--- Diff: docs/dev/table/sql.md ---
@@ -2020,7 +2020,16 @@ COUNT(*)
<p>Returns the number of input rows.</p>
</td>
</tr>
-
+<tr>
+ <td>
+ {% highlight text %}
+CARDINALITY_COUNT(rsd, value)
--- End diff --
Would it be clearer to the user to have the function have the word "approximate" in it such that the user understands the count is an estimate? I see Apache Spark calls it `approx_count_distinct`(https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/sql/functions.html#approx_count_distinct-org.apache.spark.sql.Column-double-) and Redshift has it as `APPROXIMATE COUNT(DISTINCT column)` (http://docs.aws.amazon.com/redshift/latest/dg/r_COUNT.html).
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140830364
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/functions/aggfunctions/cardinality/ICardinality.java ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.aggfunctions.cardinality;
+
+import java.io.IOException;
+
+/**
+ * An interface definition for implementation of cardinality.
+ */
+public interface ICardinality {
--- End diff --
Why is the interface named `ICardinlaity`? Is this common in the Flink codebase? I know prefixing "I" is common in C#.
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r143956930
--- Diff: docs/dev/table/sql.md ---
@@ -2020,7 +2020,16 @@ COUNT(*)
<p>Returns the number of input rows.</p>
</td>
</tr>
-
+<tr>
+ <td>
+ {% highlight text %}
+CARDINALITY_COUNT(rsd, value)
--- End diff --
I agree with @jparkie. It must be clear from the name of the function that it computes an approximated count.
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140826158
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/functions/aggfunctions/cardinality/MurmurHash.java ---
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.aggfunctions.cardinality;
+
+/**
+ * This is a very fast, non-cryptographic hash suitable for general hash-based
+ * lookup. See http://murmurhash.googlepages.com/ for more details.
+ * <p/>
+ */
+public class MurmurHash {
--- End diff --
What was the decision in choosing MurmurHash2 (in https://github.com/tnm/murmurhash-java) versus MurmurHash3 (in https://github.com/google/guava/blob/master/guava/src/com/google/common/hash/Murmur3_32HashFunction.java)?
If I recall correctly, MurmurHash3 is more collision resistant and slightly faster than MurmurHash2.
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140828795
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/functions/aggfunctions/cardinality/ICardinality.java ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.aggfunctions.cardinality;
+
+import java.io.IOException;
+
+/**
+ * An interface definition for implementation of cardinality.
+ */
+public interface ICardinality {
+
+ /**
+ * Check whether the element is impact estimate.
+ *
+ * @param o stream element
+ * @return false if the value returned by cardinality() is unaffected by the appearance of o in the stream.
+ */
+ boolean offer(Object o);
+
+ /**
+ * Offer the value as a hashed long value.
+ *
+ * @param hashedLong - the hash of the item to offer to the estimator
+ * @return false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream
+ */
+ boolean offerHashed(long hashedLong);
+
+ /**
+ * Offer the value as a hashed long value.
+ *
+ * @param hashedInt - the hash of the item to offer to the estimator
+ * @return false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream
+ */
+ boolean offerHashed(int hashedInt);
+
+ /**
+ * @return the number of unique elements in the stream or an estimate thereof.
+ */
+ long cardinality();
+
+ /**
+ * @return size in bytes needed for serialization.
+ */
+ int sizeof();
+
+ /**
+ * Get the byte array used for the calculation.
+ *
+ * @return The byte array used for the calculation
+ * @throws IOException
+ */
+ byte[] getBytes() throws IOException;
+
+ /**
+ * Merges estimators to produce a new estimator for the combined streams
+ * of this estimator and those passed as arguments.
+ * <p/>
+ * Nor this estimator nor the one passed as parameters are modified.
+ *
+ * @param estimators Zero or more compatible estimators
+ * @throws Exception If at least one of the estimators is not compatible with this one
+ */
+ ICardinality merge(ICardinality... estimators) throws Exception;
--- End diff --
Wouldn't it be nicer to have a more specific Exception?
---
[GitHub] flink pull request #4652: [FLINK-7465][table]Add cardinality count for table...
Posted by jparkie <gi...@git.apache.org>.
Github user jparkie commented on a diff in the pull request:
https://github.com/apache/flink/pull/4652#discussion_r140833539
--- Diff: flink-libraries/flink-table/src/main/java/org/apache/flink/table/runtime/functions/aggfunctions/cardinality/HyperLogLog.java ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.functions.aggfunctions.cardinality;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutput;
+import java.io.Serializable;
+
+/**
+ * Java implementation of HyperLogLog (HLL) algorithm from this paper:
+ * <p/>
+ * http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf
+ * <p/>
+ * HLL is an improved version of LogLog that is capable of estimating
+ * the cardinality of a set with accuracy = 1.04/sqrt(m) where
+ * m = 2^b. So we can control accuracy vs space usage by increasing
+ * or decreasing b.
+ * <p/>
+ * The main benefit of using HLL over LL is that it only requires 64%
+ * of the space that LL does to get the same accuracy.
+ * <p/>
+ * <p>
+ * Note that this implementation does not include the long range correction function
+ * defined in the original paper. Empirical evidence shows that the correction
+ * function causes more harm than good.
+ * </p>
+ */
+public class HyperLogLog implements ICardinality, Serializable {
+
+ private final RegisterSet registerSet;
+ private final int log2m;
+ private final double alphaMM;
+
+
+ /**
+ * Create a new HyperLogLog instance using the specified standard deviation.
+ *
+ * @param rsd - the relative standard deviation for the counter.
+ * smaller values create counters that require more space.
+ */
+ public HyperLogLog(double rsd) {
+ this(log2m(rsd));
+ }
+
+ private static int log2m(double rsd) {
+ return (int) (Math.log((1.106 / rsd) * (1.106 / rsd)) / Math.log(2));
+ }
+
+ private static double rsd(int log2m) {
+ return 1.106 / Math.sqrt(Math.exp(log2m * Math.log(2)));
+ }
+
+ private static void validateLog2m(int log2m) {
+ if (log2m < 0 || log2m > 30) {
+ throw new IllegalArgumentException("log2m argument is "
+ + log2m + " and is outside the range [0, 30]");
+ }
+ }
+
+ private static double linearCounting(int m, double v) {
+ return m * Math.log(m / v);
+ }
+
+ /**
+ * Create a new HyperLogLog instance. The log2m parameter defines the accuracy of
+ * the counter. The larger the log2m the better the accuracy.
+ * <p/>
+ * accuracy = 1.04/sqrt(2^log2m)
+ *
+ * @param log2m - the number of bits to use as the basis for the HLL instance
+ */
+ public HyperLogLog(int log2m) {
+ this(log2m, new RegisterSet(1 << log2m));
+ }
+
+ /**
+ * Creates a new HyperLogLog instance using the given registers. Used for unmarshalling a serialized
+ * instance and for merging multiple counters together.
+ *
+ * @param registerSet - the initial values for the register set
+ */
+ public HyperLogLog(int log2m, RegisterSet registerSet) {
+ validateLog2m(log2m);
+ this.registerSet = registerSet;
+ this.log2m = log2m;
+ int m = 1 << this.log2m;
+
+ alphaMM = getAlphaMM(log2m, m);
+ }
+
+ @Override
+ public boolean offerHashed(long hashedValue) {
+ // j becomes the binary address determined by the first b log2m of x
+ // j will be between 0 and 2^log2m
+ final int j = (int) (hashedValue >>> (Long.SIZE - log2m));
+ final int r = Long.numberOfLeadingZeros((hashedValue << this.log2m) | (1 << (this.log2m - 1)) + 1) + 1;
+ return registerSet.updateIfGreater(j, r);
+ }
+
+ @Override
+ public boolean offerHashed(int hashedValue) {
+ // j becomes the binary address determined by the first b log2m of x
+ // j will be between 0 and 2^log2m
+ final int j = hashedValue >>> (Integer.SIZE - log2m);
+ final int r = Integer.numberOfLeadingZeros((hashedValue << this.log2m) | (1 << (this.log2m - 1)) + 1) + 1;
+ return registerSet.updateIfGreater(j, r);
+ }
+
+ @Override
+ public boolean offer(Object o) {
+ final int x = MurmurHash.hash(o);
+ return offerHashed(x);
+ }
+
+ @Override
+ public long cardinality() {
+ double registerSum = 0;
+ int count = registerSet.count;
+ double zeros = 0.0;
+ for (int j = 0; j < registerSet.count; j++) {
+ int val = registerSet.get(j);
+ registerSum += 1.0 / (1 << val);
+ if (val == 0) {
+ zeros++;
+ }
+ }
+
+ double estimate = alphaMM * (1 / registerSum);
+
+ if (estimate <= (5.0 / 2.0) * count) {
+ // Small Range Estimate
+ return Math.round(linearCounting(count, zeros));
+ } else {
+ return Math.round(estimate);
+ }
+ }
+
+ @Override
+ public int sizeof() {
+ return registerSet.size * 4;
+ }
+
+ @Override
+ public byte[] getBytes() throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutput dos = new DataOutputStream(baos);
+ writeBytes(dos);
+
+ return baos.toByteArray();
+ }
+
+ private void writeBytes(DataOutput serializedByteStream) throws IOException {
+ serializedByteStream.writeInt(log2m);
+ serializedByteStream.writeInt(registerSet.size * 4);
+ for (int x : registerSet.readOnlyBits()) {
+ serializedByteStream.writeInt(x);
+ }
+ }
+
+ /**
+ * Add all the elements of the other set to this set.
+ * <p/>
+ * This operation does not imply a loss of precision.
+ *
+ * @param other A compatible Hyperloglog instance (same log2m)
+ * @throws Exception if other is not compatible
+ */
+ public void addAll(HyperLogLog other) throws Exception {
+ if (this.sizeof() != other.sizeof()) {
+ throw new Exception("Cannot merge estimators of different sizes");
+ }
+
+ registerSet.merge(other.registerSet);
+ }
+
+ @Override
+ public ICardinality merge(ICardinality... estimators) throws Exception {
+ HyperLogLog merged = new HyperLogLog(log2m, new RegisterSet(this.registerSet.count));
+ merged.addAll(this);
+
+ if (estimators == null) {
+ return merged;
+ }
+
+ for (ICardinality estimator : estimators) {
+ if (!(estimator instanceof HyperLogLog)) {
+ throw new Exception("Cannot merge estimators of different class");
+ }
+ HyperLogLog hll = (HyperLogLog) estimator;
+ merged.addAll(hll);
+ }
+
+ return merged;
+ }
+
+ private Object writeReplace() {
+ return new SerializationHolder(this);
+ }
+
+ /**
+ * This class exists to support Externalizable semantics for
+ * HyperLogLog objects without having to expose a public
+ * constructor, public write/read methods, or pretend final
+ * fields aren't final.
+ *
+ * <p>
+ * In short, Externalizable allows you to skip some of the more
+ * verbose meta-data default Serializable gets you, but still
+ * includes the class name. In that sense, there is some cost
+ * to this holder object because it has a longer class name. I
+ * imagine people who care about optimizing for that have their
+ * own work-around for long class names in general, or just use
+ * a custom serialization framework. Therefore we make no attempt
+ * to optimize that here (eg. by raising this from an inner class
+ * and giving it an unhelpful name).
+ * </p>
+ */
+ private static class SerializationHolder implements Externalizable {
+
+ HyperLogLog hyperLogLogHolder;
+
+ public SerializationHolder(HyperLogLog hyperLogLogHolder) {
+ this.hyperLogLogHolder = hyperLogLogHolder;
+ }
+
+ @Override
+ public void writeExternal(ObjectOutput out) throws IOException {
+ hyperLogLogHolder.writeBytes(out);
+ }
+
+ @Override
+ public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
+ hyperLogLogHolder = Builder.build(in);
+ }
+
+ private Object readResolve() {
+ return hyperLogLogHolder;
+ }
+ }
+
+ /**
+ * Build a HyperLogLog instance.
+ */
+ public static class Builder implements Serializable {
--- End diff --
Do we really need a builder when there are only one modifiable parameter? Why pay for an instance of a Builder when you can have normal static factory methods? We can argue the JIT may make it irrelevant, but it's hard to be definitive about the JIT.
---