You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by "AlexanderSaydakov (via GitHub)" <gi...@apache.org> on 2023/08/26 00:46:39 UTC
[GitHub] [datasketches-cpp] AlexanderSaydakov opened a new pull request, #392: Generate sketches for java
AlexanderSaydakov opened a new pull request, #392:
URL: https://github.com/apache/datasketches-cpp/pull/392
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] jmalkin commented on a diff in pull request #392: Generate sketches for java
Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on code in PR #392:
URL: https://github.com/apache/datasketches-cpp/pull/392#discussion_r1310960309
##########
sampling/test/var_opt_union_serialize_for_java.cpp:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <var_opt_union.hpp>
+
+namespace datasketches {
+
+TEST_CASE("var opt union doulbe sampling", "[serialize_for_java]") {
Review Comment:
typo: double
##########
fi/test/frequent_items_sketch_serialize_for_java.cpp:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <frequent_items_sketch.hpp>
+
+namespace datasketches {
+
+TEST_CASE("frequent longs sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ frequent_items_sketch<long> sketch(6);
+ for (unsigned i = 1; i <= n; ++i) sketch.update(i);
+ REQUIRE(sketch.is_empty() == (n == 0));
+ if (n > 10) {
+ REQUIRE(sketch.get_maximum_error() > 0);
+ } else {
+ REQUIRE(sketch.get_maximum_error() == 0);
+ }
+ REQUIRE(sketch.get_total_weight() == n);
+ std::ofstream os("frequent_long_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.serialize(os);
+ }
+}
+
+TEST_CASE("frequent strings sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ frequent_items_sketch<std::string> sketch(6);
+ for (unsigned i = 1; i <= n; ++i) sketch.update(std::to_string(i));
+ REQUIRE(sketch.is_empty() == (n == 0));
+ if (n > 10) {
+// REQUIRE(sketch.get_maximum_error() > 0);
Review Comment:
no max error here?
##########
theta/test/theta_sketch_serialize_for_java.cpp:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <theta_sketch.hpp>
+
+namespace datasketches {
+
+TEST_CASE("theta sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ auto sketch = update_theta_sketch::builder().build();
+ for (unsigned i = 0; i < n; ++i) sketch.update(i);
+ REQUIRE(sketch.is_empty() == (n == 0));
+ REQUIRE(sketch.get_estimate() == Approx(n).margin(n * 0.03));
+ std::ofstream os("theta_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.compact().serialize(os);
+ }
+}
+
+TEST_CASE("theta sketch generate compressed", "[serialize_for_java]") {
+ const unsigned n_arr[] = {10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ auto sketch = update_theta_sketch::builder().build();
+ for (unsigned i = 0; i < n; ++i) sketch.update(i);
+ REQUIRE_FALSE(sketch.is_empty());
+ REQUIRE(sketch.get_estimate() == Approx(n).margin(n * 0.03));
+ std::ofstream os("theta_compressed_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.compact().serialize(os);
+ }
+}
+
+TEST_CASE("theta sketch generate non-empty no entries", "[serialize_for_java]") {
+ auto sketch = update_theta_sketch::builder().set_p(0.01).build();
+ sketch.update(1);
Review Comment:
We do this for AOD, too. It does depend on the hash function vs p=0.01. I don't think it's critical but it is conditioning a test on an an implicit assumption around the hash function.
##########
sampling/test/var_opt_union_serialize_for_java.cpp:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <var_opt_union.hpp>
+
+namespace datasketches {
+
+TEST_CASE("var opt union doulbe sampling", "[serialize_for_java]") {
+ const unsigned k_small = 16;
+ const unsigned k_max = 128;
+ const unsigned n1 = 32;
+ const unsigned n2 = 64;
+
+ // small k sketch, but sampling
+ var_opt_sketch<double> sketch1(k_small);
+ for (unsigned i = 0; i < n1; ++i) sketch1.update(i);
+ // heavy items have negative weights to allow a simple predicate to filter
Review Comment:
s/weights/values/
##########
sampling/test/var_opt_sketch_serialize_for_java.cpp:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <var_opt_sketch.hpp>
+
+namespace datasketches {
+
+TEST_CASE("varopt sketch long generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ var_opt_sketch<long> sketch(32);
+ for (unsigned i = 1; i <= n; ++i) sketch.update(i);
+ std::ofstream os("varopt_sketch_long_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.serialize(os);
+ }
+}
+
+TEST_CASE("varopt sketch string exact", "[serialize_for_java]") {
+ var_opt_sketch<std::string> sketch(1024);
+ for (unsigned i = 1; i <= 200; ++i) sketch.update(std::to_string(i), 1000.0 / i);
+ std::ofstream os("varopt_sketch_string_exact_cpp.sk", std::ios::binary);
+ sketch.serialize(os);
+}
+
+TEST_CASE("varopt sketch long sampling", "[serialize_for_java]") {
+ var_opt_sketch<long> sketch(1024);
+ for (unsigned i = 0; i < 2000; ++i) sketch.update(i);
+ // heavy items have negative weights to allow a simple predicate to filter
Review Comment:
This was an error in the comment in the original test. Negative values, not weights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] AlexanderSaydakov commented on a diff in pull request #392: Generate sketches for java
Posted by "AlexanderSaydakov (via GitHub)" <gi...@apache.org>.
AlexanderSaydakov commented on code in PR #392:
URL: https://github.com/apache/datasketches-cpp/pull/392#discussion_r1310965596
##########
theta/test/theta_sketch_serialize_for_java.cpp:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <theta_sketch.hpp>
+
+namespace datasketches {
+
+TEST_CASE("theta sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ auto sketch = update_theta_sketch::builder().build();
+ for (unsigned i = 0; i < n; ++i) sketch.update(i);
+ REQUIRE(sketch.is_empty() == (n == 0));
+ REQUIRE(sketch.get_estimate() == Approx(n).margin(n * 0.03));
+ std::ofstream os("theta_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.compact().serialize(os);
+ }
+}
+
+TEST_CASE("theta sketch generate compressed", "[serialize_for_java]") {
+ const unsigned n_arr[] = {10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ auto sketch = update_theta_sketch::builder().build();
+ for (unsigned i = 0; i < n; ++i) sketch.update(i);
+ REQUIRE_FALSE(sketch.is_empty());
+ REQUIRE(sketch.get_estimate() == Approx(n).margin(n * 0.03));
+ std::ofstream os("theta_compressed_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.compact().serialize(os);
+ }
+}
+
+TEST_CASE("theta sketch generate non-empty no entries", "[serialize_for_java]") {
+ auto sketch = update_theta_sketch::builder().set_p(0.01).build();
+ sketch.update(1);
Review Comment:
right, but because 1 always has the same hash, and the test passes, it will stay that way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] AlexanderSaydakov commented on a diff in pull request #392: Generate sketches for java
Posted by "AlexanderSaydakov (via GitHub)" <gi...@apache.org>.
AlexanderSaydakov commented on code in PR #392:
URL: https://github.com/apache/datasketches-cpp/pull/392#discussion_r1310967269
##########
fi/test/frequent_items_sketch_serialize_for_java.cpp:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <catch2/catch.hpp>
+#include <fstream>
+#include <frequent_items_sketch.hpp>
+
+namespace datasketches {
+
+TEST_CASE("frequent longs sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ frequent_items_sketch<long> sketch(6);
+ for (unsigned i = 1; i <= n; ++i) sketch.update(i);
+ REQUIRE(sketch.is_empty() == (n == 0));
+ if (n > 10) {
+ REQUIRE(sketch.get_maximum_error() > 0);
+ } else {
+ REQUIRE(sketch.get_maximum_error() == 0);
+ }
+ REQUIRE(sketch.get_total_weight() == n);
+ std::ofstream os("frequent_long_n" + std::to_string(n) + "_cpp.sk", std::ios::binary);
+ sketch.serialize(os);
+ }
+}
+
+TEST_CASE("frequent strings sketch generate", "[serialize_for_java]") {
+ const unsigned n_arr[] = {0, 1, 10, 100, 1000, 10000, 100000, 1000000};
+ for (const unsigned n: n_arr) {
+ frequent_items_sketch<std::string> sketch(6);
+ for (unsigned i = 1; i <= n; ++i) sketch.update(std::to_string(i));
+ REQUIRE(sketch.is_empty() == (n == 0));
+ if (n > 10) {
+// REQUIRE(sketch.get_maximum_error() > 0);
Review Comment:
forgot to uncomment after some experiment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] coveralls commented on pull request #392: Generate sketches for java
Posted by "coveralls (via GitHub)" <gi...@apache.org>.
coveralls commented on PR #392:
URL: https://github.com/apache/datasketches-cpp/pull/392#issuecomment-1696523532
## Pull Request Test Coverage Report for [Build 6005659837](https://coveralls.io/builds/62283474)
* **0** of **0** changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage decreased (**-0.002%**) to **98.96%**
---
| Totals | [](https://coveralls.io/builds/62283474) |
| :-- | --: |
| Change from base [Build 5957904519](https://coveralls.io/builds/62190462): | -0.002% |
| Covered Lines: | 15610 |
| Relevant Lines: | 15774 |
---
##### 💛 - [Coveralls](https://coveralls.io)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org
[GitHub] [datasketches-cpp] AlexanderSaydakov merged pull request #392: Generate sketches for java
Posted by "AlexanderSaydakov (via GitHub)" <gi...@apache.org>.
AlexanderSaydakov merged PR #392:
URL: https://github.com/apache/datasketches-cpp/pull/392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org