You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by GitBox <gi...@apache.org> on 2022/08/09 03:41:47 UTC

[GitHub] [datasketches-hive] zmin1217 opened a new issue, #64: the error of intersect reaches 41%

zmin1217 opened a new issue, #64:
URL: https://github.com/apache/datasketches-hive/issues/64

   On the TPC-H dataset，i use theta sketch to get intersect，Error of some results reaches 41%, but the doc say the default size(4096) about 3% error.
   
   spark.sql("create temporary function data2sketch as 'org.apache.datasketches.hive.theta.DataToSketchUDAF'")
   spark.sql("create temporary function intersect as 'org.apache.datasketches.hive.theta.IntersectSketchUDF'")
   spark.sql("create temporary function estimate as 'org.apache.datasketches.hive.theta.EstimateSketchUDF'")
   
   scala> lineitem.select("l_suppkey").intersect(order.select("o_orderkey")).count
   res17: Long = 250000
   but theta sketch result is 145593, the error is 0.41
   
   scala> customer.select("c_custkey").intersect(lineitem.select("l_orderkey")).count
   res18: Long = 3750000
   but theta sketch result is 4404198, the error is 0.14


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org

Re: [I] the error of intersect reaches 41% [datasketches-hive]

Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on issue #64:
URL: https://github.com/apache/datasketches-hive/issues/64#issuecomment-2111375075

   There has been no further discussion on this issue for almost 2 years so I am closing this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org

[GitHub] [datasketches-hive] jmalkin commented on issue #64: the error of intersect reaches 41%

Posted by GitBox <gi...@apache.org>.
jmalkin commented on issue #64:
URL: https://github.com/apache/datasketches-hive/issues/64#issuecomment-1211562159

   Was hoping someone a little better with longer explanations would jump on this. But I guess not.
   
   The error bounds based on sketch size k apply only when constructing a sketch or performing unions. Theta sketch intersections provide substantially better error bounds than what you can get with a sketch like HLL, which must use an include/exclude rule for estimation, but the error of intersections depends not only on the size of the initial sketches but also on the overlap of the two sketches.
   
   There's a much longer discussion here: [https://datasketches.apache.org/docs/Theta/ThetaAccuracyPlots.html](https://datasketches.apache.org/docs/Theta/ThetaAccuracyPlots.html), specifically in the section `Accuracy of Set Intersections & Differences`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org

[GitHub] [datasketches-hive] zmin1217 commented on issue #64: the error of intersect reaches 41%

Posted by GitBox <gi...@apache.org>.
zmin1217 commented on issue #64:
URL: https://github.com/apache/datasketches-hive/issues/64#issuecomment-1212701074

   the doc says:
   Set intersections and differences can have considerably more relative error than a base Theta sketch fed directly with data. Be cautious about situations where the result of an intersection (or difference) is orders-of-magnitude smaller than the union of the two arguments. 
   
   maybe the the result of intersection(145593) is orders-of-magnitude smaller than the union lead to a larger error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org

Re: [I] the error of intersect reaches 41% [datasketches-hive]

Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho closed issue #64: the error of intersect reaches 41%
URL: https://github.com/apache/datasketches-hive/issues/64


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org