You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafu.apache.org by ey...@apache.org on 2025/05/18 11:02:38 UTC
svn commit: r1925663 - in /datafu/site/blog/2025: ./ 04/ 04/27/ 04/27/datafu-2-1-0-released.html
Author: eyal
Date: Sun May 18 11:02:38 2025
New Revision: 1925663
URL: http://svn.apache.org/viewvc?rev=1925663&view=rev
Log:
Add release blog post
Added:
datafu/site/blog/2025/
datafu/site/blog/2025/04/
datafu/site/blog/2025/04/27/
datafu/site/blog/2025/04/27/datafu-2-1-0-released.html
Added: datafu/site/blog/2025/04/27/datafu-2-1-0-released.html
URL: http://svn.apache.org/viewvc/datafu/site/blog/2025/04/27/datafu-2-1-0-released.html?rev=1925663&view=auto
==============================================================================
--- datafu/site/blog/2025/04/27/datafu-2-1-0-released.html (added)
+++ datafu/site/blog/2025/04/27/datafu-2-1-0-released.html Sun May 18 11:02:38 2025
@@ -0,0 +1,127 @@
+
+
+
+<!doctype html>
+<html>
+ <head>
+ <meta charset="utf-8">
+
+ <!-- Always force latest IE rendering engine or request Chrome Frame -->
+ <meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible">
+ <meta name="google-site-verification" content="9N7qTOUYyX4kYfXYc0OIomWJku3PVvGrf6oTNWg2CHI" />
+
+ <meta name="twitter:card" content="summary" />
+ <meta name="twitter:site" content="@apachedatafu" />
+ <meta name="twitter:title" content="Apache DataFu-Spark 2.1.0 Released" />
+ <meta name="twitter:description" content="I'd like to announce the release of Apache DataFu-Spark 2.1.0. In this release, Spark versions 3.0.0 to 3.4.2 are supported. Additions Add dedupByAllExcept method (DATAFU-167). This is a new method for reducing rows when there is one column... " />
+ <meta property="og:url" content="http://datafu.apache.org/blog/2025/04/27/datafu-2-1-0-released.html" />
+ <meta property="og:type" content="article" />
+ <meta property="og:title" content="Apache DataFu-Spark 2.1.0 Released" />
+ <meta property="og:description" content="I'd like to announce the release of Apache DataFu-Spark 2.1.0. In this release, Spark versions 3.0.0 to 3.4.2 are supported. Additions Add dedupByAllExcept method (DATAFU-167). This is a new method for reducing rows when there is one column... " />
+
+
+ <!-- Use title if it's in the page YAML frontmatter -->
+ <title>Apache DataFu-Spark 2.1.0 Released</title>
+
+ <link href="/stylesheets/all.css" rel="stylesheet" /><link href="/stylesheets/highlight.css" rel="stylesheet" />
+ <script src="/javascripts/all.js"></script>
+
+ </head>
+
+ <body class="blog blog_2025 blog_2025_04 blog_2025_04_27 blog_2025_04_27_datafu-2-1-0-released">
+
+ <div class="container">
+
+
+<div class="header">
+
+ <ul class="nav nav-pills pull-right">
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+
+ <h3 class="header-title"><a href="/">Apache DataFu™</a></h3>
+
+</div>
+
+
+ <div class="row">
+ <article class="col-lg-10">
+ <h1>Apache DataFu-Spark 2.1.0 Released</h1>
+ <h5 class="text-muted"><time>Apr 27, 2025</time></h5>
+ <h5 class="text-muted">Eyal Allweil</h5>
+
+ <hr>
+
+ <p>I'd like to announce the release of Apache DataFu-Spark 2.1.0.</p>
+
+<p>In this release, Spark versions 3.0.0 to 3.4.2 are supported.</p>
+
+<p><br></p>
+
+<p><strong>Additions</strong></p>
+
+<ul>
+<li>Add dedupByAllExcept method (DATAFU-167).
+
+<ul>
+<li>This is a new method for reducing rows when there is one column whose value is not important, but you don't want to lose any actual data from the other rows. For example if a server creates events with an autogenerated event id, and sometimes events are duplicated. You don't want double rows just for the event ids, but if any of the other fields are distinct you want to keep the rows (with their event ids)</li>
+<li>Syntax:
+<em>val result = df.dedupByAllExcept("event</em>id")_</li>
+</ul></li>
+<li>Add collectNumberOrderedElements (DATAFU-176).
+
+<ul>
+<li>This is a new UDAF for aggregating and collecting data with a possibility of skew. For example if you want to create a list of top customers for a company. Using a window function would require sending all the data for a given company to the same executor. This method will filter rows out in the combiner stage.</li>
+<li>Syntax:
+<em>val result = purchases.groupBy("company").agg(collectNumberOrderedElements(col("customer"), 4, ascending = false)</em></li>
+</ul></li>
+</ul>
+
+<p><strong>Improvements</strong></p>
+
+<ul>
+<li>Spark 3.0.0 - 3.4.x supported (DATAFU-175, DATAFU-179)</li>
+<li>Expose dedupRandomN in Python (DATAFU-180)</li>
+</ul>
+
+<p><strong>Breaking changes</strong></p>
+
+<ul>
+<li>The four deprecated classes in SparkUDAFs - MultiSet, MultiArraySet, MapMerge and CountDistinctUpTo have been removed. Instead of them, there are new versions which use the Spark Aggregator API.</li>
+</ul>
+
+<p><br></p>
+
+<p>The source release can be obtained from:</p>
+
+<p><a href="http://www.apache.org/dyn/closer.cgi/datafu/apache-datafu-2.1.0/apache-datafu-sources-2.1.0.tgz">http://www.apache.org/dyn/closer.cgi/datafu/apache-datafu-2.1.0/apache-datafu-sources-2.1.0.tgz</a></p>
+
+<p>Artifacts for DataFu are published in Apache's Maven Repository:</p>
+
+<p><a href="https://repository.apache.org/content/groups/public/org/apache/datafu/">https://repository.apache.org/content/groups/public/org/apache/datafu/</a></p>
+
+<p>Please visit the <a href="/docs/download.html">Download</a> page for instructions on building from source or retrieving the artifacts in your build system.</p>
+
+
+ </article>
+</div>
+
+
+
+<div class="footer">
+
+<div class="feather">
+<a href="http://www.apache.org/" target="_blank"><img src="/images/feather.png" alt="Apache Feather" title="Apache Feather"/></a>
+</div>
+
+<div class="copyright">
+ Copyright © 2011-2025 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>
+ Apache DataFu, DataFu, Apache Pig, Apache Hadoop, Hadoop, Apache, and the Apache feather logo are either registered trademarks or trademarks of the <a href="http://www.apache.org/">Apache Software Foundation</a> in the United States and other countries.
+</div>
+</div>
+
+
+ </div>
+
+ </body>
+</html>
\ No newline at end of file