You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2021/08/30 10:05:19 UTC
svn commit: r1892729 - in /kylin/site:
cn/docs40/howto/howto_cleanup_storage.html cn/download/index.html
docs40/howto/howto_cleanup_storage.html docs40/release_notes.html
download/index.html feed.xml
Author: lidong
Date: Mon Aug 30 10:05:19 2021
New Revision: 1892729
URL: http://svn.apache.org/viewvc?rev=1892729&view=rev
Log:
Release Kylin 4.0.0
Modified:
kylin/site/cn/docs40/howto/howto_cleanup_storage.html
kylin/site/cn/download/index.html
kylin/site/docs40/howto/howto_cleanup_storage.html
kylin/site/docs40/release_notes.html
kylin/site/download/index.html
kylin/site/feed.xml
Modified: kylin/site/cn/docs40/howto/howto_cleanup_storage.html
URL: http://svn.apache.org/viewvc/kylin/site/cn/docs40/howto/howto_cleanup_storage.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/cn/docs40/howto/howto_cleanup_storage.html (original)
+++ kylin/site/cn/docs40/howto/howto_cleanup_storage.html Mon Aug 30 10:05:19 2021
@@ -187,25 +187,7 @@ var _hmt = _hmt || [];
<article class="post-content" >
- <p>Kylin å¨æ建 cube æé´ä¼å¨ HDFS ä¸çæ临æ¶æ件ï¼é¤æ¤ä¹å¤ï¼å½æ¸
ç/å é¤/å并 cube æ¶ï¼ä¸äº parquet æ件å¯è½è¢«éçä½æ¯ä»¥ååä¹ä¸ä¼è¢«æ¥è¯¢ï¼è½ç¶ Kylin å·²ç»å¼å§åèªå¨åçåå¾åæ¶ï¼ä½ä¸ä¸å®è½è¦çå°ææçæ
åµï¼ä½ å¯ä»¥å®æå离线çåå¨æ¸
çï¼</p>
-
-<p>å¯ä»¥è¢«å é¤çæ件å
æ¬ï¼<br />
-- 临æ¶çä»»å¡æ件<br />
-<code class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/job_tmp</code><br />
-- ä¸ä¼å被ç¨å°çsegmentçcuboidæ件<br />
-<code class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/${cube_name}/${non_used_segment} </code></p>
-
-<p>æ¥éª¤ï¼<br />
-1. æ£æ¥åªäºèµæºå¯ä»¥æ¸
çï¼è¿ä¸æ¥ä¸ä¼å é¤ä»»ä½ä¸è¥¿ï¼</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false</code></pre></div>
-
-<p>请å°è¿éç (version) æ¿æ¢ä¸ºä½ å®è£
ç Kylin jar çæ¬ã<br />
-2. ä½ å¯ä»¥æ½æ¥ä¸ä¸¤ä¸ªèµæºæ¥æ£æ¥å®ä»¬æ¯å¦å·²ç»æ²¡æ被å¼ç¨äºï¼ç¶åå ä¸ââdelete trueâé项è¿è¡æ¸
çã</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true</code></pre></div>
-
+ <p>Wiki : https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4</p>
</article>
</div>
Modified: kylin/site/cn/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/cn/download/index.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/cn/download/index.html (original)
+++ kylin/site/cn/download/index.html Mon Aug 30 10:05:19 2021
@@ -188,14 +188,15 @@ var _hmt = _hmt || [];
</header>
<p>æ¨å¯ä»¥æç
§è¿äº<a href="https://www.apache.org/info/verification.html">æ¥éª¤</a> 并使ç¨è¿äº<a href="https://www.apache.org/dist/kylin/KEYS">KEYS</a>æ¥éªè¯ä¸è½½æ件çæææ§.</p>
-<h4 id="v400-beta">v4.0.0-beta</h4>
+<h4 id="v400">v4.0.0</h4>
<ul>
- <li>è¿æ¯ 4.0.0-alpha çæ¬åçä¸ä¸ªä¸»è¦çæ¬ï¼å
å«25个æ°åè½ä»¥åæ¹è¿å14个é®é¢çä¿®å¤ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æã</li>
+ <li>è¿æ¯ Kylin 4 ç第ä¸ä¸ªæ£å¼çæ¬ï¼å
å«32个æ°åè½ä»¥åæ¹è¿å10个é®é¢çä¿®å¤ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æã</li>
<li><a href="/docs/release_notes.html">åå¸è¯´æ</a>, <a href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide">å®è£
æå</a> and <a href="/docs40/howto/howto_upgrade.html">å级æå</a></li>
- <li>æºç ä¸è½½: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip">apache-kylin-4.0.0-beta-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.sha256">sha256</a>]</li>
- <li>Hadoop 2 å Hadoop 3 äºè¿å¶å
(请为 Kylin 4.x 使ç¨æå®çæ¬ç Sparkï¼çæ¬ä¸º Apache Spark 2.4.6, èä¸æ¯ç¯å¢èªå¸¦ç Spark):
+ <li>æºç ä¸è½½: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip">apache-kylin-4.0.0-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.sha256">sha256</a>]</li>
+ <li>äºè¿å¶å
(éæ©äºè¿å¶å
å请æ£æ¥ææ¡£ <a href="https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4">Hadoop æ¯æç©éµ</a>):
<ul>
- <li><a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz">apache-kylin-4.0.0-beta-bin.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.sha256">sha256</a>] (å·²ç»å¨ CDH 5.7, CDH 6.2, AWS EMR 5.31, AWS EMR 6.0.0, HDP 2.4 ç¯å¢ä¸éªè¯, Hadoop3 å EMR ç¯å¢éåé¢å¤é
ç½®ï¼è¯·<a href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide">æ¥çå®è£
æå</a>)</li>
+ <li>for Apache Spark 2.4.7 <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz">apache-kylin-4.0.0-bin-spark2.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.sha256">sha256</a>]</li>
+ <li>for Apache Spark 3.1.1 <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz">apache-kylin-4.0.0-bin-spark3.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.sha256">sha256</a>]</li>
</ul>
</li>
</ul>
Modified: kylin/site/docs40/howto/howto_cleanup_storage.html
URL: http://svn.apache.org/viewvc/kylin/site/docs40/howto/howto_cleanup_storage.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/docs40/howto/howto_cleanup_storage.html (original)
+++ kylin/site/docs40/howto/howto_cleanup_storage.html Mon Aug 30 10:05:19 2021
@@ -187,27 +187,7 @@ var _hmt = _hmt || [];
<article class="post-content" >
- <p>Kylin will generate intermediate files in HDFS during the cube building; Besides, when purge/drop/merge cubes, some Parquet file may be left and will no longer be queried; Although Kylin has started to do some <br />
-automated garbage collection, it might not cover all cases; You can do an offline storage cleanup periodically:<br />
-Which can be deleted:<br />
-- temp job files<br />
-<code class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/job_tmp</code><br />
-- none used segment cuboid files<br />
-<code class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/${cube_name}/${non_used_segment}</code></p>
-
-<p>Steps:<br />
-1. Check which resources can be cleanup, this will not remove anything:</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false</code></pre></div>
-
-<p>Here please replace (version) with the specific Kylin jar version in your installation;<br />
-2. You can pickup 1 or 2 resources to check whether theyâre no longer be referred; Then add the ââdelete trueâ option to start the cleanup:</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true</code></pre></div>
-
-<p>On finish, the temp job files and none used segment cuboid files should be dropped;</p>
-
+ <p>Wiki : https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4</p>
</article>
</div>
Modified: kylin/site/docs40/release_notes.html
URL: http://svn.apache.org/viewvc/kylin/site/docs40/release_notes.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/docs40/release_notes.html (original)
+++ kylin/site/docs40/release_notes.html Mon Aug 30 10:05:19 2021
@@ -187,7 +187,7 @@ var _hmt = _hmt || [];
<article class="post-content" >
- <p>To download latest release, please visit: <a href="http://kylin.apache.org/download/">http://kylin.apache.org/download/</a>, <br />
+ <p>To download the latest release, please visit: <a href="http://kylin.apache.org/download/">http://kylin.apache.org/download/</a>, <br />
there are source code package, binary package and installation guide avaliable.</p>
<p>Any problem or issue, please report to Apache Kylin JIRA project: <a href="https://issues.apache.org/jira/browse/KYLIN">https://issues.apache.org/jira/browse/KYLIN</a></p>
@@ -199,6 +199,65 @@ there are source code package, binary pa
<li>Development relative: <a href="mailto:dev@kylin.apache.org">dev@kylin.apache.org</a></li>
</ul>
+<h2 id="v400---2021-08-30">v4.0.0 - 2021-08-30</h2>
+
+<p><strong>New Feature</strong></p>
+
+<ul>
+ <li>[KYLIN-4498] - CubePlaner for Kylin on Parquet</li>
+ <li>[KYLIN-4895] - change spark deploy mode of kylin4.0 engine from local to cluster</li>
+ <li>[KYLIN-4905] - Support limit .. offset ⦠in spark query engine</li>
+ <li>[KYLIN-4925] - Use Spark 3 as build and query engine for Kylin 4</li>
+ <li>[KYLIN-4948] - Provide an API to allow users to adjust cuboids manually</li>
+ <li>[KYLIN-4966] - Refresh the existing segment according to the new cuboid list in kylin4</li>
+ <li>[KYLIN-5011] - Detect and scatter skewed data in dict encoding step</li>
+ <li>[KYLIN-5019] - Avoid building global dictionary from all data of fact table each time</li>
+ <li>[KYLIN-5059] - Fix error when using different HDFS cluster in cube building</li>
+</ul>
+
+<p><strong>Bug Fix</strong></p>
+
+<ul>
+ <li>[KYLIN-4729] - The hive table will be overwrited when add csv table with the same name</li>
+ <li>[KYLIN-4879] - The function of sql to remove comments is not perfect. In some cases, the sql query conditions used will be modified</li>
+ <li>[KYLIN-4887] - Segment pruner support string type partition col in spark query engine</li>
+ <li>[KYLIN-4889] - Query error when spark engine in local mode</li>
+ <li>[KYLIN-4935] - Existing JobOutputâs extend info will be lost when it is updated</li>
+ <li>[KYLIN-4967] - Forbid to set âspark.sql.adaptive.enabledâ to true when building cube with Spark 2.X</li>
+ <li>[KYLIN-5013] - Write table_snapshot to wrong cluster in Kylin4.0</li>
+ <li>[KYLIN-5014] - Spark driver log is abnormal in yarn cluster mode</li>
+ <li>[KYLIN-5021] - FilePruner in Spark3 throws NPE when no partition columns in cube</li>
+ <li>[KYLIN-5040] - Cuboid should not be exactly matched when there is no group by time partition column and there are multiple segments in the query</li>
+</ul>
+
+<p><strong>Improvement</strong></p>
+
+<ul>
+ <li>[KYLIN-4554] - Validate âfilter conditionâ on model saving</li>
+ <li>[KYLIN-4888] - Performance optimization of union query with spark engine</li>
+ <li>[KYLIN-4890] - Use numSlices = 1 to reduce task num when executing sparder canary</li>
+ <li>[KYLIN-4892] - Reduce the times of fetching files status from HDFS in FilePruner</li>
+ <li>[KYLIN-4893] - Optimize query performance when using shard by column</li>
+ <li>[KYLIN-4894] - Upgrade Apache Spark version to 2.4.7</li>
+ <li>[KYLIN-4897] - Add table snapshot and global dictionary cleaning in StorageCleanupJob</li>
+ <li>[KYLIN-4898] - Add automated test cases</li>
+ <li>[KYLIN-4903] - cache parent datasource to accelerate next layerâs cuboid building</li>
+ <li>[KYLIN-4906] - support query/job server dynamic register and discovery in kylin4</li>
+ <li>[KYLIN-4908] - Segment pruner support integer partition col in spark query engine</li>
+ <li>[KYLIN-4910] - Return hostname as Sparder URL address when spark master is set to local</li>
+ <li>[KYLIN-4917] - Fix some problem of logger system in kylin4</li>
+ <li>[KYLIN-4923] - CubeMigration Tools support migrate meta from 2.x/3.x cluster to 4.0 cluster</li>
+ <li>[KYLIN-4926] - Optimize Global Dict building: replace operation âmapPartitions.count()â with âforeachPartitionsâ</li>
+ <li>[KYLIN-4927] - Forbid to use AE when building Global Dict</li>
+ <li>[KYLIN-4936] - Exactly aggregation canât transform to project</li>
+ <li>[KYLIN-4937] - Verify the uniqueness of the global dictionary after building global dictionary</li>
+ <li>[KYLIN-4944] - Upgrade CentOS version, Hadoop version and Spark version for Kylin Docker image</li>
+ <li>[KYLIN-4945] - Repartition encoded dataset to avoid data skew caused by a single column</li>
+ <li>[KYLIN-4980] - Support prunning segments from complex filter conditions</li>
+ <li>[KYLIN-5027] - Add the config of whether to build base cuboid in kylin4</li>
+ <li>[KYLIN-5031] - The last_build_job_id of segment is null when the semgent status is RUNNING or ERROR.</li>
+</ul>
+
<h2 id="v312---2021-04-26">v3.1.2 - 2021-04-26</h2>
<p><strong>New Feature</strong></p>
Modified: kylin/site/download/index.html
URL: http://svn.apache.org/viewvc/kylin/site/download/index.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/download/index.html (original)
+++ kylin/site/download/index.html Mon Aug 30 10:05:19 2021
@@ -193,14 +193,15 @@ var _hmt = _hmt || [];
</header>
<p>You can verify the download by following these <a href="https://www.apache.org/info/verification.html">procedures</a> and using these <a href="https://www.apache.org/dist/kylin/KEYS">KEYS</a>.</p>
-<h4 id="v400-beta">v4.0.0-beta</h4>
+<h4 id="v400">v4.0.0</h4>
<ul>
- <li>This is a major release after 4.0.0-alpha, with 25 new features/improvements and 14 bug fixes. Check the release notes.</li>
+ <li>This is the first GA release for Kylin 4, with 32 new features/improvements and 10 bug fixes. Check the release notes.</li>
<li><a href="/docs/release_notes.html">Release notes</a>, <a href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide">installation guide</a> and <a href="https://cwiki.apache.org/confluence/display/KYLIN/How+to+upgrade">upgrade guide</a></li>
- <li>Source download: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip">apache-kylin-4.0.0-beta-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.sha256">sha256</a>]</li>
- <li>Binary for Apache Hadoop 2 and Hadoop 3 download (Please use the specified version Spark for Kylin 4.X, the version should be Apache Spark 2.4.6, not the Spark that provided by the environment):
+ <li>Source download: <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip">apache-kylin-4.0.0-source-release.zip</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.sha256">sha256</a>]</li>
+ <li>Binary for the download (check this to see which binary you should choose <a href="https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4">Hadoop Matrix supported</a>) :
<ul>
- <li><a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz">apache-kylin-4.0.0-beta-bin.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.sha256">sha256</a>] (Verified on CDH 5.7, CDH 6.2, AWS EMR 5.31, AWS EMR 6.0.0, HDP 2.4, Hadoop 3 and EMR environments require additional configuration, please check the <a href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide">installation guide</a>)</li>
+ <li>for Apache Spark 2.4.7 <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz">apache-kylin-4.0.0-bin-spark2.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.sha256">sha256</a>]</li>
+ <li>for Apache Spark 3.1.1 <a href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz">apache-kylin-4.0.0-bin-spark3.tar.gz</a> [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.asc">asc</a>] [<a href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.sha256">sha256</a>]</li>
</ul>
</li>
</ul>
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Mon Aug 30 10:05:19 2021
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Mon, 23 Aug 2021 06:59:11 -0700</pubDate>
- <lastBuildDate>Mon, 23 Aug 2021 06:59:11 -0700</lastBuildDate>
+ <pubDate>Mon, 30 Aug 2021 02:47:11 -0700</pubDate>
+ <lastBuildDate>Mon, 30 Aug 2021 02:47:11 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>