You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2018/02/12 08:55:40 UTC
svn commit: r1823929 - in /kylin/site: development/dev_env.html
development/howto_contribute.html development/index.html
docs21/tutorial/cube_streaming.html docs23/howto/howto_ldap_and_sso.html
docs23/howto/howto_use_dashboard.html feed.xml
Author: lidong
Date: Mon Feb 12 08:55:39 2018
New Revision: 1823929
URL: http://svn.apache.org/viewvc?rev=1823929&view=rev
Log:
update howto contribute doc
Modified:
kylin/site/development/dev_env.html
kylin/site/development/howto_contribute.html
kylin/site/development/index.html
kylin/site/docs21/tutorial/cube_streaming.html
kylin/site/docs23/howto/howto_ldap_and_sso.html
kylin/site/docs23/howto/howto_use_dashboard.html
kylin/site/feed.xml
Modified: kylin/site/development/dev_env.html
URL: http://svn.apache.org/viewvc/kylin/site/development/dev_env.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/development/dev_env.html (original)
+++ kylin/site/development/dev_env.html Mon Feb 12 08:55:39 2018
@@ -569,7 +569,7 @@
<h2 id="environment-on-the-hadoop-cli">Environment on the Hadoop CLI</h2>
-<p>Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop sandbox) as well as your local develop machine. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like <a href="http://hortonworks.com/products/hortonworks-sandbox/">http://hortonworks.com/products/hortonworks-sandbox/</a>. In the following tutorial weâll go with <strong>Hortonworks Sandbox 2.2.4</strong>. It is recommended that you provide enough memory to your sandbox, 8G or more is preferred.</p>
+<p>Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop sandbox) as well as your local develop machine. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like <a href="http://hortonworks.com/products/hortonworks-sandbox/">http://hortonworks.com/products/hortonworks-sandbox/</a>. In the following tutorial weâll go with <strong>Hortonworks Sandbox 2.4.0.0-169</strong>. It is recommended that you provide enough memory to your sandbox, 8G or more is preferred.</p>
<h3 id="start-hadoop">Start Hadoop</h3>
@@ -596,17 +596,12 @@ ln -s /root/apache-maven-3.2.5/bin/mvn /
<h3 id="install-spark">Install Spark</h3>
-<p>Manually install spark-1.6.3-bin-hadoop2.6 in a local folder like /usr/local/spark</p>
+<p>Manually install spark-2.1.2-bin-hadoop2.7 in a local folder like /usr/local/spark</p>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">wget -O /tmp/spark-1.6.3-bin-hadoop2.6.tgz http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">wget -O /tmp/spark-2.1.2-bin-hadoop2.7.tgz http://d3kbcqa49mib13.cloudfront.net/spark-2.1.2-bin-hadoop2.7.tgz
cd /usr/local
-tar -zxvf /tmp/spark-1.6.3-bin-hadoop2.6.tgz
-ln -s spark-1.6.3-bin-hadoop2.6 spark</code></pre></div>
-
-<p>Upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS):</p>
-
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">hadoop fs -mkdir /kylin/spark/
-hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/</code></pre></div>
+tar -zxvf /tmp/spark-2.1.2-bin-hadoop2.7.tgz
+ln -s spark-2.1.2-bin-hadoop2.7 spark</code></pre></div>
<p>Create local temp folder for hbase client (if it doesnât exist):</p>
Modified: kylin/site/development/howto_contribute.html
URL: http://svn.apache.org/viewvc/kylin/site/development/howto_contribute.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/development/howto_contribute.html (original)
+++ kylin/site/development/howto_contribute.html Mon Feb 12 08:55:39 2018
@@ -570,11 +570,37 @@
<ul>
<li><code class="highlighter-rouge">master</code>: Main development branch for new features</li>
- <li><code class="highlighter-rouge">2.0.x</code>: Maintenance branch for a certain release</li>
+ <li><code class="highlighter-rouge">2.[n].x</code>: Maintenance branch for a certain major release</li>
<li><code class="highlighter-rouge">document</code>: Document branch</li>
</ul>
-<h2 id="pick-an-open-task">Pick an Open Task</h2>
+<h2 id="components-and-owners">Components and owners</h2>
+<p>Apache Kylin has several sub-components. And for each component we will arrange one or multiple component owners.</p>
+
+<p>Component owners is listed in the description field on this Apache Kylin <a href="https://issues.apache.org/jira/projects/KYLIN?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page">JIRA components page</a>. The owners are listed in the âDescriptionâ field rather than in the âComponent Leadâ field because the latter only allows us to list one individual whereas it is encouraged that components have multiple owners.</p>
+
+<ul>
+ <li>
+ <p>Component owners are volunteers who are expert in their component domain and may have an agenda on how they think their Apache Kylin component should evolve. The owner needs to be an Apache Kylin committer at this moment.</p>
+ </li>
+ <li>
+ <p>Owners will try and review patches that land within their componentâs scope.</p>
+ </li>
+ <li>
+ <p>Owners can rotate, based on his aspiration.</p>
+ </li>
+ <li>
+ <p>When nominate or vote a new committer, the nominator needs to state which component the candidate can be the owner.</p>
+ </li>
+ <li>
+ <p>If youâre already an Apache Kylin committer and would like to be a volunteer as a component owner, just write to the dev list and weâll sign you up.</p>
+ </li>
+ <li>
+ <p>If you think the component list need be updated (add, remove, rename, etc), write to the dev list and weâll review that.</p>
+ </li>
+</ul>
+
+<h2 id="pick-a-task">Pick a task</h2>
<p>There are open tasks waiting to be done, tracked by JIRA. To make it easier to search, there are a few JIRA filters.</p>
<ul>
@@ -585,6 +611,22 @@
<p>Do not forget to discuss in <a href="/community/index.html">mailing list</a> before working on a big task.</p>
+<p>If create a new JIRA for bug or feature, remember to provide enough information for the community:</p>
+
+<ul>
+ <li>A well summary for the problem or feature</li>
+ <li>A detail description, which may include:
+ <ul>
+ <li>the environment of this problem occurred</li>
+ <li>the steps to reproduce the problem</li>
+ <li>the error trace or log files (as attachment)</li>
+ <li>the metadata of the model or cube</li>
+ </ul>
+ </li>
+ <li>Related components: we will arrange reviewer based on this selection.</li>
+ <li>Affected version: which Kylin youâre using.</li>
+</ul>
+
<h2 id="making-code-changes">Making Code Changes</h2>
<ul>
<li><a href="/development/dev_env.html">Setup dev env</a></li>
@@ -593,7 +635,7 @@
<li>Make changes in your fork
<ul>
<li>No strict code style at the moment, but the general rule is keep consistent with existing files. E.g. use 4-space indent for java files.</li>
- <li>Add unit test for your code change as much as possible.</li>
+ <li>Add test case for your code change as much as possible.</li>
<li>Make sure âmvn clean packageâ and âmvn testâ can get success.</li>
<li>Sufficient unit test and integration test is a mandatory part of code change.</li>
</ul>
@@ -625,6 +667,28 @@
<li>Alternatively, you can also manually generate a patch. Please use <code class="highlighter-rouge">git rebase -i</code> first, to combine (squash) smaller commits into a single larger one. Then use <code class="highlighter-rouge">git format-patch</code> command to generate the patch, for a detail guide you can refer to <a href="https://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git/">How to create and apply a patch with Git</a></li>
</ul>
+<h2 id="code-review">Code Review</h2>
+<p>The reviewer need to review the patch from the following perspectives:</p>
+
+<ul>
+ <li><em>Functionality</em>: the patch MUST address the issue and has been verified by the contributor before submitting for review.</li>
+ <li><em>Test coverage</em>: the change MUST be covered by a UT or the Integration test, otherwise it is not maintainable. Execptional case includes GUI, shell script, etc.</li>
+ <li><em>Performance</em>: the change SHOULD NOT downgrade Kylinâs performance.</li>
+ <li><em>Metadata compatibility</em>: the change should support old metadata definition. Otherwise, a metadata migration tool and documentation is required.</li>
+ <li><em>API compatibility</em>: the change SHOULD NOT break public APIâs functionality and behavior; If an old API need be replaced by the new one, print warning message there.</li>
+ <li><em>Documentation</em>: if the Kylin document need be updated together, create another JIRA with âDocumentâ as the component to track. In the document JIRA, attach the doc change patch which is againt the âdocumentâ branch.</li>
+</ul>
+
+<p>A patch which doesnât comply with the above rules may not get merged.</p>
+
+<h2 id="patch-1-policy">Patch +1 Policy</h2>
+
+<p>Patches that fit within the scope of a single component require, at least, a +1 by one of the componentâs owners before commit. If owners are absent â busy or otherwise â two +1s by non-owners but committers will suffice.</p>
+
+<p>Patches that span components need at least two +1s before they can be committed, preferably +1s by owners of components touched by the x-component patch.</p>
+
+<p>Any -1 on a patch by anyone vetoes a patch; it cannot be committed until the justification for the -1 is addressed.</p>
+
<h2 id="apply-patch">Apply Patch</h2>
<ul>
<li>Committer will review Pull Requests and Patches in JIRA regarding correctness, performance, design, coding style, test coverage</li>
Modified: kylin/site/development/index.html
URL: http://svn.apache.org/viewvc/kylin/site/development/index.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/development/index.html (original)
+++ kylin/site/development/index.html Mon Feb 12 08:55:39 2018
@@ -571,12 +571,13 @@
<h3 id="source-repository">Source Repository</h3>
<p>Apache Kylin⢠source code is version controlled using Git version control:<br />
Commits <a href="https://git-wip-us.apache.org/repos/asf?p=kylin.git;a=summary">Summary</a> <br />
-Source Repo: <a href="git://git.apache.org/kylin.git">git://git.apache.org/kylin.git</a> <br />
+Source Repo: <a href="https://git-wip-us.apache.org/repos/asf/kylin.git">https://git-wip-us.apache.org/repos/asf/kylin.git</a> <br />
Mirrored to Github: <a href="https://github.com/apache/kylin">https://github.com/apache/kylin</a></p>
<h3 id="ci-and-code-analysis">CI and Code Analysis</h3>
<p>UT on master branch with JDK 1.7(deprecated): <a href="https://builds.apache.org/job/Kylin-Master-JDK-1.7/">Kylin-Master-JDK-1.7</a> <br />
UT on master branch with JDK 1.8: <a href="https://builds.apache.org/job/Kylin-Master-JDK-1.8/">Kylin-Master-JDK-1.8</a> <br />
+Integretion test within Hadoop Sandbox (HDP 2.4) : <a href="http://34.226.50.254:8081/">http://34.226.50.254:8081/</a><br />
Static Code Analysis: <a href="https://builds.apache.org/analysis/overview?id=org.apache.kylin%3Akylin">SonarCube dashboard</a></p>
<p><a href="https://travis-ci.org/apache/kylin"><img src="https://travis-ci.org/apache/kylin.svg?branch=master" alt="Build Status" /></a><a href="https://www.codacy.com/app/kyligence-git/kylin?utm_source=github.com&utm_medium=referral&utm_content=apache/kylin&utm_campaign=Badge_Grade"><img src="https://api.codacy.com/project/badge/Grade/74f0139786cd4e8a8ce69bb0c17c2e71" alt="Codacy Badge" /></a><br />
@@ -588,10 +589,10 @@ Static Code Analysis: <a href="https://b
<h3 id="roadmap">Roadmap</h3>
<ul>
<li>Hadoop 3.0 support (Erasure Coding)</li>
- <li>Spark cubing enhancement</li>
+ <li>Spark Cubing enhancement</li>
<li>Connect more sources (JDBC, SparkSQL)</li>
- <li>Ad-hoc queries without cubing</li>
- <li>Better storage (Kudu?)</li>
+ <li>Ad-hoc queries without Cubing</li>
+ <li>Better storage (Druid, Kudu, etc)</li>
<li>Real-time analytics with Lambda Architecture</li>
</ul>
Modified: kylin/site/docs21/tutorial/cube_streaming.html
URL: http://svn.apache.org/viewvc/kylin/site/docs21/tutorial/cube_streaming.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/docs21/tutorial/cube_streaming.html (original)
+++ kylin/site/docs21/tutorial/cube_streaming.html Mon Feb 12 08:55:39 2018
@@ -4058,23 +4058,23 @@ bin/kafka-server-start.sh config/server.
<h2 id="create-sample-kafka-topic-and-populate-data">Create sample Kafka topic and populate data</h2>
-<p>Create a sample topic âkylindemoâ, with 3 partitions:</p>
+<p>Create a sample topic âkylin_streaming_topicâ, with 3 partitions:</p>
-<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic kylindemo
-Created topic "kylindemo".</code></pre></div>
+<div class="highlight"><pre><code class="language-groff" data-lang="groff">bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic kylin_streaming_topic
+Created topic "kylin_streaming_topic".</code></pre></div>
<p>Put sample data to this topic; Kylin has an utility class which can do this;</p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">export KAFKA_HOME=/usr/local/kafka_2.10-0.10.0.0
-export KYLIN_HOME=/usr/local/apache-kylin-1.6.0-bin
+export KYLIN_HOME=/usr/local/apache-kylin-2.1.0-bin
cd $KYLIN_HOME
-./bin/kylin.sh org.apache.kylin.source.kafka.util.KafkaSampleProducer --topic kylindemo --broker localhost:9092</code></pre></div>
+./bin/kylin.sh org.apache.kylin.source.kafka.util.KafkaSampleProducer --topic kylin_streaming_topic --broker localhost:9092</code></pre></div>
<p>This tool will send 100 records to Kafka every second. Please keep it running during this tutorial. You can check the sample message with kafka-console-consumer.sh now:</p>
<div class="highlight"><pre><code class="language-groff" data-lang="groff">cd $KAFKA_HOME
-bin/kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic kylindemo --from-beginning
+bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kylin_streaming_topic --from-beginning
{"amount":63.50375137330458,"category":"TOY","order_time":1477415932581,"device":"Other","qty":4,"user":{"id":"bf249f36-f593-4307-b156-240b3094a1c3","age":21,"gender":"Male"},"currency":"USD","country":"CHINA"}
{"amount":22.806058795736583,"category":"ELECTRONIC","order_time":1477415932591,"device":"Andriod","qty":1,"user":{"id":"00283efe-027e-4ec1-bbed-c2bbda873f1d","age":27,"gender":"Female"},"currency":"USD","country":"INDIA"}</code></pre></div>
@@ -4093,7 +4093,7 @@ bin/kafka-console-consumer.sh --zookeepe
<p><img src="/images/tutorial/1.6/Kylin-Cube-Streaming-Tutorial/2_Define_streaming_table.png" alt="" /></p>
-<p>Click âNextâ. On this page, provide the Kafka cluster information; Enter âkylindemoâ as âTopicâ name; The cluster has 1 broker, whose host name is âsandboxâ, port is â9092â, click âSaveâ.</p>
+<p>Click âNextâ. On this page, provide the Kafka cluster information; Enter âkylin_streaming_topicâ as âTopicâ name; The cluster has 1 broker, whose host name is âsandboxâ, port is â9092â, click âSaveâ.</p>
<p><img src="/images/tutorial/1.6/Kylin-Cube-Streaming-Tutorial/3_Kafka_setting.png" alt="" /></p>
Modified: kylin/site/docs23/howto/howto_ldap_and_sso.html
URL: http://svn.apache.org/viewvc/kylin/site/docs23/howto/howto_ldap_and_sso.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/docs23/howto/howto_ldap_and_sso.html (original)
+++ kylin/site/docs23/howto/howto_ldap_and_sso.html Mon Feb 12 08:55:39 2018
@@ -4249,16 +4249,17 @@ ldap.user.groupSearchBase=OU=Group,DC=my
<p>If you have service accounts (e.g, for system integration) which also need be authenticated, configure them in ldap.service.*; Otherwise, leave them be empty;</p>
-<h3 id="configure-the-administrator-group-and-default-role">Configure the administrator group and default role</h3>
+<h3 id="configure-the-administrator-group">Configure the administrator group</h3>
-<p>To map an LDAP group to the admin group in Kylin, need set the âkylin.security.acl.admin-roleâ to âROLE_â + GROUP_NAME. For example, in LDAP the group âKYLIN-ADMIN-GROUPâ is the list of administrators, here need set it as:</p>
+<p>To map an LDAP group to the admin group in Kylin, need set the âkylin.security.acl.admin-roleâ to the LDAP group name(shall keep the original case), and the users in this group will be global admin in Kylin.</p>
-<div class="highlighter-rouge"><pre class="highlight"><code>kylin.security.acl.admin-role=ROLE_KYLIN-ADMIN-GROUP
-kylin.security.acl.default-role=ROLE_ANALYST,ROLE_MODELER
+<p>For example, in LDAP the group âKYLIN-ADMIN-GROUPâ is the list of administrators, here need set it as:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>kylin.security.acl.admin-role=KYLIN-ADMIN-GROUP
</code></pre>
</div>
-<p>The âkylin.security.acl.default-roleâ is a list of the default roles that grant to everyone, keep it as-is.</p>
+<p><em>Attention: When upgrading from Kylin 2.3 ealier version to 2.3 or later, please remove the âROLE_â in this setting as this required in the 2.3 earlier version and keep the group name in original case. And the kylin.security.acl.default-role is deprecated.</em></p>
<h4 id="enable-ldap">Enable LDAP</h4>
Modified: kylin/site/docs23/howto/howto_use_dashboard.html
URL: http://svn.apache.org/viewvc/kylin/site/docs23/howto/howto_use_dashboard.html?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/docs23/howto/howto_use_dashboard.html (original)
+++ kylin/site/docs23/howto/howto_use_dashboard.html Mon Feb 12 08:55:39 2018
@@ -4232,7 +4232,7 @@
<p>To enable Dashboard on WebUI, you need to ensure these are all set:<br />
* Set <strong>kylin.web.dashboard-enabled=true</strong> in <strong>kylin.properties</strong>.<br />
-* Setup system cubes according to <img src="howto_setup_systemcube.html" alt="toturial" />.</p>
+* Setup system cubes according to <a href="howto_setup_systemcube.html">toturial</a>.</p>
<h2 id="how-to-use-it">How to use it</h2>
Modified: kylin/site/feed.xml
URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1823929&r1=1823928&r2=1823929&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Mon Feb 12 08:55:39 2018
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Tue, 06 Feb 2018 05:59:33 -0800</pubDate>
- <lastBuildDate>Tue, 06 Feb 2018 05:59:33 -0800</lastBuildDate>
+ <pubDate>Mon, 12 Feb 2018 00:52:51 -0800</pubDate>
+ <lastBuildDate>Mon, 12 Feb 2018 00:52:51 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>