You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by jh...@apache.org on 2014/10/19 00:32:29 UTC
[1/4] git commit: Update release HOWTO.
Repository: incubator-optiq
Updated Branches:
refs/heads/master 1b16f3c45 -> 59191a7ad
Update release HOWTO.
Project: http://git-wip-us.apache.org/repos/asf/incubator-optiq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-optiq/commit/a47ed3ec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-optiq/tree/a47ed3ec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-optiq/diff/a47ed3ec
Branch: refs/heads/master
Commit: a47ed3ec1f7e0f1a0a0e14c0f754ef2fb369f819
Parents: 4b0a202
Author: Julian Hyde <jh...@apache.org>
Authored: Fri Oct 3 20:18:02 2014 -0700
Committer: Julian Hyde <jh...@apache.org>
Committed: Sat Oct 11 17:53:46 2014 -0700
----------------------------------------------------------------------
README | 2 +-
doc/HOWTO.md | 42 ++++++++++++++++++++++++++++--------------
2 files changed, 29 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/a47ed3ec/README
----------------------------------------------------------------------
diff --git a/README b/README
index 9eaef94..a14c0b9 100644
--- a/README
+++ b/README
@@ -14,7 +14,7 @@ doc/HOWTO.md.
README.md contains examples of running Calcite.
Further information about Apache Calcite is available at its web site,
-http://incubator.apache.org/optiq.
+http://calcite.incubator.apache.org.
Disclaimer
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/a47ed3ec/doc/HOWTO.md
----------------------------------------------------------------------
diff --git a/doc/HOWTO.md b/doc/HOWTO.md
index d37bec0..2d5e797 100644
--- a/doc/HOWTO.md
+++ b/doc/HOWTO.md
@@ -304,6 +304,7 @@ When the dry-run has succeeded, change `install` to `deploy`.
Before you start:
* Set up signing keys as described above.
* Make sure you are using JDK 1.7 (not 1.6 or 1.8).
+* Check that README and README.md have the correct version number.
* Make sure build and tests succeed, including with
-Dcalcite.test.db={mysql,hsqldb}, -Dcalcite.test.slow=true,
-Dcalcite.test.mongodb=true, -Dcalcite.test.splunk=true.
@@ -312,11 +313,18 @@ Before you start:
by merging the latest code into the `julianhyde/coverity_scan` branch,
and when it completes, make sure that there are no important issues.
* Make sure that
- <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20OPTIQ%20AND%20status%20%3D%20Resolved">
+ <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20OPTIQ%20AND%20status%20%3D%20Resolved%20and%20fixVersion%20is%20null">
every "resolved" JIRA case</a> (including duplicates) has
a fix version assigned (most likely the version we are
just about to release)
+Now, set up your environment and do a dry run. The dry run will not
+commit any changes back to git and gives you the opportunity to verify
+that the release process will complete as expected.
+
+If any of the steps fail, clean up (see below), fix the problem, and
+start again from the top.
+
```bash
# Set passphrase variable without putting it into shell history
read GPG_PASSPHRASE
@@ -324,17 +332,10 @@ read GPG_PASSPHRASE
# Make sure that there are no junk files in the sandbox
git clean -xn
-# Set the version numbers
-mvn -DskipTests -DreleaseVersion=X.Y.Z-incubating -DdevelopmentVersion=X.Y.Z+1-incubating-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE}" clean release:prepare 2>&1 | tee /tmp/prepare.log
-
-# Perform the release
-mvn -DskipTests -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE}" clean release:prepare 2>&1 | tee /tmp/perform.log
-
+# Do a dry run of the release:prepare step, which sets version numbers.
+mvn -DdryRun=true -DskipTests -DreleaseVersion=X.Y.Z-incubating -DdevelopmentVersion=X.Y.Z+1-incubating-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE}" clean release:prepare 2>&1 | tee /tmp/prepare-dry.log
```
-If any of the above steps fail, clean up (see below), fix the problem,
-and start again from the top.
-
Check the artifacts:
* In the `target` directory should be these 8 files, among others:
* apache-calcite-X.Y.Z-incubating-src.tar.gz
@@ -361,6 +362,22 @@ Check the artifacts:
`NOTICE` and `git.properties`
* Check PGP, per https://httpd.apache.org/dev/verification.html
+Now, remove the `-DdryRun` flag and run the release for real.
+
+```bash
+# Prepare sets the version numbers, creates a tag, and pushes it to git.
+mvn -DdryRun=false -DskipTests -DreleaseVersion=X.Y.Z-incubating -DdevelopmentVersion=X.Y.Z+1-incubating-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE}" clean release:prepare 2>&1 | tee /tmp/prepare.log
+
+# Perform checks out the tagged version, builds, and deploys to the staging repository
+mvn -DskipTests -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE}" clean release:perform 2>&1 | tee /tmp/perform.log
+```
+
+Verify the staged artifacts in the Nexus repository:
+* Go to https://repository.apache.org/
+* Enterprise → Staging
+* Staging tab → Name column → org.apache.calcite
+* Navigate through the artifact tree and make sure the .jar, .pom, .asc files are present
+
Upload the artifacts to a staging area (in this case, your
people.apache.org home directory):
@@ -401,9 +418,6 @@ git reset --hard HEAD
gpg --recv-keys key
# Check keys
-curl http://people.apache.org/keys/group/calcite.asc > KEYS
-
-# Check keys
curl -O https://dist.apache.org/repos/dist/release/incubator/calcite/KEYS
# Sign/check md5 and sha1 hashes
@@ -436,7 +450,7 @@ function checkHash() {
fi
done
}
-checkHash apache-calcite-X.Y.Z-incubating
+checkHash apache-calcite-X.Y.Z-incubating-rcN
```
## Get approval for a release via Apache voting process (for Calcite committers)
[2/4] git commit: Add tests for windowed aggregates without ORDER BY.
(Works already.)
Posted by jh...@apache.org.
Add tests for windowed aggregates without ORDER BY. (Works already.)
Project: http://git-wip-us.apache.org/repos/asf/incubator-optiq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-optiq/commit/4b0a2026
Tree: http://git-wip-us.apache.org/repos/asf/incubator-optiq/tree/4b0a2026
Diff: http://git-wip-us.apache.org/repos/asf/incubator-optiq/diff/4b0a2026
Branch: refs/heads/master
Commit: 4b0a20268ad34fe363142da05df52a722b8437d4
Parents: 1b16f3c
Author: Julian Hyde <jh...@apache.org>
Authored: Thu Oct 2 17:37:34 2014 -0700
Committer: Julian Hyde <jh...@apache.org>
Committed: Sat Oct 11 17:53:46 2014 -0700
----------------------------------------------------------------------
core/src/test/resources/sql/winagg.oq | 36 ++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/4b0a2026/core/src/test/resources/sql/winagg.oq
----------------------------------------------------------------------
diff --git a/core/src/test/resources/sql/winagg.oq b/core/src/test/resources/sql/winagg.oq
index 7dc2c6a..4a4b23a 100644
--- a/core/src/test/resources/sql/winagg.oq
+++ b/core/src/test/resources/sql/winagg.oq
@@ -168,4 +168,40 @@ select *, count(*) over (order by deptno), first_value(ename) over (order by dep
!ok
!}
+# Without ORDER BY
+select *, count(*) over (partition by deptno) as c from emp;
+ ENAME | DEPTNO | GENDER | C
+-------+--------+--------+---
+ Adam | 50 | M | 2
+ Alice | 30 | F | 2
+ Bob | 10 | M | 2
+ Eric | 20 | M | 1
+ Eve | 50 | F | 2
+ Grace | 60 | F | 1
+ Jane | 10 | F | 2
+ Susan | 30 | F | 2
+ Wilma | | F | 1
+(9 rows)
+
+!ok
+# No ORDER BY, windows defined in WINDOW clause.
+select deptno, gender, min(gender) over w1 as a, min(gender) over w2 as d
+from emp
+window w1 as (),
+ w2 as (partition by deptno);
+ DEPTNO | GENDER | A | D
+--------+--------+---+---
+ 10 | F | F | F
+ 10 | M | F | F
+ 20 | M | F | M
+ 30 | F | F | F
+ 30 | F | F | F
+ 50 | F | F | F
+ 50 | M | F | F
+ 60 | F | F | F
+ | F | F | F
+(9 rows)
+
+!ok
+
# End winagg.oq
[3/4] git commit: Add test case for LIKE ... ESCAPE.
Posted by jh...@apache.org.
Add test case for LIKE ... ESCAPE.
Project: http://git-wip-us.apache.org/repos/asf/incubator-optiq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-optiq/commit/ee903d56
Tree: http://git-wip-us.apache.org/repos/asf/incubator-optiq/tree/ee903d56
Diff: http://git-wip-us.apache.org/repos/asf/incubator-optiq/diff/ee903d56
Branch: refs/heads/master
Commit: ee903d567b822498d17a1ba162dfb6ffc5990960
Parents: a47ed3e
Author: Julian Hyde <ju...@gmail.com>
Authored: Wed Oct 15 17:43:39 2014 -0400
Committer: Julian Hyde <ju...@gmail.com>
Committed: Wed Oct 15 17:49:29 2014 -0400
----------------------------------------------------------------------
.../org/eigenbase/sql/test/SqlOperatorBaseTest.java | 12 ++++++++++++
1 file changed, 12 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/ee903d56/core/src/test/java/org/eigenbase/sql/test/SqlOperatorBaseTest.java
----------------------------------------------------------------------
diff --git a/core/src/test/java/org/eigenbase/sql/test/SqlOperatorBaseTest.java b/core/src/test/java/org/eigenbase/sql/test/SqlOperatorBaseTest.java
index a51bc9b..c1ce991 100644
--- a/core/src/test/java/org/eigenbase/sql/test/SqlOperatorBaseTest.java
+++ b/core/src/test/java/org/eigenbase/sql/test/SqlOperatorBaseTest.java
@@ -2980,6 +2980,18 @@ public abstract class SqlOperatorBaseTest {
tester.checkBoolean("'abc' not like '_b_'", Boolean.FALSE);
}
+ @Test public void testLikeEscape() {
+ tester.setFor(SqlStdOperatorTable.LIKE);
+ tester.checkBoolean("'a_c' like 'a#_c' escape '#'", Boolean.TRUE);
+ tester.checkBoolean("'axc' like 'a#_c' escape '#'", Boolean.FALSE);
+ tester.checkBoolean("'a_c' like 'a\\_c' escape '\\'", Boolean.TRUE);
+ tester.checkBoolean("'axc' like 'a\\_c' escape '\\'", Boolean.FALSE);
+ tester.checkBoolean("'a%c' like 'a\\%c' escape '\\'", Boolean.TRUE);
+ tester.checkBoolean("'a%cde' like 'a\\%c_e' escape '\\'", Boolean.TRUE);
+ tester.checkBoolean("'abbc' like 'a%c' escape '\\'", Boolean.TRUE);
+ tester.checkBoolean("'abbc' like 'a\\%c' escape '\\'", Boolean.FALSE);
+ }
+
@Test public void testLikeOperator() {
tester.setFor(SqlStdOperatorTable.LIKE);
tester.checkBoolean("'' like ''", Boolean.TRUE);
[4/4] git commit: [OPTIQ-438] Push predicates through SemiJoinRel
Posted by jh...@apache.org.
[OPTIQ-438] Push predicates through SemiJoinRel
Project: http://git-wip-us.apache.org/repos/asf/incubator-optiq/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-optiq/commit/59191a7a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-optiq/tree/59191a7a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-optiq/diff/59191a7a
Branch: refs/heads/master
Commit: 59191a7add9f3141138d7c2cb5298939a0fa99e0
Parents: ee903d5
Author: John Pullokkaran <jp...@hortonworks.com>
Authored: Fri Oct 17 11:45:33 2014 -0400
Committer: Julian Hyde <ju...@gmail.com>
Committed: Fri Oct 17 11:45:33 2014 -0400
----------------------------------------------------------------------
.../rel/rules/PushFilterPastJoinRule.java | 37 ++++++++++++++++++--
.../java/org/eigenbase/relopt/RelOptUtil.java | 4 ++-
.../org/eigenbase/test/RelOptRulesTest.java | 22 ++++++++++++
.../org/eigenbase/test/RelOptRulesTest.xml | 29 +++++++++++++++
4 files changed, 88 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/59191a7a/core/src/main/java/org/eigenbase/rel/rules/PushFilterPastJoinRule.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/eigenbase/rel/rules/PushFilterPastJoinRule.java b/core/src/main/java/org/eigenbase/rel/rules/PushFilterPastJoinRule.java
index 969e93c..90369ad 100644
--- a/core/src/main/java/org/eigenbase/rel/rules/PushFilterPastJoinRule.java
+++ b/core/src/main/java/org/eigenbase/rel/rules/PushFilterPastJoinRule.java
@@ -24,6 +24,7 @@ import org.eigenbase.rex.*;
import org.eigenbase.util.Holder;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Sets;
/**
* PushFilterPastJoinRule implements the rule for pushing filters above and
@@ -69,11 +70,9 @@ public abstract class PushFilterPastJoinRule extends RelOptRule {
protected void perform(RelOptRuleCall call, FilterRelBase filter,
JoinRelBase join) {
- if (join instanceof SemiJoinRel) {
- return;
- }
final List<RexNode> joinFilters =
RelOptUtil.conjunctions(join.getCondition());
+ final List<RexNode> origJoinFilters = ImmutableList.copyOf(joinFilters);
// If there is only the joinRel,
// make sure it does not match a cartesian product joinRel
@@ -117,6 +116,19 @@ public abstract class PushFilterPastJoinRule extends RelOptRule {
filterPushed = true;
}
+ // Move join filters up if needed
+ validateJoinFilters(aboveFilters, joinFilters, join);
+
+ // If no filter got pushed after validate reset filterPushed flag
+ if (leftFilters.isEmpty()
+ && rightFilters.isEmpty()
+ && joinFilters.size() == origJoinFilters.size()) {
+ if (Sets.newHashSet(joinFilters)
+ .equals(Sets.newHashSet(origJoinFilters))) {
+ filterPushed = false;
+ }
+ }
+
// Try to push down filters in ON clause. A ON clause filter can only be
// pushed down if it does not affect the non-matching set, i.e. it is
// not on the side which is preserved.
@@ -204,6 +216,25 @@ public abstract class PushFilterPastJoinRule extends RelOptRule {
}
/**
+ * Validates that target execution framework can satisfy join filters.
+ *
+ * <p>If the join filter cannot be satifisfied (for example, if it is
+ * {@code l.c1 > r.c2} and the join only supports equi-join), removes the
+ * filter from {@code joinFilters} and adds it to {@code aboveFilters}.
+ *
+ * <p>The default implementation does nothing; i.e. the join can handle all
+ * conditions.
+ *
+ * @param aboveFilters Filter above Join
+ * @param joinFilters Filters in join condition
+ * @param join Join
+ */
+ protected void validateJoinFilters(List<RexNode> aboveFilters,
+ List<RexNode> joinFilters, JoinRelBase join) {
+ return;
+ }
+
+ /**
* If the filter list passed in is non-empty, creates a FilterRel on top of
* the existing RelNode; otherwise, just returns the RelNode
*
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/59191a7a/core/src/main/java/org/eigenbase/relopt/RelOptUtil.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/eigenbase/relopt/RelOptUtil.java b/core/src/main/java/org/eigenbase/relopt/RelOptUtil.java
index bd9c306..d5ceaa0 100644
--- a/core/src/main/java/org/eigenbase/relopt/RelOptUtil.java
+++ b/core/src/main/java/org/eigenbase/relopt/RelOptUtil.java
@@ -1960,7 +1960,9 @@ public abstract class RelOptUtil {
final List<RelDataTypeField> rightFields =
joinRel.getInputs().get(1).getRowType().getFieldList();
final int nFieldsRight = rightFields.size();
- assert nTotalFields == nSysFields + nFieldsLeft + nFieldsRight;
+ assert nTotalFields == (joinRel instanceof SemiJoinRel
+ ? nSysFields + nFieldsLeft
+ : nSysFields + nFieldsLeft + nFieldsRight);
// set the reference bitmaps for the left and right children
BitSet leftBitmap =
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/59191a7a/core/src/test/java/org/eigenbase/test/RelOptRulesTest.java
----------------------------------------------------------------------
diff --git a/core/src/test/java/org/eigenbase/test/RelOptRulesTest.java b/core/src/test/java/org/eigenbase/test/RelOptRulesTest.java
index 5c5513e..8668907 100644
--- a/core/src/test/java/org/eigenbase/test/RelOptRulesTest.java
+++ b/core/src/test/java/org/eigenbase/test/RelOptRulesTest.java
@@ -192,6 +192,28 @@ public class RelOptRulesTest extends RelOptTestBase {
+ " and emp.sal > 100)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/OPTIQ-438">[OPTIQ-438],
+ * Push predicates through SemiJoinRel</a>. */
+ @Test public void testPushFilterThroughSemiJoin() {
+ final HepProgram preProgram =
+ HepProgram.builder()
+ .addRuleInstance(SemiJoinRule.INSTANCE)
+ .build();
+
+ final HepProgram program =
+ HepProgram.builder()
+ .addRuleInstance(PushFilterPastProjectRule.INSTANCE)
+ .addRuleInstance(PushFilterPastJoinRule.FILTER_ON_JOIN)
+ .addRuleInstance(PushFilterPastJoinRule.JOIN)
+ .build();
+ checkPlanning(tester.withDecorrelation(true).withTrim(false), preProgram,
+ new HepPlanner(program),
+ "select * from (select * from dept where dept.deptno in (\n"
+ + " select emp.deptno from emp\n"
+ + " ))R where R.deptno <=10 ");
+ }
+
protected void semiJoinTrim() {
final DiffRepository diffRepos = getDiffRepos();
String sql = diffRepos.expand(null, "${sql}");
http://git-wip-us.apache.org/repos/asf/incubator-optiq/blob/59191a7a/core/src/test/resources/org/eigenbase/test/RelOptRulesTest.xml
----------------------------------------------------------------------
diff --git a/core/src/test/resources/org/eigenbase/test/RelOptRulesTest.xml b/core/src/test/resources/org/eigenbase/test/RelOptRulesTest.xml
index a4d2d31..4ef190a 100644
--- a/core/src/test/resources/org/eigenbase/test/RelOptRulesTest.xml
+++ b/core/src/test/resources/org/eigenbase/test/RelOptRulesTest.xml
@@ -108,6 +108,35 @@ ProjectRel(EXPR$0=[1])
]]>
</Resource>
</TestCase>
+ <TestCase name="testPushFilterThroughSemiJoin">
+ <Resource name="sql">
+ <![CDATA[select * from (select * from dept where dept.deptno in (select emp.deptno from emp))R where R.deptno <=10 ]]>
+ </Resource>
+ <Resource name="planBefore">
+ <![CDATA[
+ProjectRel(DEPTNO=[$0], NAME=[$1])
+ FilterRel(condition=[<=($0, 10)])
+ ProjectRel(DEPTNO=[$0], NAME=[$1])
+ SemiJoinRel(condition=[=($2, $3)], joinType=[inner])
+ ProjectRel($f0=[$0], $f1=[$1], $f2=[$0])
+ TableAccessRel(table=[[CATALOG, SALES, DEPT]])
+ ProjectRel(DEPTNO=[$7])
+ TableAccessRel(table=[[CATALOG, SALES, EMP]])
+]]>
+ </Resource>
+ <Resource name="planAfter">
+ <![CDATA[
+ProjectRel(DEPTNO=[$0], NAME=[$1])
+ ProjectRel(DEPTNO=[$0], NAME=[$1])
+ SemiJoinRel(condition=[=($2, $3)], joinType=[inner])
+ FilterRel(condition=[<=($0, 10)])
+ ProjectRel($f0=[$0], $f1=[$1], $f2=[$0])
+ TableAccessRel(table=[[CATALOG, SALES, DEPT]])
+ ProjectRel(DEPTNO=[$7])
+ TableAccessRel(table=[[CATALOG, SALES, EMP]])
+]]>
+ </Resource>
+ </TestCase>
<TestCase name="testPushFilterPastAgg">
<Resource name="sql">
<![CDATA[select dname, c from (select name dname, count(*) as c from dept group by name) t where dname = 'Charlie']]>