You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by jh...@apache.org on 2014/08/28 23:52:14 UTC
svn commit: r1621213 - /incubator/optiq/site/index.html
Author: jhyde
Date: Thu Aug 28 21:52:14 2014
New Revision: 1621213
URL: http://svn.apache.org/r1621213
Log:
Simple site.
Added:
incubator/optiq/site/index.html
Added: incubator/optiq/site/index.html
URL: http://svn.apache.org/viewvc/incubator/optiq/site/index.html?rev=1621213&view=auto
==============================================================================
--- incubator/optiq/site/index.html (added)
+++ incubator/optiq/site/index.html Thu Aug 28 21:52:14 2014
@@ -0,0 +1,227 @@
+<html>
+<head>
+ <title>Apache Optiq</title>
+</head>
+<body>
+<h1>
+<a name="user-content-apache-optiq" class="anchor" href="#apache-optiq" aria-hidden="true"><span class="octicon octicon-link"></span></a>Apache Optiq</h1>
+
+<p>Apache Optiq is a dynamic data management framework.</p>
+
+<h2>
+<a name="user-content-getting-optiq" class="anchor" href="#getting-optiq" aria-hidden="true"><span class="octicon octicon-link"></span></a>Getting Optiq</h2>
+
+<p>To run Apache Optiq, you can either
+<a href="http://www.apache.org/dyn/closer.cgi/incubator/optiq">download a release</a>
+or <a href="https://github.com/apache/incubator-optiq/blob/master/doc/HOWTO.md#building-from-a-source-distribution">build from the source code</a>.</p>
+
+<h2>
+<a name="user-content-example" class="anchor" href="#example" aria-hidden="true"><span class="octicon octicon-link"></span></a>Example</h2>
+
+<p>Optiq makes data anywhere, of any format, look like a database. For
+example, you can execute a complex ANSI-standard SQL statement on
+in-memory collections:</p>
+
+<div class="highlight highlight-java"><pre><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">HrSchema</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kd">final</span> <span class="n">Employee</span><span class="o">[]</span> <span class="n">emps</span> <span class="o">=</span> <span class="o">...</span> <span class="o">;</span>
+ <span class="kd">public</span> <span class="kd">final</span> <span class="n">Department</span><span class="o">[]</span> <span class="n">depts</span> <span class="o">=</span> <span class="o">...;</span>
+<span class="o">}</span>
+
+<span class="n">Class</span><span class="o">.</span><span class="na">forName</span><span class="o">(</span><span class="s">"net.hydromatic.optiq.jdbc.Driver"</span><span class="o">);</span>
+<span class="n">Connection</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">DriverManager</span><span class="o">.</span><span class="na">getConnection</span><span class="o">(</span><span class="s">"jdbc:optiq:"</span><span class="o">);</span>
+<span class="n">OptiqConnection</span> <span class="n">optiqConnection</span> <span class="o">=</span>
+ <span class="n">connection</span><span class="o">.</span><span class="na">unwrap</span><span class="o">(</span><span class="n">OptiqConnection</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+<span class="n">ReflectiveSchema</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">optiqConnection</span><span class="o">,</span>
+ <span class="n">optiqConnection</span><span class="o">.</span><span class="na">getRootSchema</span><span class="o">(),</span> <span class="s">"hr"</span><span class="o">,</span> <span class="k">new</span> <span class="nf">HrSchema</span><span class="o">());</span>
+<span class="n">Statement</span> <span class="n">statement</span> <span class="o">=</span> <span class="n">optiqConnection</span><span class="o">.</span><span class="na">createStatement</span><span class="o">();</span>
+<span class="n">ResultSet</span> <span class="n">resultSet</span> <span class="o">=</span> <span class="n">statement</span><span class="o">.</span><span class="na">executeQuery</span><span class="o">(</span>
+ <span class="s">"select d.\"deptno\", min(e.\"empid\")\n"</span>
+ <span class="o">+</span> <span class="s">"from \"hr\".\"emps\" as e\n"</span>
+ <span class="o">+</span> <span class="s">"join \"hr\".\"depts\" as d\n"</span>
+ <span class="o">+</span> <span class="s">" on e.\"deptno\" = d.\"deptno\"\n"</span>
+ <span class="o">+</span> <span class="s">"group by d.\"deptno\"\n"</span>
+ <span class="o">+</span> <span class="s">"having count(*) > 1"</span><span class="o">);</span>
+<span class="n">print</span><span class="o">(</span><span class="n">resultSet</span><span class="o">);</span>
+<span class="n">resultSet</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+<span class="n">statement</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+<span class="n">connection</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+</pre></div>
+
+<p>Where is the database? There is no database. The connection is
+completely empty until <code>ReflectiveSchema.create</code> registers
+a Java object as a schema and its collection fields <code>emps</code>
+and <code>depts</code> as tables.</p>
+
+<p>Optiq does not want to own data; it does not even have favorite data
+format. This example used in-memory data sets, and processed them
+using operators such as <code>groupBy</code> and <code>join</code>
+from the linq4j
+library. But Optiq can also process data in other data formats, such
+as JDBC. In the first example, replace</p>
+
+<div class="highlight highlight-java"><pre><span class="n">ReflectiveSchema</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">optiqConnection</span><span class="o">,</span>
+ <span class="n">optiqConnection</span><span class="o">.</span><span class="na">getRootSchema</span><span class="o">(),</span> <span class="s">"hr"</span><span class="o">,</span> <span class="k">new</span> <span class="nf">HrSchema</span><span class="o">());</span>
+</pre></div>
+
+<p>with</p>
+
+<div class="highlight highlight-java"><pre><span class="n">Class</span><span class="o">.</span><span class="na">forName</span><span class="o">(</span><span class="s">"com.mysql.jdbc.Driver"</span><span class="o">);</span>
+<span class="n">BasicDataSource</span> <span class="n">dataSource</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">BasicDataSource</span><span class="o">();</span>
+<span class="n">dataSource</span><span class="o">.</span><span class="na">setUrl</span><span class="o">(</span><span class="s">"jdbc:mysql://localhost"</span><span class="o">);</span>
+<span class="n">dataSource</span><span class="o">.</span><span class="na">setUsername</span><span class="o">(</span><span class="s">"sa"</span><span class="o">);</span>
+<span class="n">dataSource</span><span class="o">.</span><span class="na">setPassword</span><span class="o">(</span><span class="s">""</span><span class="o">);</span>
+<span class="n">JdbcSchema</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">optiqConnection</span><span class="o">,</span> <span class="n">dataSource</span><span class="o">,</span> <span class="n">rootSchema</span><span class="o">,</span> <span class="s">"hr"</span><span class="o">,</span> <span class="s">""</span><span class="o">);</span>
+</pre></div>
+
+<p>and Optiq will execute the same query in JDBC. To the application, the
+data and API are the same, but behind the scenes the implementation is
+very different. Optiq uses optimizer rules
+to push the <code>JOIN</code> and <code>GROUP BY</code> operations to
+the source database.</p>
+
+<p>In-memory and JDBC are just two familiar examples. Optiq can handle
+any data source and data format. To add a data source, you need to
+write an adapter that tells Optiq
+what collections in the data source it should consider "tables".</p>
+
+<p>For more advanced integration, you can write optimizer
+rules. Optimizer rules allow Optiq to access data of a new format,
+allow you to register new operators (such as a better join algorithm),
+and allow Optiq to optimize how queries are translated to
+operators. Optiq will combine your rules and operators with built-in
+rules and operators, apply cost-based optimization, and generate an
+efficient plan.</p>
+
+<h3>
+<a name="user-content-non-jdbc-access" class="anchor" href="#non-jdbc-access" aria-hidden="true"><span class="octicon octicon-link"></span></a>Non-JDBC access</h3>
+
+<p>Optiq also allows front-ends other than SQL/JDBC. For example, you can
+execute queries in <a href="https://github.com/julianhyde/linq4j">linq4j</a>:</p>
+
+<div class="highlight highlight-java"><pre><span class="kd">final</span> <span class="n">OptiqConnection</span> <span class="n">connection</span> <span class="o">=</span> <span class="o">...;</span>
+<span class="n">ParameterExpression</span> <span class="n">c</span> <span class="o">=</span> <span class="n">Expressions</span><span class="o">.</span><span class="na">parameter</span><span class="o">(</span><span class="n">Customer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="s">"c"</span><span class="o">);</span>
+<span class="k">for</span> <span class="o">(</span><span class="n">Customer</span> <span class="n">customer</span>
+ <span class="o">:</span> <span class="n">connection</span><span class="o">.</span><span class="na">getRootSchema</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">getSubSchema</span><span class="o">(</span><span class="s">"foodmart"</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">getTable</span><span class="o">(</span><span class="s">"customer"</span><span class="o">,</span> <span class="n">Customer</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">where</span><span class="o">(</span>
+ <span class="n">Expressions</span><span class="o">.<</span><span class="n">Predicate1</span><span class="o"><</span><span class="n">Customer</span><span class="o">>></span><span class="n">lambda</span><span class="o">(</span>
+ <span class="n">Expressions</span><span class="o">.</span><span class="na">lessThan</span><span class="o">(</span>
+ <span class="n">Expressions</span><span class="o">.</span><span class="na">field</span><span class="o">(</span><span class="n">c</span><span class="o">,</span> <span class="s">"customer_id"</span><span class="o">),</span>
+ <span class="n">Expressions</span><span class="o">.</span><span class="na">constant</span><span class="o">(</span><span class="mi">5</span><span class="o">)),</span>
+ <span class="n">c</span><span class="o">)))</span> <span class="o">{</span>
+ <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">name</span><span class="o">);</span>
+<span class="o">}</span>
+</pre></div>
+
+<p>Linq4j understands the full query parse tree, and the Linq4j query
+provider for Optiq invokes Optiq as an query optimizer. If the
+<code>customer</code> table comes from a JDBC database (based on
+this code fragment, we really can't tell) then the optimal plan
+will be to send the query</p>
+
+<div class="highlight highlight-SQL"><pre><span class="k">SELECT</span> <span class="o">*</span>
+<span class="k">FROM</span> <span class="ss">"customer"</span>
+<span class="k">WHERE</span> <span class="ss">"customer_id"</span> <span class="o"><</span> <span class="mi">5</span>
+</pre></div>
+
+<p>to the JDBC data source.</p>
+
+<h3>
+<a name="user-content-writing-an-adapter" class="anchor" href="#writing-an-adapter" aria-hidden="true"><span class="octicon octicon-link"></span></a>Writing an adapter</h3>
+
+<p>The <a href="https://github.com/julianhyde/optiq-csv">optiq-csv</a>
+project provides a CSV adapter, which is fully functional for use in applications
+but is also simple enough to serve as a good template if you are writing
+your own adapter.</p>
+
+<p>See the <a href="https://github.com/julianhyde/optiq-csv/blob/master/TUTORIAL.md">optiq-csv tutorial</a>
+for information on using optiq-csv and writing adapters.</p>
+
+<p>See the <a href="https://github.com/apache/incubator-optiq/blob/master/doc/HOWTO.md">HOWTO</a> for more information about using other
+adapters, and about using Optiq in general.</p>
+
+<h2>
+<a name="user-content-status" class="anchor" href="#status" aria-hidden="true"><span class="octicon octicon-link"></span></a>Status</h2>
+
+<p>The following features are complete.</p>
+
+<ul class="task-list">
+<li>Query parser, validator and optimizer</li>
+<li>Support for reading models in JSON format</li>
+<li>Many standard functions and aggregate functions</li>
+<li>JDBC queries against Linq4j and JDBC back-ends</li>
+<li>
+<a href="https://github.com/julianhyde/linq4j">Linq4j</a> front-end</li>
+<li>SQL features: SELECT, FROM (including JOIN syntax), WHERE, GROUP BY (and aggregate functions including COUNT(DISTINCT ...)), HAVING, ORDER BY (including NULLS FIRST/LAST), set operations (UNION, INTERSECT, MINUS), sub-queries (including correlated sub-queries), windowed aggregates, LIMIT (syntax as <a href="http://www.postgresql.org/docs/8.4/static/sql-select.html#SQL-LIMIT">Postgres</a>)</li>
+</ul><p>For more details, see the <a href="https://github.com/apache/incubator-optiq/blob/master/doc/REFERENCE.md">Reference guide</a>.</p>
+
+<h3>
+<a name="user-content-drivers" class="anchor" href="#drivers" aria-hidden="true"><span class="octicon octicon-link"></span></a>Drivers</h3>
+
+<ul class="task-list">
+<li><a href="http://www.hydromatic.net/optiq/optiq-core/apidocs/net/hydromatic/optiq/jdbc/package-summary.html">JDBC driver</a></li>
+</ul><h3>
+<a name="user-content-adapters" class="anchor" href="#adapters" aria-hidden="true"><span class="octicon octicon-link"></span></a>Adapters</h3>
+
+<ul class="task-list">
+<li><a href="https://github.com/apache/incubator-drill">Apache Drill adapter</a></li>
+<li>Cascading adapter (<a href="https://github.com/Cascading/lingual">Lingual</a>)</li>
+<li>CSV adapter (<a href="https://github.com/julianhyde/optiq-csv">optiq-csv</a>)</li>
+<li>JDBC adapter (part of <a href="http://www.hydromatic.net/optiq/optiq-core/apidocs/net/hydromatic/optiq/impl/jdbc/package-summary.html">optiq-core</a>)</li>
+<li>MongoDB adapter (<a href="http://www.hydromatic.net/optiq/optiq-mongodb/apidocs/net/hydromatic/optiq/impl/mongodb/package-summary.html">optiq-mongodb</a>)</li>
+<li>Spark adapter (<a href="http://www.hydromatic.net/optiq/optiq-spark/apidocs/net/hydromatic/optiq/impl/spark/package-summary.html">optiq-spark</a>)</li>
+<li>Splunk adapter (<a href="http://www.hydromatic.net/optiq/optiq-splunk/apidocs/net/hydromatic/optiq/impl/splunk/package-summary.html">optiq-splunk</a>)</li>
+<li>Eclipse Memory Analyzer (MAT) adapter (<a href="https://github.com/vlsi/optiq-mat-plugin">optiq-mat-plugin</a>)</li>
+</ul><h2>
+<a name="user-content-more-information" class="anchor" href="#more-information" aria-hidden="true"><span class="octicon octicon-link"></span></a>More information</h2>
+
+<ul class="task-list">
+<li>License: Apache License, Version 2.0.</li>
+<li>Author: Julian Hyde</li>
+<li>Blog: <a href="http://julianhyde.blogspot.com">http://julianhyde.blogspot.com</a>
+</li>
+<li>Incubation status page: <a href="http://incubator.apache.org/projects/optiq.html">http://incubator.apache.org/projects/optiq.html</a>
+</li>
+<li>Project page: <a href="http://incubator.apache.org/optiq">http://incubator.apache.org/optiq</a>
+</li>
+<li>Source code: <a href="http://github.com/apache/incubator-optiq">http://github.com/julianhyde/optiq</a>
+</li>
+<li>Issues: <a href="https://issues.apache.org/jira/browse/OPTIQ">Apache JIRA</a>
+</li>
+<li>Developers list: <a href="mailto:dev@optiq.incubator.apache.org">dev at optiq.incubator.apache.org</a> (<a href="http://mail-archives.apache.org/mod_mbox/incubator-optiq-dev/">archive</a>, <a href="mailto:dev-subscribe@optiq.incubator.apache.org">subscribe</a>)</li>
+<li><a href="https://github.com/apache/incubator-optiq/blob/master/doc/HOWTO.md">HOWTO</a></li>
+<li><a href="https://github.com/apache/incubator-optiq/blob/master/doc/MODEL.md">JSON model</a></li>
+<li><a href="https://github.com/apache/incubator-optiq/blob/master/doc/REFERENCE.md">Reference guide</a></li>
+<li><a href="https://github.com/apache/incubator-optiq/blob/master/doc/HISTORY.md">Release notes and history</a></li>
+</ul>
+
+<h3><a name="user-content-presentations" class="anchor" href="#presentations" aria-hidden="true"><span class="octicon octicon-link"></span></a>Presentations</h3>
+
+<ul class="task-list">
+<li>
+<a href="http://www.slideshare.net/julianhyde/how-to-integrate-splunk-with-any-data-solution">How to integrate Splunk with any data solution</a> (Splunk User Conference, 2012)</li>
+<li>
+<a href="https://github.com/julianhyde/share/blob/master/slides/optiq-drill-user-group-2013.pdf?raw=true">Drill / SQL / Optiq</a> (2013)</li>
+<li>
+<a href="https://github.com/julianhyde/share/blob/master/slides/optiq-richrelevance-2013.pdf?raw=true">SQL on Big Data using Optiq</a> (2013)</li>
+<li>
+<a href="https://github.com/julianhyde/share/blob/master/slides/optiq-nosql-now-2013.pdf?raw=true">SQL Now!</a> (NoSQL Now! conference, 2013)</li>
+<li>
+<a href="https://github.com/julianhyde/share/blob/master/slides/hive-cbo-summit-2014.pdf?raw=true">Cost-based optimization in Hive</a> (Hadoop Summit, 2014)</li>
+<li>
+<a href="https://github.com/julianhyde/share/blob/master/slides/dmmq-summit-2014.pdf?raw=true">Discardable, in-memory materialized query for Hadoop</a> (Hadoop Summit, 2014)</li>
+</ul><h2>
+<a name="user-content-disclaimer" class="anchor" href="#disclaimer" aria-hidden="true"><span class="octicon octicon-link"></span></a>Disclaimer</h2>
+
+<p>Apache Optiq is an effort undergoing incubation at The Apache Software
+Foundation (ASF), sponsored by the Apache Incubator. Incubation is
+required of all newly accepted projects until a further review
+indicates that the infrastructure, communications, and decision making
+process have stabilized in a manner consistent with other successful
+ASF projects. While incubation status is not necessarily a reflection
+of the completeness or stability of the code, it does indicate that
+ the project has yet to be fully endorsed by the ASF.</p>
+
+</body>
+</html>