You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2023/09/29 14:07:25 UTC
[arrow-datafusion] branch asf-site updated: Publish built docs triggered by 6ed2e84da8154931c393a9885b80c4f8a9d47ebb
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cb2323a6e1 Publish built docs triggered by 6ed2e84da8154931c393a9885b80c4f8a9d47ebb
cb2323a6e1 is described below
commit cb2323a6e11a651d25e83f31c2cd614054c754b7
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Sep 29 14:07:20 2023 +0000
Publish built docs triggered by 6ed2e84da8154931c393a9885b80c4f8a9d47ebb
---
.../custom-table-providers.md.txt | 16 +++++++++
library-user-guide/custom-table-providers.html | 39 ++++++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 56 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/custom-table-providers.md.txt b/_sources/library-user-guide/custom-table-providers.md.txt
index 30721d6a5b..2de4b1ba05 100644
--- a/_sources/library-user-guide/custom-table-providers.md.txt
+++ b/_sources/library-user-guide/custom-table-providers.md.txt
@@ -121,6 +121,22 @@ impl TableProvider for CustomDataSource {
With this, and the implementation of the omitted methods, we can now use the `CustomDataSource` as a `TableProvider` in DataFusion.
+##### Additional `TableProvider` Methods
+
+`scan` has no default implementation, so it needed to be written. There are other methods on the `TableProvider` that have default implementations, but can be overridden if needed to provide additional functionality.
+
+###### `supports_filters_pushdown`
+
+The `supports_filters_pushdown` method can be overridden to indicate which filter expressions support being pushed down to the data source and within that the specificity of the pushdown.
+
+This returns a `Vec` of `TableProviderFilterPushDown` enums where each enum represents a filter that can be pushed down. The `TableProviderFilterPushDown` enum has three variants:
+
+- `TableProviderFilterPushDown::Unsupported` - the filter cannot be pushed down
+- `TableProviderFilterPushDown::Exact` - the filter can be pushed down and the data source can guarantee that the filter will be applied completely to all rows. This is the highest performance option.
+- `TableProviderFilterPushDown::Inexact` - the filter can be pushed down, but the data source cannot guarantee that the filter will be applied to all rows. DataFusion will apply `Inexact` filters again after the scan to ensure correctness.
+
+For filters that can be pushed down, they'll be passed to the `scan` method as the `filters` parameter and they can be made use of there.
+
## Using the Custom Table Provider
In order to use the custom table provider, we need to register it with DataFusion. This is done by creating a `TableProvider` and registering it with the `ExecutionContext`.
diff --git a/library-user-guide/custom-table-providers.html b/library-user-guide/custom-table-providers.html
index 98eaa85d91..d944b6a7f1 100644
--- a/library-user-guide/custom-table-providers.html
+++ b/library-user-guide/custom-table-providers.html
@@ -365,6 +365,30 @@
<a class="reference internal nav-link" href="#scan-revisited">
Scan Revisited
</a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h5 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#additional-tableprovider-methods">
+ Additional
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ TableProvider
+ </span>
+ </code>
+ Methods
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h6 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#supports-filters-pushdown">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ supports_filters_pushdown
+ </span>
+ </code>
+ </a>
+ </li>
+ </ul>
+ </li>
+ </ul>
</li>
</ul>
</li>
@@ -523,6 +547,21 @@
</pre></div>
</div>
<p>With this, and the implementation of the omitted methods, we can now use the <code class="docutils literal notranslate"><span class="pre">CustomDataSource</span></code> as a <code class="docutils literal notranslate"><span class="pre">TableProvider</span></code> in DataFusion.</p>
+<section id="additional-tableprovider-methods">
+<h5>Additional <code class="docutils literal notranslate"><span class="pre">TableProvider</span></code> Methods<a class="headerlink" href="#additional-tableprovider-methods" title="Link to this heading">¶</a></h5>
+<p><code class="docutils literal notranslate"><span class="pre">scan</span></code> has no default implementation, so it needed to be written. There are other methods on the <code class="docutils literal notranslate"><span class="pre">TableProvider</span></code> that have default implementations, but can be overridden if needed to provide additional functionality.</p>
+<section id="supports-filters-pushdown">
+<h6><code class="docutils literal notranslate"><span class="pre">supports_filters_pushdown</span></code><a class="headerlink" href="#supports-filters-pushdown" title="Link to this heading">¶</a></h6>
+<p>The <code class="docutils literal notranslate"><span class="pre">supports_filters_pushdown</span></code> method can be overridden to indicate which filter expressions support being pushed down to the data source and within that the specificity of the pushdown.</p>
+<p>This returns a <code class="docutils literal notranslate"><span class="pre">Vec</span></code> of <code class="docutils literal notranslate"><span class="pre">TableProviderFilterPushDown</span></code> enums where each enum represents a filter that can be pushed down. The <code class="docutils literal notranslate"><span class="pre">TableProviderFilterPushDown</span></code> enum has three variants:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">TableProviderFilterPushDown::Unsupported</span></code> - the filter cannot be pushed down</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">TableProviderFilterPushDown::Exact</span></code> - the filter can be pushed down and the data source can guarantee that the filter will be applied completely to all rows. This is the highest performance option.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">TableProviderFilterPushDown::Inexact</span></code> - the filter can be pushed down, but the data source cannot guarantee that the filter will be applied to all rows. DataFusion will apply <code class="docutils literal notranslate"><span class="pre">Inexact</span></code> filters again after the scan to ensure correctness.</p></li>
+</ul>
+<p>For filters that can be pushed down, they’ll be passed to the <code class="docutils literal notranslate"><span class="pre">scan</span></code> method as the <code class="docutils literal notranslate"><span class="pre">filters</span></code> parameter and they can be made use of there.</p>
+</section>
+</section>
</section>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index 07df441b04..16edd8a80f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["contributor-guide/architecture", "contributor-guide/communication", "contributor-guide/index", "contributor-guide/quarterly_roadmap", "contributor-guide/roadmap", "contributor-guide/specification/index", "contributor-guide/specification/invariants", "contributor-guide/specification/output-field-name-semantic", "index", "library-user-guide/adding-udfs", "library-user-guide/building-logical-plans", "library-user-guide/catalogs", "library-user-guide/custom-tab [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["contributor-guide/architecture", "contributor-guide/communication", "contributor-guide/index", "contributor-guide/quarterly_roadmap", "contributor-guide/roadmap", "contributor-guide/specification/index", "contributor-guide/specification/invariants", "contributor-guide/specification/output-field-name-semantic", "index", "library-user-guide/adding-udfs", "library-user-guide/building-logical-plans", "library-user-guide/catalogs", "library-user-guide/custom-tab [...]
\ No newline at end of file