[jira] [Resolved] (DRILL-8169) Add UDFs to HTTP Plugin to Facilitate Joins

["Vitalii Diravka (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-8169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vitalii Diravka resolved DRILL-8169.
------------------------------------
    Resolution: Fixed

> Add UDFs to HTTP Plugin to Facilitate Joins
> -------------------------------------------
>
>                 Key: DRILL-8169
>                 URL: https://issues.apache.org/jira/browse/DRILL-8169
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Other
>    Affects Versions: 1.20.0
>            Reporter: Charles Givre
>            Assignee: Charles Givre
>            Priority: Major
>             Fix For: 2.0.0
>
>
> There are some situations where a user might want to join data with an API result and the pushdowns prevent that from happening. The main situation where this happens is when 
> an API has parameters which are part of the URL AND these parameters are dynamically populated via a join. 
> In this case, there are two functions `http_get_url` and `http_get` which you can use to faciliate these joins. 
> * `http_get('<storage_plugin_name>', <params>)`: This function accepts a storage plugin as input and an optional list of parameters to include in a URL.
> * `http_get_url(<url>, <params>)`: This function works in the same way except that it does not pull any configuration information from existing storage plugins.
> ### Example Queries
> Let's say that you have a storage plugin called `github` with an endpoint called `repos` which points to the url: https://github.com/orgs/\{org}/repos. It is easy enough to 
> write a query like this:
> ```sql
> SELECT * 
> FROM github.repos
> WHERE org='apache'
> ```
> However, if you had a file with organizations and wanted to join this with the API, the query would fail. Using the functions listed above you could get this data as follows:
> ```sql
> SELECT http_get('github.repos', `org`)
> FROM dfs.`some_data.csvh`
> ```
> or
> ```sql
> SELECT http_get('https://github.com/orgs/\{org}/repos', `org`)
> FROM dfs.`some_data.csvh`
> ```
> ** WARNING: This functionality will execute an HTTP Request FOR EVERY ROW IN YOUR DATA. Use with caution. **
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)