You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/06/15 18:46:09 UTC
[jira] [Commented] (HAWQ-823) Amazon S3 External Table Support
[ https://issues.apache.org/jira/browse/HAWQ-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332301#comment-15332301 ]
ASF GitHub Bot commented on HAWQ-823:
-------------------------------------
GitHub user kdunn926 opened a pull request:
https://github.com/apache/incubator-hawq/pull/720
HAWQ-823 - Initial Amazon S3 External Table Support
Seeing some success using GPDB S3 external protocol support:
```
# On master instance
$ PGOPTIONS='-c gp_session_role=utility' psql postgres
postgres=# CREATE OR REPLACE FUNCTION read_from_s3() RETURNS integer AS '$libdir/gps3ext.so', 's3_import' LANGUAGE C STABLE;
postgres=# CREATE PROTOCOL s3 (readfunc = read_from_s3);
---
# On segment instance
$ PGOPTIONS='-c gp_session_role=utility' psql template1 -p 40000
template1=# CREATE OR REPLACE FUNCTION read_from_s3() RETURNS integer AS '$libdir/gps3ext.so', 's3_import' LANGUAGE C STABLE;
template1=# CREATE PROTOCOL s3 (readfunc = read_from_s3);
---
# On master instance
$ PGOPTIONS='-c gp_session_role=utility' psql postgres
postgres=# select * from pg_extprotocol ;
ptcname | ptcreadfn | ptcwritefn | ptcvalidatorfn | ptcowner | ptctrusted | ptcacl
---------+-----------+------------+----------------+----------+------------+--------
pxf | 10888 | 10887 | 10889 | 10 | t |
s3 | 18881 | 0 | 0 | 10 | f |
(2 rows)
postgres=# CREATE READABLE EXTERNAL TABLE S3TBL (row text)
LOCATION('s3://s3.amazonaws.com/goddard.datasets/osm_exerpt_ config=/home/gpadmin/s3.conf') FORMAT 'csv' (delimiter '>');
CREATE EXTERNAL TABLE
postgres=# select count(*) from S3TBL ;
count
---------
3995735
(1 row)
```
In order to `CREATE PROTOCOL` - I had to access both the master and segment instance (only 1 in sandbox) using utility mode to circumvent the syntax check. Not able to run the test suite as it seems to be GPDB-specific.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kdunn926/incubator-hawq s3extpr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/720.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #720
----
commit 43fcb40b2255aa790c8c02f73e9a4f4408056d0c
Author: Kyle Dunn <kd...@gmail.com>
Date: 2016-06-15T17:56:47Z
Initial HAWQ S3 External Table Support
----
> Amazon S3 External Table Support
> --------------------------------
>
> Key: HAWQ-823
> URL: https://issues.apache.org/jira/browse/HAWQ-823
> Project: Apache HAWQ
> Issue Type: Improvement
> Components: External Tables
> Reporter: Kyle R Dunn
> Assignee: Lei Chang
>
> As a cloud user, I'd like to be able to create readable external tables with data in Amazon S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)