You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by orhankislal <gi...@git.apache.org> on 2018/10/26 14:25:58 UTC
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
GitHub user orhankislal opened a pull request:
https://github.com/apache/madlib/pull/337
Madpack: Add UDO and UDOC automation
JIRA: MADLIB-1281
- Add scripts for detecting changed/dropped UDOs and UDOCs.
- Expand the create_changelist.py file to consume these scripts and
create changelists with these fields filled if necessary.
- Fix the update_util.py to use the correct dictionary key.
- Add drop operator class command to the svac.sql_in to make sure the
old class is removed before creating the updated one.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/madlib/madlib madpack/complete-changelist
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/madlib/pull/337.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #337
----
commit 09c3bb2e55417309a45f0729f370920273be40b4
Author: Orhan Kislal <ok...@...>
Date: 2018-10-24T12:55:34Z
Madpack: Add UDO and UDOC automation
JIRA: MADLIB-1281
- Add scripts for detecting changed/dropped UDOs and UDOCs.
- Expand the create_changelist.py file to consume these scripts and
create changelists with these fields filled if necessary.
- Fix the update_util.py to use the correct dictionary key.
- Add drop operator class command to the svac.sql_in to make sure the
old class is removed before creating the updated one.
----
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by jingyimei <gi...@git.apache.org>.
Github user jingyimei commented on a diff in the pull request:
https://github.com/apache/madlib/pull/337#discussion_r231972206
--- Diff: src/madpack/diff_udo.sql ---
@@ -0,0 +1,81 @@
+------------------------------------------------------------------------------
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+
+-- http://www.apache.org/licenses/LICENSE-2.0
+
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+------------------------------------------------------------------------------
+
+SET client_min_messages to ERROR;
+\x on
+
+CREATE OR REPLACE FUNCTION filter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", '')
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION alter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", 'schema_madlib.')
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION get_udos(table_name text, schema_name text,
+ type_filter text)
+RETURNS VOID AS
+$$
+ import plpy
+
+ plpy.execute("""
+ create table {table_name} AS
+ SELECT *
+ FROM (
+ SELECT n.nspname AS "Schema",
+ o.oprname AS name,
+ filter_schema(o.oprcode::text, '{schema_name}') AS oprcode,
+ alter_schema(pg_catalog.format_type(o.oprleft, NULL), '{schema_name}') AS oprleft,
+ alter_schema(pg_catalog.format_type(o.oprright, NULL), '{schema_name}') AS oprright,
+ alter_schema(pg_catalog.format_type(o.oprresult, NULL), '{schema_name}') AS rettype
+ FROM pg_catalog.pg_operator o
+ LEFT JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace
+ WHERE n.nspname OPERATOR(pg_catalog.~) '^({schema_name})$'
--- End diff --
can you please explain this where clause a bit? I am a bit confused about the 'OPERATOR' part
---
[GitHub] madlib issue #337: Madpack: Add UDO and UDOC automation
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/337
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/708/
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/madlib/pull/337#discussion_r232279216
--- Diff: src/madpack/create_changelist.py ---
@@ -237,6 +325,13 @@
print "Something went wrong! The changelist might be wrong/corrupted."
raise
finally:
- os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt "
- "/tmp/madlib_tmp_udt.txt /tmp/madlib_tmp_cl.yaml "
--- End diff --
Nice catch, it should still be removed.
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/madlib/pull/337#discussion_r232282360
--- Diff: src/madpack/diff_udo.sql ---
@@ -0,0 +1,81 @@
+------------------------------------------------------------------------------
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+
+-- http://www.apache.org/licenses/LICENSE-2.0
+
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+------------------------------------------------------------------------------
+
+SET client_min_messages to ERROR;
+\x on
+
+CREATE OR REPLACE FUNCTION filter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", '')
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION alter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", 'schema_madlib.')
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION get_udos(table_name text, schema_name text,
+ type_filter text)
+RETURNS VOID AS
+$$
+ import plpy
+
+ plpy.execute("""
+ create table {table_name} AS
+ SELECT *
+ FROM (
+ SELECT n.nspname AS "Schema",
+ o.oprname AS name,
+ filter_schema(o.oprcode::text, '{schema_name}') AS oprcode,
+ alter_schema(pg_catalog.format_type(o.oprleft, NULL), '{schema_name}') AS oprleft,
+ alter_schema(pg_catalog.format_type(o.oprright, NULL), '{schema_name}') AS oprright,
+ alter_schema(pg_catalog.format_type(o.oprresult, NULL), '{schema_name}') AS rettype
+ FROM pg_catalog.pg_operator o
+ LEFT JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace
+ WHERE n.nspname OPERATOR(pg_catalog.~) '^({schema_name})$'
--- End diff --
I use the `\do madlib.*` command of `psql` as a basis. The corresponding query (you can get this if you start with `psql -E`) uses this particular phrase to get all of the operators of a particular schema.
Basically, this regex looks at the schema name(n.nspname) and filters that don't start (^) and end ($) with madlib schema name.
---
[GitHub] madlib issue #337: Madpack: Add UDO and UDOC automation
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/madlib/pull/337
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/707/
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by jingyimei <gi...@git.apache.org>.
Github user jingyimei commented on a diff in the pull request:
https://github.com/apache/madlib/pull/337#discussion_r231953942
--- Diff: src/madpack/create_changelist.py ---
@@ -237,6 +325,13 @@
print "Something went wrong! The changelist might be wrong/corrupted."
raise
finally:
- os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt "
- "/tmp/madlib_tmp_udt.txt /tmp/madlib_tmp_cl.yaml "
--- End diff --
what is this madlib_tmp_cl.yaml file here? why we don't remove it now
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by jingyimei <gi...@git.apache.org>.
Github user jingyimei commented on a diff in the pull request:
https://github.com/apache/madlib/pull/337#discussion_r232721653
--- Diff: src/madpack/diff_udo.sql ---
@@ -0,0 +1,81 @@
+------------------------------------------------------------------------------
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+
+-- http://www.apache.org/licenses/LICENSE-2.0
+
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+------------------------------------------------------------------------------
+
+SET client_min_messages to ERROR;
+\x on
+
+CREATE OR REPLACE FUNCTION filter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", '')
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION alter_schema(argstr text, schema_name text)
+RETURNS text AS $$
+ if argstr is None:
+ return "NULL"
+ return argstr.replace(schema_name + ".", 'schema_madlib.')
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION get_udos(table_name text, schema_name text,
+ type_filter text)
+RETURNS VOID AS
+$$
+ import plpy
+
+ plpy.execute("""
+ create table {table_name} AS
+ SELECT *
+ FROM (
+ SELECT n.nspname AS "Schema",
+ o.oprname AS name,
+ filter_schema(o.oprcode::text, '{schema_name}') AS oprcode,
+ alter_schema(pg_catalog.format_type(o.oprleft, NULL), '{schema_name}') AS oprleft,
+ alter_schema(pg_catalog.format_type(o.oprright, NULL), '{schema_name}') AS oprright,
+ alter_schema(pg_catalog.format_type(o.oprresult, NULL), '{schema_name}') AS rettype
+ FROM pg_catalog.pg_operator o
+ LEFT JOIN pg_catalog.pg_namespace n ON n.oid = o.oprnamespace
+ WHERE n.nspname OPERATOR(pg_catalog.~) '^({schema_name})$'
--- End diff --
Got it. Thanks for the explanation!
---
[GitHub] madlib pull request #337: Madpack: Add UDO and UDOC automation
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/madlib/pull/337
---