You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by njayaram2 <gi...@git.apache.org> on 2017/06/22 22:18:29 UTC
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
GitHub user njayaram2 opened a pull request:
https://github.com/apache/incubator-madlib/pull/144
Feautre: Weakly Connected Components
JIRA: MADLIB-1071
Implement a new module in graph, that finds all weakly connected
components of a directed graph. A weakly connected component is a
subgraph where every node has a path to every other node, ignoring
edge directions.
This does not have grouping support yet, although the interface
has it defined already.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/njayaram2/incubator-madlib feature/graph_wcc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-madlib/pull/144.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #144
----
commit a737d8f006f8c394eccf02274ae837fa72140c42
Author: Nandish Jayaram <nj...@apache.org>
Date: 2017-06-22T21:49:51Z
Feautre: Weakly Connected Components
JIRA: MADLIB-1071
Implement a new module in graph, that finds all weakly connected
components of a directed graph. A weakly connected component is a
subgraph where every node has a path to every other node, ignoring
edge directions.
This does not have grouping support yet, although the interface
has it defined already.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #144: Feautre: Weakly Connected Components
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/144
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/90/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #144: Feautre: Weakly Connected Components
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/144
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/95/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on the issue:
https://github.com/apache/incubator-madlib/pull/144
01788982 closes the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124682273
--- Diff: src/ports/postgres/modules/graph/wcc.py_in ---
@@ -0,0 +1,329 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Weakly Connected Components
+
+# Please refer to the wcc.sql_in file for the documentation
+
+"""
+@file wcc.py_in
+
+@namespace graph
+"""
+
+import plpy
+from utilities.control import MinWarning
+from utilities.utilities import _assert
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import unique_string, split_quoted_delimited_str
+from utilities.validate_args import columns_exist_in_table, get_cols_and_types
+from graph_utils import *
+
+m4_changequote(`<!', `!>')
+
+
+def validate_wcc_args(schema_madlib, vertex_table, vertex_id, edge_table,
+ edge_params, out_table, grouping_cols_list, module_name):
+ """
+ Function to validate input parameters for wcc
+ """
+ validate_graph_coding(vertex_table, vertex_id, edge_table, edge_params,
+ out_table, module_name)
+ if grouping_cols_list:
+ # validate the grouping columns. We currently only support grouping_cols
+ # to be column names in the edge_table, and not expressions!
+ _assert(columns_exist_in_table(edge_table, grouping_cols_list, schema_madlib),
+ "Weakly Connected Components error: One or more grouping columns specified do not exist!")
+ with MinWarning("warning"):
+ plpy.warning("Grouping is not currently supported at the moment.")
+
+def wcc(schema_madlib, vertex_table, vertex_id, edge_table, edge_args,
+ out_table, grouping_cols, **kwargs):
+ """
+ Function that computes the wcc
+
+ Args:
+ @param vertex_table
+ @param vertex_id
+ @param edge_table
+ @param dest_vertex
+ @param out_table
+ @param grouping_cols
+ """
+ old_msg_level = plpy.execute("""
+ SELECT setting
+ FROM pg_settings
+ WHERE name='client_min_messages'
+ """)[0]['setting']
+ plpy.execute('SET client_min_messages TO warning')
+ params_types = {'src': str, 'dest': str}
+ default_args = {'src': 'src', 'dest': 'dest'}
+ edge_params = extract_keyvalue_params(edge_args,
+ params_types, default_args)
+
+ # populate default values for optional params if null
+ if vertex_id is None:
+ vertex_id = "id"
+ if not grouping_cols:
+ grouping_cols = ''
+
+ grouping_cols_list = split_quoted_delimited_str(grouping_cols)
+ validate_wcc_args(schema_madlib, vertex_table, vertex_id, edge_table,
+ edge_params, out_table, grouping_cols_list, 'Weakly Connected Components')
+ src = edge_params["src"]
+ dest = edge_params["dest"]
+
+ message = unique_string(desp='message')
+ oldupdate = unique_string(desp='oldupdate')
+ newupdate = unique_string(desp='newupdate')
+ toupdate = unique_string(desp='toupdate')
+ temp_out_table = unique_string(desp='tempout')
+
+ distribution = m4_ifdef(<!__POSTGRESQL__!>, <!''!>,
+ <!"DISTRIBUTED BY ({0})".format(vertex_id)!>)
+
+ is_hawq = m4_ifdef(<!__HAWQ__!>, <!True!>, <!False!>)
+
+ INT_MAX = 2147483647
+ component_id = 'component_id'
+ plpy.execute("""
+ CREATE TABLE {newupdate} AS
+ SELECT {vertex_id}, CAST({INT_MAX} AS INT) AS {component_id}
+ FROM {vertex_table}
+ {distribution}
+ """.format(**locals()))
+ if is_hawq:
+ plpy.execute("""
+ CREATE TABLE {temp_out_table} AS
+ SELECT * FROM {newupdate}
+ LIMIT 0
+ {distribution}
+ """.format(**locals()))
+ plpy.execute("""
+ CREATE TEMP TABLE {message} AS
+ SELECT {vertex_id}, CAST({vertex_id} AS INT) AS {component_id}
+ FROM {vertex_table}
+ {distribution}
+ """.format(**locals()))
+ nodes_to_update = 1
+ while nodes_to_update > 0:
+ # This idea here is simple. Look at all the neighbors of a node, and
+ # assign the smallest node id among the neighbors as its component_id.
+ # The next table starts off with very high component_id (INT_MAX). The
+ # component_id of all nodes which obtain a smaller component_id after
+ # looking at its neighbors are updated in the next table. At every
+ # iteration update only those nodes whose component_id in the previous
+ # iteration are greater than what was found in the current iteration.
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(oldupdate))
+ plpy.execute("""
+ CREATE TEMP TABLE {oldupdate} AS
+ SELECT {message}.{vertex_id},
+ MIN({message}.{component_id}) AS {component_id}
+ FROM {message}
+ GROUP BY {vertex_id}
+ {distribution}
+ """.format(**locals()))
+
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(toupdate))
+ plpy.execute("""
+ CREATE TEMP TABLE {toupdate} AS
+ SELECT {oldupdate}.{vertex_id}, {oldupdate}.{component_id}
+ FROM {oldupdate}, {newupdate}
+ WHERE {oldupdate}.{vertex_id}={newupdate}.{vertex_id}
+ AND {oldupdate}.{component_id}<{newupdate}.{component_id}
+ {distribution}
+ """.format(**locals()))
+
+ if is_hawq:
+ plpy.execute("""
+ TRUNCATE TABLE {temp_out_table};
+ INSERT INTO {temp_out_table}
+ SELECT *
+ FROM {newupdate}
+ WHERE NOT EXISTS (
+ SELECT *
+ FROM {toupdate}
+ WHERE {newupdate}.{vertex_id}={toupdate}.{vertex_id}
+ )
+ UNION
+ SELECT * FROM {toupdate};
+ """.format(**locals()))
+ plpy.execute("DROP TABLE {0}".format(newupdate))
+ plpy.execute("ALTER TABLE {0} RENAME TO {1}".format(temp_out_table,
+ newupdate))
+ plpy.execute("""
+ CREATE TABLE {temp_out_table} AS
+ SELECT * FROM {newupdate}
+ LIMIT 0
+ {distribution}
+ """.format(**locals()))
+ else:
+ plpy.execute("""
+ UPDATE {newupdate} SET
+ {component_id}={toupdate}.{component_id}
+ FROM {toupdate}
+ WHERE {newupdate}.{vertex_id}={toupdate}.{vertex_id}
+ """.format(**locals()))
+
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(message))
+ plpy.execute("""
+ CREATE TEMP TABLE {message} AS
+ SELECT {vertex_id}, MIN({component_id}) AS {component_id}
+ FROM (
+ SELECT {edge_table}.{src} AS {vertex_id}, {toupdate}.{component_id}
+ FROM {toupdate}, {edge_table}
+ WHERE {edge_table}.{dest} = {toupdate}.{vertex_id}
+ UNION ALL
+ SELECT {edge_table}.{dest} AS {vertex_id}, {toupdate}.{component_id}
+ FROM {toupdate}, {edge_table}
--- End diff --
What this query is trying to do is the following:
For each node, find all its neighbors and assign this node's `component_id` as their `component_id`. The `MIN()` in the outer `SELECT` clause will then choose the minimum of all possible neighbors' component_id and assign it to a node.
I don't think the suggestion in this comment will help me do it any more efficiently. We will still have to do the joins, since the suggested query will only find all edges in the `edge_table` that have `{vertex_id}` in either src or dest. Hence assuming a no-op on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124681781
--- Diff: src/ports/postgres/modules/graph/test/wcc.sql_in ---
@@ -0,0 +1,92 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+DROP TABLE IF EXISTS vertex, edge;
+CREATE TABLE vertex(
+vertex_id INTEGER
+);
+CREATE TABLE edge(
+src_node INTEGER,
+dest_node INTEGER,
+user_id INTEGER
+);
+INSERT INTO vertex VALUES
+(0),
+(1),
+(2),
+(3),
+(4),
+(5),
+(6),
+(10),
+(11),
+(12),
+(13),
+(14),
+(15),
+(16);
+INSERT INTO edge VALUES
+(0, 1, 1),
+(0, 2, 1),
+(1, 2, 1),
+(1, 3, 1),
+(2, 3, 1),
+(2, 5, 1),
+(2, 6, 1),
+(3, 0, 1),
+(5, 6, 1),
+(6, 3, 1),
+(10, 11, 2),
+(10, 12, 2),
+(11, 12, 2),
+(11, 13, 2),
+(12, 13, 2),
+(13, 10, 2),
+(15, 16, 2),
+(15, 14, 2);
+
+DROP TABLE IF EXISTS wcc_out;
+select weakly_connected_components(
+ 'vertex',
+ 'vertex_id',
+ 'edge',
+ 'src=src_node,dest=dest_node',
+ 'wcc_out');
+
+SELECT assert(relative_error(count(distinct component_id), 4) < 0.00001,
+ 'Weakly Connected Components: Number of components found is not 4.'
+ ) FROM wcc_out;
+
+-- DROP TABLE IF EXISTS wcc_out;
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124611697
--- Diff: src/ports/postgres/modules/graph/wcc.sql_in ---
@@ -0,0 +1,261 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file graph.sql_in
+ *
+ * @brief SQL functions for graph analytics
+ * @date June 2017
+ *
+ * @sa Provides various graph algorithms.
+ *
+ *//* ----------------------------------------------------------------------- */
+m4_include(`SQLCommon.m4')
+
+
+/**
+@addtogroup grp_wcc
+
+<div class="toc"><b>Contents</b>
+<ul>
+<li><a href="#wcc">Weakly Connected Components</a></li>
+<li><a href="#notes">Notes</a></li>
+<li><a href="#examples">Examples</a></li>
+<li><a href="#literature">Literature</a></li>
+</ul>
+</div>
+
+@brief Find all weakly connected components of a graph.
+
+Given a directed graph, a weakly connected component is a subgraph of the original
+graph where all vertices are connected to each other by some path, ignoring the
+direction of edges. In case of an undirected graph, a weakly connected component is
+also a strongly connected component.
+
+@anchor wcc
+@par Weakly Connected Components
+<pre class="syntax">
+weakly_connected_components( vertex_table,
+ vertex_id,
+ edge_table,
+ edge_args,
+ out_table,
+ grouping_cols
+ )
+</pre>
+
+\b Arguments
+<dl class="arglist">
+<dt>vertex_table</dt>
+<dd>TEXT. Name of the table containing the vertex data for the graph. Must contain the
+column specified in the 'vertex_id' parameter below.</dd>
+
+<dt>vertex_id</dt>
+<dd>TEXT, default = 'id'. Name of the column in 'vertex_table' containing
+vertex ids. The vertex ids are of type INTEGER with no duplicates.
+They do not need to be contiguous.</dd>
+
+<dt>edge_table</dt>
+<dd>TEXT. Name of the table containing the edge data. The edge table must
+contain columns for source vertex and destination vertex.</dd>
+
+<dt>edge_args</dt>
+<dd>TEXT. A comma-delimited string containing multiple named arguments of
+the form "name=value". The following parameters are supported for
+this string argument:
+ - src (INTEGER): Name of the column containing the source vertex ids in the edge table.
+ Default column name is 'src'.
+ - dest (INTEGER): Name of the column containing the destination vertex ids in the edge table.
+ Default column name is 'dest'.</dd>
+
+<dt>out_table</dt>
+<dd>TEXT. Name of the table to store the component ID associated with each vertex.
+It will contain a row for every vertex from 'vertex_table' with
+the following columns:
+ - vertex_id : The id of a vertex. Will use the input parameter 'vertex_id' for column naming.
+ - component_id : The vertex's component.
+ - grouping_cols : Grouping column (if any) values associated with the vertex_id.</dd>
+
+<dt>grouping_cols (optional)</dt>
--- End diff --
Since this isn't implemented yet, we should put a note.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124604406
--- Diff: src/ports/postgres/modules/graph/test/wcc.sql_in ---
@@ -0,0 +1,92 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+DROP TABLE IF EXISTS vertex, edge;
+CREATE TABLE vertex(
+vertex_id INTEGER
+);
+CREATE TABLE edge(
+src_node INTEGER,
+dest_node INTEGER,
+user_id INTEGER
+);
+INSERT INTO vertex VALUES
+(0),
+(1),
+(2),
+(3),
+(4),
--- End diff --
This vertex doesn't have an edge (not even a <4,NULL,NULL> record on the edge table) but is still considered a connected component by itself. When we add grouping to this function, it will be hard to distinguish an unconnected vertex and a vertex that belongs to only some of the groups.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124681767
--- Diff: src/ports/postgres/modules/graph/test/wcc.sql_in ---
@@ -0,0 +1,92 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+DROP TABLE IF EXISTS vertex, edge;
+CREATE TABLE vertex(
+vertex_id INTEGER
+);
+CREATE TABLE edge(
+src_node INTEGER,
+dest_node INTEGER,
+user_id INTEGER
+);
+INSERT INTO vertex VALUES
+(0),
+(1),
+(2),
+(3),
+(4),
+(5),
+(6),
+(10),
+(11),
+(12),
+(13),
+(14),
+(15),
+(16);
+INSERT INTO edge VALUES
+(0, 1, 1),
+(0, 2, 1),
+(1, 2, 1),
+(1, 3, 1),
+(2, 3, 1),
+(2, 5, 1),
+(2, 6, 1),
+(3, 0, 1),
+(5, 6, 1),
+(6, 3, 1),
+(10, 11, 2),
+(10, 12, 2),
+(11, 12, 2),
+(11, 13, 2),
+(12, 13, 2),
+(13, 10, 2),
+(15, 16, 2),
+(15, 14, 2);
+
+DROP TABLE IF EXISTS wcc_out;
+select weakly_connected_components(
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #144: Feautre: Weakly Connected Components
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/144
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/91/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124682389
--- Diff: src/ports/postgres/modules/graph/wcc.sql_in ---
@@ -0,0 +1,261 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file graph.sql_in
+ *
+ * @brief SQL functions for graph analytics
+ * @date June 2017
+ *
+ * @sa Provides various graph algorithms.
+ *
+ *//* ----------------------------------------------------------------------- */
+m4_include(`SQLCommon.m4')
+
+
+/**
+@addtogroup grp_wcc
+
+<div class="toc"><b>Contents</b>
+<ul>
+<li><a href="#wcc">Weakly Connected Components</a></li>
+<li><a href="#notes">Notes</a></li>
+<li><a href="#examples">Examples</a></li>
+<li><a href="#literature">Literature</a></li>
+</ul>
+</div>
+
+@brief Find all weakly connected components of a graph.
+
+Given a directed graph, a weakly connected component is a subgraph of the original
+graph where all vertices are connected to each other by some path, ignoring the
+direction of edges. In case of an undirected graph, a weakly connected component is
+also a strongly connected component.
+
+@anchor wcc
+@par Weakly Connected Components
+<pre class="syntax">
+weakly_connected_components( vertex_table,
+ vertex_id,
+ edge_table,
+ edge_args,
+ out_table,
+ grouping_cols
+ )
+</pre>
+
+\b Arguments
+<dl class="arglist">
+<dt>vertex_table</dt>
+<dd>TEXT. Name of the table containing the vertex data for the graph. Must contain the
+column specified in the 'vertex_id' parameter below.</dd>
+
+<dt>vertex_id</dt>
+<dd>TEXT, default = 'id'. Name of the column in 'vertex_table' containing
+vertex ids. The vertex ids are of type INTEGER with no duplicates.
+They do not need to be contiguous.</dd>
+
+<dt>edge_table</dt>
+<dd>TEXT. Name of the table containing the edge data. The edge table must
+contain columns for source vertex and destination vertex.</dd>
+
+<dt>edge_args</dt>
+<dd>TEXT. A comma-delimited string containing multiple named arguments of
+the form "name=value". The following parameters are supported for
+this string argument:
+ - src (INTEGER): Name of the column containing the source vertex ids in the edge table.
+ Default column name is 'src'.
+ - dest (INTEGER): Name of the column containing the destination vertex ids in the edge table.
+ Default column name is 'dest'.</dd>
+
+<dt>out_table</dt>
+<dd>TEXT. Name of the table to store the component ID associated with each vertex.
+It will contain a row for every vertex from 'vertex_table' with
+the following columns:
+ - vertex_id : The id of a vertex. Will use the input parameter 'vertex_id' for column naming.
+ - component_id : The vertex's component.
+ - grouping_cols : Grouping column (if any) values associated with the vertex_id.</dd>
+
+<dt>grouping_cols (optional)</dt>
--- End diff --
Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #144: Feautre: Weakly Connected Components
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on the issue:
https://github.com/apache/incubator-madlib/pull/144
LGTM +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124653460
--- Diff: src/ports/postgres/modules/graph/wcc.py_in ---
@@ -0,0 +1,329 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Weakly Connected Components
+
+# Please refer to the wcc.sql_in file for the documentation
+
+"""
+@file wcc.py_in
+
+@namespace graph
+"""
+
+import plpy
+from utilities.control import MinWarning
+from utilities.utilities import _assert
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import unique_string, split_quoted_delimited_str
+from utilities.validate_args import columns_exist_in_table, get_cols_and_types
+from graph_utils import *
+
+m4_changequote(`<!', `!>')
+
+
+def validate_wcc_args(schema_madlib, vertex_table, vertex_id, edge_table,
+ edge_params, out_table, grouping_cols_list, module_name):
+ """
+ Function to validate input parameters for wcc
+ """
+ validate_graph_coding(vertex_table, vertex_id, edge_table, edge_params,
+ out_table, module_name)
+ if grouping_cols_list:
+ # validate the grouping columns. We currently only support grouping_cols
+ # to be column names in the edge_table, and not expressions!
+ _assert(columns_exist_in_table(edge_table, grouping_cols_list, schema_madlib),
+ "Weakly Connected Components error: One or more grouping columns specified do not exist!")
+ with MinWarning("warning"):
+ plpy.warning("Grouping is not currently supported at the moment.")
+
+def wcc(schema_madlib, vertex_table, vertex_id, edge_table, edge_args,
+ out_table, grouping_cols, **kwargs):
+ """
+ Function that computes the wcc
+
+ Args:
+ @param vertex_table
+ @param vertex_id
+ @param edge_table
+ @param dest_vertex
+ @param out_table
+ @param grouping_cols
+ """
+ old_msg_level = plpy.execute("""
+ SELECT setting
+ FROM pg_settings
+ WHERE name='client_min_messages'
+ """)[0]['setting']
+ plpy.execute('SET client_min_messages TO warning')
+ params_types = {'src': str, 'dest': str}
+ default_args = {'src': 'src', 'dest': 'dest'}
+ edge_params = extract_keyvalue_params(edge_args,
+ params_types, default_args)
+
+ # populate default values for optional params if null
+ if vertex_id is None:
+ vertex_id = "id"
+ if not grouping_cols:
+ grouping_cols = ''
+
+ grouping_cols_list = split_quoted_delimited_str(grouping_cols)
+ validate_wcc_args(schema_madlib, vertex_table, vertex_id, edge_table,
+ edge_params, out_table, grouping_cols_list, 'Weakly Connected Components')
+ src = edge_params["src"]
+ dest = edge_params["dest"]
+
+ message = unique_string(desp='message')
+ oldupdate = unique_string(desp='oldupdate')
+ newupdate = unique_string(desp='newupdate')
+ toupdate = unique_string(desp='toupdate')
+ temp_out_table = unique_string(desp='tempout')
+
+ distribution = m4_ifdef(<!__POSTGRESQL__!>, <!''!>,
+ <!"DISTRIBUTED BY ({0})".format(vertex_id)!>)
+
+ is_hawq = m4_ifdef(<!__HAWQ__!>, <!True!>, <!False!>)
+
+ INT_MAX = 2147483647
+ component_id = 'component_id'
+ plpy.execute("""
+ CREATE TABLE {newupdate} AS
+ SELECT {vertex_id}, CAST({INT_MAX} AS INT) AS {component_id}
+ FROM {vertex_table}
+ {distribution}
+ """.format(**locals()))
+ if is_hawq:
+ plpy.execute("""
+ CREATE TABLE {temp_out_table} AS
+ SELECT * FROM {newupdate}
+ LIMIT 0
+ {distribution}
+ """.format(**locals()))
+ plpy.execute("""
+ CREATE TEMP TABLE {message} AS
+ SELECT {vertex_id}, CAST({vertex_id} AS INT) AS {component_id}
+ FROM {vertex_table}
+ {distribution}
+ """.format(**locals()))
+ nodes_to_update = 1
+ while nodes_to_update > 0:
+ # This idea here is simple. Look at all the neighbors of a node, and
+ # assign the smallest node id among the neighbors as its component_id.
+ # The next table starts off with very high component_id (INT_MAX). The
+ # component_id of all nodes which obtain a smaller component_id after
+ # looking at its neighbors are updated in the next table. At every
+ # iteration update only those nodes whose component_id in the previous
+ # iteration are greater than what was found in the current iteration.
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(oldupdate))
+ plpy.execute("""
+ CREATE TEMP TABLE {oldupdate} AS
+ SELECT {message}.{vertex_id},
+ MIN({message}.{component_id}) AS {component_id}
+ FROM {message}
+ GROUP BY {vertex_id}
+ {distribution}
+ """.format(**locals()))
+
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(toupdate))
+ plpy.execute("""
+ CREATE TEMP TABLE {toupdate} AS
+ SELECT {oldupdate}.{vertex_id}, {oldupdate}.{component_id}
+ FROM {oldupdate}, {newupdate}
+ WHERE {oldupdate}.{vertex_id}={newupdate}.{vertex_id}
+ AND {oldupdate}.{component_id}<{newupdate}.{component_id}
+ {distribution}
+ """.format(**locals()))
+
+ if is_hawq:
+ plpy.execute("""
+ TRUNCATE TABLE {temp_out_table};
+ INSERT INTO {temp_out_table}
+ SELECT *
+ FROM {newupdate}
+ WHERE NOT EXISTS (
+ SELECT *
+ FROM {toupdate}
+ WHERE {newupdate}.{vertex_id}={toupdate}.{vertex_id}
+ )
+ UNION
+ SELECT * FROM {toupdate};
+ """.format(**locals()))
+ plpy.execute("DROP TABLE {0}".format(newupdate))
+ plpy.execute("ALTER TABLE {0} RENAME TO {1}".format(temp_out_table,
+ newupdate))
+ plpy.execute("""
+ CREATE TABLE {temp_out_table} AS
+ SELECT * FROM {newupdate}
+ LIMIT 0
+ {distribution}
+ """.format(**locals()))
+ else:
+ plpy.execute("""
+ UPDATE {newupdate} SET
+ {component_id}={toupdate}.{component_id}
+ FROM {toupdate}
+ WHERE {newupdate}.{vertex_id}={toupdate}.{vertex_id}
+ """.format(**locals()))
+
+ plpy.execute("DROP TABLE IF EXISTS {0}".format(message))
+ plpy.execute("""
+ CREATE TEMP TABLE {message} AS
+ SELECT {vertex_id}, MIN({component_id}) AS {component_id}
+ FROM (
+ SELECT {edge_table}.{src} AS {vertex_id}, {toupdate}.{component_id}
+ FROM {toupdate}, {edge_table}
+ WHERE {edge_table}.{dest} = {toupdate}.{vertex_id}
+ UNION ALL
+ SELECT {edge_table}.{dest} AS {vertex_id}, {toupdate}.{component_id}
+ FROM {toupdate}, {edge_table}
--- End diff --
I think it is possible to avoid joining `edge_table` and `toupdate` a second time if you create a view or CTE using:
```
SELECT {edge_table}.{src}, {edge_table}.{dest}, {toupdate}.{component_id}
FROM {toupdate}, {edge_table}
WHERE {edge_table}.{dest} = {toupdate}.{vertex_id} OR {edge_table}.{src} = {toupdate}.{vertex_id}
````
and then union it by itself for to create the message table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by orhankislal <gi...@git.apache.org>.
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124603434
--- Diff: src/ports/postgres/modules/graph/test/wcc.sql_in ---
@@ -0,0 +1,92 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+DROP TABLE IF EXISTS vertex, edge;
+CREATE TABLE vertex(
+vertex_id INTEGER
+);
+CREATE TABLE edge(
+src_node INTEGER,
+dest_node INTEGER,
+user_id INTEGER
+);
+INSERT INTO vertex VALUES
+(0),
+(1),
+(2),
+(3),
+(4),
+(5),
+(6),
+(10),
+(11),
+(12),
+(13),
+(14),
+(15),
+(16);
+INSERT INTO edge VALUES
+(0, 1, 1),
+(0, 2, 1),
+(1, 2, 1),
+(1, 3, 1),
+(2, 3, 1),
+(2, 5, 1),
+(2, 6, 1),
+(3, 0, 1),
+(5, 6, 1),
+(6, 3, 1),
+(10, 11, 2),
+(10, 12, 2),
+(11, 12, 2),
+(11, 13, 2),
+(12, 13, 2),
+(13, 10, 2),
+(15, 16, 2),
+(15, 14, 2);
+
+DROP TABLE IF EXISTS wcc_out;
+select weakly_connected_components(
--- End diff --
select -> SELECT
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 closed the pull request at:
https://github.com/apache/incubator-madlib/pull/144
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #144: Feautre: Weakly Connected Components
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/144#discussion_r124678919
--- Diff: src/ports/postgres/modules/graph/test/wcc.sql_in ---
@@ -0,0 +1,92 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+DROP TABLE IF EXISTS vertex, edge;
+CREATE TABLE vertex(
+vertex_id INTEGER
+);
+CREATE TABLE edge(
+src_node INTEGER,
+dest_node INTEGER,
+user_id INTEGER
+);
+INSERT INTO vertex VALUES
+(0),
+(1),
+(2),
+(3),
+(4),
--- End diff --
When there is no grouping, a node that has no edge must to be considered a separate component.
In the current grouping implementation of other graph modules such as PageRank and SSSP, if a node has no edge, it is not associated with any group. The current implementation in this PR (if extended to support grouping) is consistent with that assumption.
If we want to capture a disconnected node as part of a particular group, we will have to discuss and see how to represent it in the edge table. That discussion should be in a new thread as it is beyond the scope of this PR. So I am assuming a no-op on this comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---