You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2024/09/17 03:31:52 UTC
[PR] [SEDONA-655] DBSCAN [sedona]
james-willis opened a new pull request, #1589:
URL: https://github.com/apache/sedona/pull/1589
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/)
## Is this PR related to a JIRA ticket?
- Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-655. The PR name follows the format `[SEDONA-XXX] my subject`.
## What changes were proposed in this PR?
This PR add a DBSCAN function in the scala and python APIs of the spark implementation of sedona.
## How was this patch tested?
unit tests
## Did this PR include necessary documentation updates?
TODO
~- Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/99239524f17389fc4ae9548ea88756f8ea538bb9/pom.xml#L29) in `vX.Y.Z` format.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2364668566
I submitted a fix to graphframes for this. I can clear the spark catalog if the disable broadcast fix doesnt resolve
https://github.com/graphframes/graphframes/pull/459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775969735
##########
spark/common/src/main/scala/org/apache/sedona/stats/Util.scala:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.stats
+
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT
+
+private[stats] object Util {
Review Comment:
what do you think about keeping code to the left ?
```scala
def getGeometryColumnName(dataframe: DataFrame): String = {
val geomFields = dataframe.schema.fields.filter(_.dataType == GeometryUDT)
if (geomFields.length == 1) {
geomFields.head.name
}
if (geomFields.isEmpty) {
throw new IllegalArgumentException("No GeometryType column found. Provide a dataframe containing a geometry column.")
}
if (geomFields.length > 1 && !geomFields.exists(_.name == "geometry")) {
throw new IllegalArgumentException(
"Multiple GeometryType columns found. Provide the column name as an argument.")
}
"geometry"
}
```
I think that way is more readable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2360223124
@james-willis The memory consumption of new tests seem to be very high?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775977256
##########
python/tests/stats/test_dbscan.py:
##########
@@ -0,0 +1,252 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyspark.sql.functions as f
+
+from sedona.sql.st_constructors import ST_MakePoint
+from sedona.sql.st_functions import ST_Buffer
+from sklearn.cluster import DBSCAN as sklearnDBSCAN
+from sedona.stats.clustering.dbscan import dbscan
+
+from tests.test_base import TestBase
+
+
+class TestDBScan(TestBase):
+
+ def get_data(self):
Review Comment:
that might be fixture
https://docs.pytest.org/en/6.2.x/fixture.html
sth like
```python
@pytest.fixture
def sample_db_scan_data():
...
```
and then for each function
```python
def test_sth(sample_db_scan_data):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775978170
##########
python/sedona/stats/utils/__init__.py:
##########
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from pyspark.sql import DataFrame, Column, SparkSession
+from sedona.sql.types import GeometryType
+
+
+def get_geometry_column_name(df: DataFrame) -> Column:
Review Comment:
I think here you can also keep happy path to the left as in the scala :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775962773
##########
python/tests/stats/test_dbscan.py:
##########
@@ -0,0 +1,252 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyspark.sql.functions as f
+
+from sedona.sql.st_constructors import ST_MakePoint
+from sedona.sql.st_functions import ST_Buffer
+from sklearn.cluster import DBSCAN as sklearnDBSCAN
+from sedona.stats.clustering.dbscan import dbscan
+
+from tests.test_base import TestBase
+
+
+class TestDBScan(TestBase):
+
+ def get_data(self):
+ return [
+ {"id": 1, "x": 1.0, "y": 2.0},
+ {"id": 2, "x": 3.0, "y": 4.0},
+ {"id": 3, "x": 2.5, "y": 4.0},
+ {"id": 4, "x": 1.5, "y": 2.5},
+ {"id": 5, "x": 3.0, "y": 5.0},
+ {"id": 6, "x": 12.8, "y": 4.5},
+ {"id": 7, "x": 2.5, "y": 4.5},
+ {"id": 8, "x": 1.2, "y": 2.5},
+ {"id": 9, "x": 1.0, "y": 3.0},
+ {"id": 10, "x": 1.0, "y": 5.0},
+ {"id": 11, "x": 1.0, "y": 2.5},
+ {"id": 12, "x": 5.0, "y": 6.0},
+ {"id": 13, "x": 4.0, "y": 3.0},
+ ]
+
+ def create_sample_dataframe(self):
+ return self.spark.createDataFrame(self.get_data()).select(
+ ST_MakePoint("x", "y").alias("arealandmark"), "id"
+ ).repartition(9)
+
+ def get_expected_result(self, input_data, epsilon, min_pts, include_outliers=True):
+ labels = (
+ sklearnDBSCAN(eps=epsilon, min_samples=min_pts)
+ .fit([[datum["x"], datum["y"]] for datum in input_data])
+ .labels_
+ )
+ expected = [(x[0] + 1, x[1]) for x in list(enumerate(labels))]
+ clusters = [x for x in set(labels) if (x != -1 or include_outliers)]
+ cluster_members = {
+ frozenset([y[0] for y in expected if y[1] == x]) for x in clusters
+ }
+ return cluster_members
+
+ def get_actual_results(
+ self,
+ input_data,
+ epsilon,
+ min_pts,
+ geometry=None,
+ id=None,
+ include_outliers=True,
+ ):
+ result = dbscan(
+ input_data, epsilon, min_pts, geometry, include_outliers=include_outliers
+ )
+ id = id or "id"
+ clusters_members = [
+ (x[id], x.cluster)
+ for x in result.collect()
+ if x.cluster != -1 or include_outliers
+ ]
+
+ result.unpersist()
+
+ clusters = {
+ frozenset([y[0] for y in clusters_members if y[1] == x])
+ for x in set([y[1] for y in clusters_members])
+ }
+
+ return clusters
+
+ def test_dbscan_valid_parameters(self):
+ # repeated broadcast joins with this small data size use a lot of RAM on broadcast references
+ self.spark.conf.set(
+ "sedona.join.autoBroadcastJoinThreshold", -1
+ )
+ self.spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+
+ df = self.create_sample_dataframe()
+ for epsilon in [0.6, 0.7, 0.8]:
+ for min_pts in [3, 4, 5]:
+ assert self.get_expected_result(
+ self.get_data(), epsilon, min_pts
+ ) == self.get_actual_results(df, epsilon, min_pts)
+
+ def test_dbscan_valid_parameters_default_column_name(self):
+ # repeated broadcast joins with this small data size use a lot of RAM on broadcast references
+ self.spark.conf.set(
+ "sedona.join.autoBroadcastJoinThreshold", -1
+ )
+ self.spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+
+ df = self.create_sample_dataframe().select(
+ "id", f.col("arealandmark").alias("geometryFieldName")
+ )
+ epsilon = 0.6
+ min_pts = 4
+
+ assert self.get_expected_result(
+ self.get_data(), epsilon, min_pts
+ ) == self.get_actual_results(df, epsilon, min_pts)
+
+ def test_dbscan_valid_parameters_polygons(self):
+ # repeated broadcast joins with this small data size use a lot of RAM on broadcast references
+ self.spark.conf.set(
+ "sedona.join.autoBroadcastJoinThreshold", -1
+ )
+ self.spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+
+ df = self.create_sample_dataframe().select(
+ "id", ST_Buffer(f.col("arealandmark"), 0.000001).alias("geometryFieldName")
+ )
+ epsilon = 0.6
+ min_pts = 4
+
+ assert self.get_expected_result(
+ self.get_data(), epsilon, min_pts
+ ) == self.get_actual_results(df, epsilon, min_pts)
+
+ def test_dbscan_supports_other_distance_function(self):
+ # repeated broadcast joins with this small data size use a lot of RAM on broadcast references
+ self.spark.conf.set(
+ "sedona.join.autoBroadcastJoinThreshold", -1
+ )
+ self.spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
+
+ df = self.create_sample_dataframe().select(
+ "id", ST_Buffer(f.col("arealandmark"), 0.000001).alias("geometryFieldName")
+ )
+ epsilon = 0.6
+ min_pts = 4
+
+ dbscan(
+ df,
+ epsilon,
+ min_pts,
+ "geometryFieldName",
+ use_spheroid=True,
+ )
+
+ def test_dbscan_invalid_epsilon(self):
+ df = self.create_sample_dataframe()
+
+ try:
+ dbscan(df, -0.1, 5, "arealandmark")
+ assert False
+ except Exception:
+ assert True
+
+ def test_dbscan_invalid_min_pts(self):
+ df = self.create_sample_dataframe()
+
+ try:
Review Comment:
I think that's equivalent
```python
with pytest.raises(Exception):
dbscan(df, 0.1, -5, "arealandmark")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775980047
##########
spark/common/src/main/scala/org/apache/sedona/stats/clustering/DBSCAN.scala:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.stats.clustering
+
+import org.apache.sedona.stats.Util.getGeometryColumnName
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT
+import org.apache.spark.sql.sedona_sql.expressions.st_functions.{ST_Distance, ST_DistanceSpheroid}
+import org.apache.spark.sql.{Column, DataFrame, SparkSession}
+import org.graphframes.GraphFrame
+
+object DBSCAN {
+
+ private val ID_COLUMN = "__id"
+
+ /**
+ * Annotates a dataframe with a cluster label for each data record using the DBSCAN algorithm.
+ * The dataframe should contain at least one GeometryType column. Rows must be unique. If one
+ * geometry column is present it will be used automatically. If two are present, the one named
+ * 'geometry' will be used. If more than one are present and neither is named 'geometry', the
+ * column name must be provided. The new column will be named 'cluster'.
+ *
+ * @param dataframe
+ * apache sedona idDataframe containing the point geometries
+ * @param epsilon
+ * minimum distance parameter of DBSCAN algorithm
+ * @param min_pts
+ * minimum number of points parameter of DBSCAN algorithm
+ * @param geometry
+ * name of the geometry column
+ * @param includeOutliers
+ * whether to include outliers in the output. Default is false
+ * @param useSpheroid
+ * whether to use a cartesian or spheroidal distance calculation. Default is false
+ * @return
+ * The input DataFrame with the cluster label added to each row. Outlier will have a cluster
+ * value of -1 if included.
+ */
+ def dbscan(
+ dataframe: DataFrame,
+ epsilon: Double,
+ min_pts: Int,
+ geometry: String = null,
+ includeOutliers: Boolean = true,
+ useSpheroid: Boolean = false): DataFrame = {
+
+ // We want to disable broadcast joins because the broadcast reference were using too much driver memory
+ val spark = SparkSession.getActiveSession.get
Review Comment:
is here possible to have an empty session ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775974765
##########
python/tests/stats/test_dbscan.py:
##########
@@ -0,0 +1,252 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pyspark.sql.functions as f
+
+from sedona.sql.st_constructors import ST_MakePoint
+from sedona.sql.st_functions import ST_Buffer
+from sklearn.cluster import DBSCAN as sklearnDBSCAN
+from sedona.stats.clustering.dbscan import dbscan
+
+from tests.test_base import TestBase
+
+
+class TestDBScan(TestBase):
+
+ def get_data(self):
+ return [
+ {"id": 1, "x": 1.0, "y": 2.0},
+ {"id": 2, "x": 3.0, "y": 4.0},
+ {"id": 3, "x": 2.5, "y": 4.0},
+ {"id": 4, "x": 1.5, "y": 2.5},
+ {"id": 5, "x": 3.0, "y": 5.0},
+ {"id": 6, "x": 12.8, "y": 4.5},
+ {"id": 7, "x": 2.5, "y": 4.5},
+ {"id": 8, "x": 1.2, "y": 2.5},
+ {"id": 9, "x": 1.0, "y": 3.0},
+ {"id": 10, "x": 1.0, "y": 5.0},
+ {"id": 11, "x": 1.0, "y": 2.5},
+ {"id": 12, "x": 5.0, "y": 6.0},
+ {"id": 13, "x": 4.0, "y": 3.0},
+ ]
+
+ def create_sample_dataframe(self):
+ return self.spark.createDataFrame(self.get_data()).select(
+ ST_MakePoint("x", "y").alias("arealandmark"), "id"
+ ).repartition(9)
+
+ def get_expected_result(self, input_data, epsilon, min_pts, include_outliers=True):
+ labels = (
+ sklearnDBSCAN(eps=epsilon, min_samples=min_pts)
+ .fit([[datum["x"], datum["y"]] for datum in input_data])
+ .labels_
+ )
+ expected = [(x[0] + 1, x[1]) for x in list(enumerate(labels))]
+ clusters = [x for x in set(labels) if (x != -1 or include_outliers)]
+ cluster_members = {
+ frozenset([y[0] for y in expected if y[1] == x]) for x in clusters
+ }
+ return cluster_members
+
+ def get_actual_results(
+ self,
+ input_data,
+ epsilon,
+ min_pts,
+ geometry=None,
+ id=None,
+ include_outliers=True,
+ ):
+ result = dbscan(
+ input_data, epsilon, min_pts, geometry, include_outliers=include_outliers
+ )
+ id = id or "id"
+ clusters_members = [
+ (x[id], x.cluster)
+ for x in result.collect()
+ if x.cluster != -1 or include_outliers
+ ]
+
+ result.unpersist()
+
+ clusters = {
+ frozenset([y[0] for y in clusters_members if y[1] == x])
+ for x in set([y[1] for y in clusters_members])
+ }
+
+ return clusters
+
+ def test_dbscan_valid_parameters(self):
Review Comment:
I think it might be a good use case for parametrized tests
https://docs.pytest.org/en/7.1.x/how-to/parametrize.html
I mean you can try to create cases based on those loops
for epsilon in [0.6, 0.7, 0.8]:
for min_pts in [3, 4, 5]:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "zwu-net (via GitHub)" <gi...@apache.org>.
zwu-net commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2372442269
> just documentation
After you guys (@james-willis @jiayuasu) finish this, I'll write an article on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2362870444
Theres some dataframes inside of the connected components algo that dont get unpersisted. I will look into raising a PR in that repo tomorrow to fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2369915453
just documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1589:
URL: https://github.com/apache/sedona/pull/1589#discussion_r1775987299
##########
spark/common/src/main/scala/org/apache/sedona/stats/clustering/DBSCAN.scala:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.sedona.stats.clustering
+
+import org.apache.sedona.stats.Util.getGeometryColumnName
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.sedona_sql.UDT.GeometryUDT
+import org.apache.spark.sql.sedona_sql.expressions.st_functions.{ST_Distance, ST_DistanceSpheroid}
+import org.apache.spark.sql.{Column, DataFrame, SparkSession}
+import org.graphframes.GraphFrame
+
+object DBSCAN {
+
+ private val ID_COLUMN = "__id"
+
+ /**
+ * Annotates a dataframe with a cluster label for each data record using the DBSCAN algorithm.
+ * The dataframe should contain at least one GeometryType column. Rows must be unique. If one
+ * geometry column is present it will be used automatically. If two are present, the one named
+ * 'geometry' will be used. If more than one are present and neither is named 'geometry', the
+ * column name must be provided. The new column will be named 'cluster'.
+ *
+ * @param dataframe
+ * apache sedona idDataframe containing the point geometries
+ * @param epsilon
+ * minimum distance parameter of DBSCAN algorithm
+ * @param min_pts
+ * minimum number of points parameter of DBSCAN algorithm
+ * @param geometry
+ * name of the geometry column
+ * @param includeOutliers
+ * whether to include outliers in the output. Default is false
+ * @param useSpheroid
+ * whether to use a cartesian or spheroidal distance calculation. Default is false
+ * @return
+ * The input DataFrame with the cluster label added to each row. Outlier will have a cluster
+ * value of -1 if included.
+ */
+ def dbscan(
+ dataframe: DataFrame,
+ epsilon: Double,
+ min_pts: Int,
+ geometry: String = null,
+ includeOutliers: Boolean = true,
+ useSpheroid: Boolean = false): DataFrame = {
+
+ // We want to disable broadcast joins because the broadcast reference were using too much driver memory
+ val spark = SparkSession.getActiveSession.get
+
+ val geometryCol = geometry match {
+ case null => getGeometryColumnName(dataframe)
+ case _ => geometry
+ }
+ validateInputs(dataframe, epsilon, min_pts, geometryCol)
+
+ val distanceFunction: (Column, Column) => Column =
+ if (useSpheroid) ST_DistanceSpheroid else ST_Distance
+
+ val hasIdColumn = dataframe.columns.contains("id")
+ val idDataframe = if (hasIdColumn) {
+ dataframe
+ .withColumnRenamed("id", ID_COLUMN)
+ .withColumn("id", sha2(to_json(struct("*")), 256))
+ } else {
+ dataframe.withColumn("id", sha2(to_json(struct("*")), 256))
Review Comment:
duplicated records should be equal and aggregated or not ? I am wondering if here `monotonically_increasing_id ` function is enough ? I think that might by costly `sha2(to_json(struct("*"))` also as key for join might not be super efficient compared to bigints ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2369616113
what are the remaining tasks for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-655] DBSCAN [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #1589:
URL: https://github.com/apache/sedona/pull/1589#issuecomment-2362820851
> @james-willis The memory consumption of new tests seem to be very high?
This is something I had worked with Kristin on in the past. We had thought it was related to cached dataframes holding references to broadcast relations. But this committed version of the code does not contain any caching, just checkpoints.
There is some persisting inside of the connected component implementation, but those should be getting cleaned up. I'll check if something here is leaking persisted dataframes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org