You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "yutannihilation (via GitHub)" <gi...@apache.org> on 2025/12/08 13:51:20 UTC

[PR] chore(docs): Add a validation script for SQL reference [sedona-db]

yutannihilation opened a new pull request, #428:
URL: https://github.com/apache/sedona-db/pull/428

   Related to #180.
   
   It seems it's a bit hard to tackle the issue directly, so this pull request adds a simple validation script to check if the document reflects the implementation correctly. This checks, (1) if the functions listed in the reference matches with the actual implementation, and (2) if the functions are sorted in alphabetical order.
   
   I need to check if each one is actual problem or just a false-positive, but this is the current result:
   
   ```console
   ❯ uv run ./docs/scripts/validate_sql.py
   
   Functions only in implementation:
     - st_aswkb
     - st_aswkt
     - st_distance
     - st_distancesphere
     - st_distancespheroid
     - st_frechetdistance
     - st_geogfromtext
     - st_geometryfromtext
     - st_geomfromtext
     - st_hausdorffdistance
     - st_line_interpolate_point
     - st_line_locate_point
     - st_numinteriorrings
   
   Functions only in document:
     - st_closestpoint
     - st_linelocatepoint
   
   
   Traceback (most recent call last):
     File "/path/to/sedona-db/./docs/scripts/validate_sql.py", line 51, in <module>
       raise RuntimeError(
   RuntimeError: There are some mismatch between the SQL reference and the actual implementation!
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #428:
URL: https://github.com/apache/sedona-db/pull/428#issuecomment-3629553144

   Thanks! Yeah, it's always good to have a validation. What I'm still wondering is when we should enforce it... Before making this check mandatory, I hope this script can have some functionality to insert the new section to `sql.md` from the description and examples if there's any missing function name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #428:
URL: https://github.com/apache/sedona-db/pull/428#issuecomment-3784872959

   Thank you for the attempt! I am hoping to get the docs migrated over to Quarto soon so we can at least have that to build on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #428:
URL: https://github.com/apache/sedona-db/pull/428#issuecomment-3784219910

   Closing this for now as I get lost (especially about handling aliases). I hope I can retry this someday...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation closed pull request #428: chore(docs): Add a validation script for SQL reference
URL: https://github.com/apache/sedona-db/pull/428


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on code in PR #428:
URL: https://github.com/apache/sedona-db/pull/428#discussion_r2616088597


##########
docs/scripts/validate_sql.py:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sedonadb
+import difflib
+from pathlib import Path
+
+HERE = Path(__file__).parent
+
+with open(HERE.parent / "reference" / "sql.md", "r") as f:
+    lines = f.readlines()
+    # Headers with `##` are the function names.
+    funs_in_doc = [line[3:-1] for line in lines if line.startswith("## ")]
+
+
+### Check if all the functions are documented
+
+sd = sedonadb.connect()
+df = sd.sql(r"""
+SELECT DISTINCT
+  routine_name,
+  function_type,
+  CASE substr(routine_name, 1, 2)
+    WHEN 'st' THEN 'vector'
+    WHEN 'rs' THEN 'raster'
+    ELSE 'unknown'
+  END AS data_type,
+  count(*) OVER (PARTITION BY description) > 1 as has_alias

Review Comment:
   It seems the information about aliases cannot be retrieved by SQL. I added a tweak to guess it from `description`, but I'm not sure if this is reliable enough.
   
   I might rewrite the validation script by using Rust just like DataFusion does to generate the documentation.
   
   https://github.com/apache/datafusion/blob/5a01e68643a198a1aaa7124524d7be5be7df24ec/datafusion/core/src/bin/print_functions_docs.rs#L174-L179



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] chore(docs): Add a validation script for SQL reference [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #428:
URL: https://github.com/apache/sedona-db/pull/428#discussion_r2599224255


##########
docs/scripts/validate_sql.py:
##########
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sedonadb
+import difflib
+
+
+with open("docs/reference/sql.md", "r") as f:

Review Comment:
   ```suggestion
   import sedonadb
   import difflib
   from pathlib import Path
   
   HERE = Path(__file__).parent
   
   with open(HERE.parent / "reference" / "sql.md", "r") as f:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org