You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/11/04 15:19:27 UTC
[GitHub] [superset] dpgaspar opened a new pull request, #22038: fix: datasource save, improve data validation
dpgaspar opened a new pull request, #22038:
URL: https://github.com/apache/superset/pull/22038
### SUMMARY
Improves data validation on datasource save, sanitise default endpoint on datasets
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on code in PR #22038:
URL: https://github.com/apache/superset/pull/22038#discussion_r1015158813
##########
tests/integration_tests/datasource_tests.py:
##########
@@ -297,6 +297,44 @@ def test_save(self):
print(k)
self.assertEqual(resp[k], datasource_post[k])
+ def test_save_default_endpoint_validation_fail(self):
+ self.login(username="admin")
+ tbl_id = self.get_table(name="birth_names").id
+
+ datasource_post = get_datasource_post()
+ datasource_post["id"] = tbl_id
+ datasource_post["owners"] = [1]
+ datasource_post["default_endpoint"] = "http://www.google.com"
+ data = dict(data=json.dumps(datasource_post))
+ resp = self.client.post("/datasource/save/", data=data)
+ assert resp.status_code == 500
Review Comment:
The frontend side is set to show a error message for 500 only. Will investigate if it's not too hardwired
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on code in PR #22038:
URL: https://github.com/apache/superset/pull/22038#discussion_r1015158813
##########
tests/integration_tests/datasource_tests.py:
##########
@@ -297,6 +297,44 @@ def test_save(self):
print(k)
self.assertEqual(resp[k], datasource_post[k])
+ def test_save_default_endpoint_validation_fail(self):
+ self.login(username="admin")
+ tbl_id = self.get_table(name="birth_names").id
+
+ datasource_post = get_datasource_post()
+ datasource_post["id"] = tbl_id
+ datasource_post["owners"] = [1]
+ datasource_post["default_endpoint"] = "http://www.google.com"
+ data = dict(data=json.dumps(datasource_post))
+ resp = self.client.post("/datasource/save/", data=data)
+ assert resp.status_code == 500
Review Comment:
The frontend side is set to show a error message for 500. Will investigate if it's not too hardwired
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar merged pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
dpgaspar merged PR #22038:
URL: https://github.com/apache/superset/pull/22038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22038:
URL: https://github.com/apache/superset/pull/22038#issuecomment-1303826220
# [Codecov](https://codecov.io/gh/apache/superset/pull/22038?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22038](https://codecov.io/gh/apache/superset/pull/22038?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (942e55d) into [master](https://codecov.io/gh/apache/superset/commit/5e1eecb3e74c5a9017b31374cf0b7d9d746e40f0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e1eecb) will **decrease** coverage by `1.42%`.
> The diff coverage is `80.95%`.
```diff
@@ Coverage Diff @@
## master #22038 +/- ##
==========================================
- Coverage 67.00% 65.57% -1.43%
==========================================
Files 1815 1815
Lines 69523 69546 +23
Branches 7479 7479
==========================================
- Hits 46585 45606 -979
- Misses 21008 22010 +1002
Partials 1930 1930
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `78.40% <80.95%> (+<0.01%)` | :arrow_up: |
| postgres | `78.45% <80.95%> (-0.01%)` | :arrow_down: |
| presto | `?` | |
| python | `78.58% <80.95%> (-2.98%)` | :arrow_down: |
| sqlite | `76.93% <80.95%> (+<0.01%)` | :arrow_up: |
| unit | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/22038?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/config.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29uZmlnLnB5) | `91.84% <ø> (+0.07%)` | :arrow_up: |
| [superset/utils/urls.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvdXJscy5weQ==) | `89.18% <75.00%> (-10.82%)` | :arrow_down: |
| [superset/views/datasource/views.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdmlld3MvZGF0YXNvdXJjZS92aWV3cy5weQ==) | `91.26% <100.00%> (+0.28%)` | :arrow_up: |
| [superset/tables/schemas.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdGFibGVzL3NjaGVtYXMucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/columns/schemas.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY29sdW1ucy9zY2hlbWFzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [...set/advanced\_data\_type/plugins/internet\_address.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvYWR2YW5jZWRfZGF0YV90eXBlL3BsdWdpbnMvaW50ZXJuZXRfYWRkcmVzcy5weQ==) | `16.32% <0.00%> (-79.60%)` | :arrow_down: |
| [superset/utils/pandas\_postprocessing/boxplot.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvcGFuZGFzX3Bvc3Rwcm9jZXNzaW5nL2JveHBsb3QucHk=) | `20.51% <0.00%> (-79.49%)` | :arrow_down: |
| [superset/charts/post\_processing.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY2hhcnRzL3Bvc3RfcHJvY2Vzc2luZy5weQ==) | `11.76% <0.00%> (-77.95%)` | :arrow_down: |
| [...perset/advanced\_data\_type/plugins/internet\_port.py](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvYWR2YW5jZWRfZGF0YV90eXBlL3BsdWdpbnMvaW50ZXJuZXRfcG9ydC5weQ==) | `18.75% <0.00%> (-77.09%)` | :arrow_down: |
| ... and [63 more](https://codecov.io/gh/apache/superset/pull/22038/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on code in PR #22038:
URL: https://github.com/apache/superset/pull/22038#discussion_r1015158813
##########
tests/integration_tests/datasource_tests.py:
##########
@@ -297,6 +297,44 @@ def test_save(self):
print(k)
self.assertEqual(resp[k], datasource_post[k])
+ def test_save_default_endpoint_validation_fail(self):
+ self.login(username="admin")
+ tbl_id = self.get_table(name="birth_names").id
+
+ datasource_post = get_datasource_post()
+ datasource_post["id"] = tbl_id
+ datasource_post["owners"] = [1]
+ datasource_post["default_endpoint"] = "http://www.google.com"
+ data = dict(data=json.dumps(datasource_post))
+ resp = self.client.post("/datasource/save/", data=data)
+ assert resp.status_code == 500
Review Comment:
The frontend side is set to show a error message for 500 only. Will investigate if it's not too hardwired
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on a diff in pull request #22038: fix: datasource save, improve data validation
Posted by GitBox <gi...@apache.org>.
villebro commented on code in PR #22038:
URL: https://github.com/apache/superset/pull/22038#discussion_r1014603466
##########
tests/unit_tests/utils/urls_tests.py:
##########
@@ -33,3 +35,24 @@ def test_convert_chart_link() -> None:
def test_convert_dashboard_link() -> None:
test_url = modify_url_query(EXPLORE_DASHBOARD_LINK, standalone="0")
assert test_url == "http://localhost:9000/superset/dashboard/3/?standalone=0"
+
+
+@pytest.mark.parametrize(
+ "url,is_safe",
+ [
+ ("http://localhost/", True),
+ ("http://localhost/superset/1", True),
+ ("localhost/superset/1", False),
+ ("ftp://localhost/superset/1", False),
+ ("http://external.com", False),
+ ("external.com", False),
+ ("///localhost", False),
+ ("xpto://localhost:[3/1/", False),
+ ],
Review Comment:
Could we also add an internal and external `https://` here?
##########
tests/integration_tests/datasource_tests.py:
##########
@@ -297,6 +297,44 @@ def test_save(self):
print(k)
self.assertEqual(resp[k], datasource_post[k])
+ def test_save_default_endpoint_validation_fail(self):
+ self.login(username="admin")
+ tbl_id = self.get_table(name="birth_names").id
+
+ datasource_post = get_datasource_post()
+ datasource_post["id"] = tbl_id
+ datasource_post["owners"] = [1]
+ datasource_post["default_endpoint"] = "http://www.google.com"
+ data = dict(data=json.dumps(datasource_post))
+ resp = self.client.post("/datasource/save/", data=data)
+ assert resp.status_code == 500
Review Comment:
Should we really be expecting 500? This may just be me, but when I see 500, I immediately think "the system borked". Could 422 work better here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org