You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/10/21 16:41:23 UTC
[GitHub] [superset] hughhhh opened a new pull request, #21912: feat: Create `ssh_tunnel_credentials` table
hughhhh opened a new pull request, #21912:
URL: https://github.com/apache/superset/pull/21912
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Related to #21789
Creating table to store ssh tunnel credentials. This a starter ticket for allowing using to enable ssh tunneling when trying to connect to a database.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1021793070
##########
superset/databases/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
Should this be `sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028278374
##########
setup.py:
##########
@@ -113,6 +113,7 @@ def get_git_sha() -> str:
"PyJWT>=2.4.0, <3.0",
"redis",
"selenium>=3.141.0",
+ "sshtunnel>=0.4.0",
Review Comment:
Should we put an upper limit to the version range here?
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
Review Comment:
Instead of referencing the app directly, pls use `current_app`.
```
from flask import current_app
app_config = current_app.config
```
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
Do we need to expose these fields? It was my understanding that we would be generating these dynamically. π€
##########
superset/models/core.py:
##########
@@ -368,14 +372,41 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params = {}
+ if ssh_tunnel:
+ # build with override
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
+ ssh_tunnel.bind_host = url.host
Review Comment:
Shouldn't this be dynamic? Doesn't seem to me that this would work if you entered some arbitrary value in the UI for `bind_host`, for example.
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> Dict[str, Any]:
+ ssh_tunnel = (
+ db.session.query(SSHTunnel)
+ .filter(SSHTunnel.database_id == database_id)
+ .one_or_none()
+ )
+
+ return dict(ssh_tunnel=ssh_tunnel)
Review Comment:
Agree - just return the model :)
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import EncryptedType, UUIDType
+
+from superset import app
+
+app_config = app.config
+
+
+def upgrade():
Review Comment:
IMO, that pattern is hard to work with -- it's hard to make a single PR with migrations that are later depended upon by other code as the migrations tend to evolve as you work on the feature. Also, encapsulating related changes in a single PR makes more sense as the revert would remove the DB migrations along with the code that's depending on them :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035416042
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
What was the decision here?
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
Review Comment:
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1005047319
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
+ Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin
+):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel_credentials"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel_credentials", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
Review Comment:
This is more of a question than a suggestion, but I see that we're using `encrypted_field_factory` for database passwords and encrypted_extra. Is there any benefit to using the EncryptedType here instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1289636610
# [Codecov](https://codecov.io/gh/apache/superset/pull/21912?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21912](https://codecov.io/gh/apache/superset/pull/21912?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (da27d8f) into [master](https://codecov.io/gh/apache/superset/commit/28c7636c591f39547f1201ff02da822e4a1bdf88?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (28c7636) will **decrease** coverage by `11.54%`.
> The diff coverage is `35.48%`.
```diff
@@ Coverage Diff @@
## master #21912 +/- ##
===========================================
- Coverage 66.90% 55.36% -11.55%
===========================================
Files 1806 1807 +1
Lines 69130 69161 +31
Branches 7391 7391
===========================================
- Hits 46249 38288 -7961
- Misses 20970 28962 +7992
Partials 1911 1911
```
| Flag | Coverage Ξ | |
|---|---|---|
| hive | `?` | |
| mysql | `?` | |
| postgres | `?` | |
| presto | `52.80% <35.48%> (-0.02%)` | :arrow_down: |
| python | `57.56% <35.48%> (-23.91%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `51.05% <35.48%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/21912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Ξ | |
|---|---|---|
| [superset/databases/models.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YWJhc2VzL21vZGVscy5weQ==) | `0.00% <0.00%> (ΓΈ)` | |
| [superset/databases/schemas.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YWJhc2VzL3NjaGVtYXMucHk=) | `83.67% <100.00%> (-14.56%)` | :arrow_down: |
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/tags/core.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdGFncy9jb3JlLnB5) | `4.54% <0.00%> (-95.46%)` | :arrow_down: |
| [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
| [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-87.88%)` | :arrow_down: |
| [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
| [superset/key\_value/commands/delete\_expired.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZV9leHBpcmVkLnB5) | `0.00% <0.00%> (-84.00%)` | :arrow_down: |
| [superset/dashboards/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9pbXBvcnRlcnMvdjAucHk=) | `15.62% <0.00%> (-76.25%)` | :arrow_down: |
| [superset/datasets/commands/create.py](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvY3JlYXRlLnB5) | `30.61% <0.00%> (-69.39%)` | :arrow_down: |
| ... and [286 more](https://codecov.io/gh/apache/superset/pull/21912/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: Weβre building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1049990722
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
Maybe if we rename `params` to `sshtunnel_kwargd` it would be clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044890310
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
Review Comment:
I don't think we have a written convention, but agree that we should. π
I would also vote for plural long-term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1043470782
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
Review Comment:
Hey @dpgaspar I created this PR: https://github.com/apache/superset/pull/22372 with the required tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1052544005
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4, unique=True, index=True),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id"), unique=True),
Review Comment:
```suggestion
sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id"), unique=True, index=True),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1004982096
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
+ Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin
+):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel_credentials"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel_credentials", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
Review Comment:
I don't think you need this foreign_key here since you declared it in the database_id column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035411269
##########
setup.py:
##########
@@ -113,6 +113,7 @@ def get_git_sha() -> str:
"PyJWT>=2.4.0, <3.0",
"redis",
"selenium>=3.141.0",
+ "sshtunnel>=0.4.0",
Review Comment:
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037433199
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> SSHTunnel:
Review Comment:
Fixed in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037480649
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
The API validates required fields (POST AND PUT call) and it throws errors if any of them is missing as evidenced in the API PR: https://github.com/apache/superset/pull/22199/files#diff-b94e273d2b41d3fe8a552cbf34cfa59ab9495154d16823b411470ae782cea38aR509
Other than that, the PUT call works like a PATCH behind the scene, so users would be able to send: `required field + any combination of optional fields` and the update will work.
Does this meet the validation you mention or do you want to validate things like:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1332345634
@hughhhh Ephemeral environment spinning up at http://34.220.62.117:8080. Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping and startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1049990118
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
Review Comment:
Talked offline, this is passed to `sshtunnel` as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1004975970
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
+ Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin
+):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel_credentials"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel_credentials", cascade="all, delete-orphan"),
Review Comment:
this is a one-to-one, correct? If so, I think you'll need `uselist=False` here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1291239138
This looks great @hughhhh. If you wanted to write a test, one scenario that would be useful to see would be the test connection where you would create instances of the db and ssh_tunnel without saving them and be able to pull information out of both of the models.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026944888
##########
superset/databases/ssh_tunnel/commands/exceptions.py:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask_babel import lazy_gettext as _
+
+from superset.commands.exceptions import (
+ CommandException,
+ CommandInvalidError,
+ DeleteFailedError,
+ UpdateFailedError,
+)
+
+
+class SSHTunnelDeleteFailedError(DeleteFailedError):
+ message = _("SSH Tunnel could not be deleted.")
+
+
+class SSHTunnelNotFoundError(CommandException):
+ status = 404
+ message = _("SSH Tunnel not found.")
+
+
+class SSHTunnelInvalidError(CommandInvalidError):
+ message = _("SSH Tunnel parameters are invalid.")
Review Comment:
can we make this a 422 status for params invalid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042100972
##########
superset/databases/schemas.py:
##########
@@ -365,6 +365,22 @@ class Meta: # pylint: disable=too-few-public-methods
)
+class DatabaseSSHTunnel(Schema):
+ id = fields.Integer()
+ database_id = fields.Integer()
Review Comment:
nit: on a POST request for creating a database with an SSH tunnel the user does not have a database_id yet, it can be ignored but will make the OpenAPI spec seem strange.
Also regarding OAS, can you add a description on each field?
##########
superset/databases/schemas.py:
##########
@@ -409,6 +425,7 @@ class Meta: # pylint: disable=too-few-public-methods
is_managed_externally = fields.Boolean(allow_none=True, default=False)
external_url = fields.String(allow_none=True)
uuid = fields.String(required=False)
+ ssh_tunnel = fields.Nested(DatabaseSSHTunnel, allow_none=True)
Review Comment:
do we need this on `DatabasePutSchema` also?
see if creating a `DatabaseSSHTunnelPostSchema` and `DatabaseSSHTunnelPutSchema` is needed
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
Review Comment:
Would be nice to declare a `TypedDict` here and use it has the return type, improved type validation and a more explicit contract
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
Review Comment:
can you add a test for update ssh tunnel not found, and verify `SSHTunnelNotFoundError`
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_ssh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
Review Comment:
can you add a test for delete ssh tunnel not found, and verify SSHTunnelNotFoundError
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
Review Comment:
we have a mix of plural and singular table names on Superset, do we have a written convention on this?
##########
tests/unit_tests/databases/ssh_tunnel/commands/create_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+def test_create_ssh_tunnel_command() -> None:
+ from superset.databases.ssh_tunnel.commands.create import CreateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ db = Database(id=1, database_name="my_database", sqlalchemy_uri="sqlite://")
+
+ properties = {
+ "database_id": db.id,
+ "server_address": "123.132.123.1",
+ "server_port": "3005",
+ "username": "foo",
+ "password": "bar",
+ }
Review Comment:
can you add a test for create ssh tunnel with an invalid `database_id`
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import EncryptedType, UUIDType
+
+from superset import app
+
+app_config = app.config
+
+
+def upgrade():
Review Comment:
@betodealmeida, I see your point, but still think it's easier to reason and easier to revert like @craig-rueda said. Also is happens you don't get things right the first time, then you end up with a stream of db mig PR's. I think it's easier to reason on 1 PR 1 feature (at least on the backend), follow up PR's are fixes or frontend code. Has always there can be exceptions on really big features.
Just adding to the discussion :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044893002
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
Review Comment:
I noticed that we generally have these timestamps nullable in the db. Does anyone know the reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042270927
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
Review Comment:
Hey @dpgaspar these tests are part of this PR: https://github.com/apache/superset/pull/22199
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_ssh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
Review Comment:
Hey @dpgaspar these tests are part of this PR: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044893269
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
Review Comment:
should we put a uniqueness constraint on this column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044884578
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
+ if ssh_tunnel_properties := self._properties.get("ssh_tunnel"):
+ ssh_tunnel = SSHTunnelDAO.create(
+ {
+ **ssh_tunnel_properties,
+ "database_id": database.id,
+ },
+ commit=False,
Review Comment:
what's the difference between creating an instance with the DAO but not committing vs through the model directly like `SSHTunnel(<some properties>)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026937885
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
Review Comment:
should we add a unique constraint on this and in the model?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1019225670
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnel_credentials",
Review Comment:
Does this means user will still be creating a `ssh_tunnel_credentials` table? because our tests are expecting `ssh_tunnel` so they are failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028074771
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
Review Comment:
Makes sense! Will change it :) Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028106536
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_shh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert 1 == result["ssh_tunnel"].database_id
+
+ DeleteSSHTunnelCommand(1).run()
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert result["ssh_tunnel"] is None
Review Comment:
Do you want those tests in this PR or the could we add them in the APIs one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026948992
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
+ exceptions: List[ValidationError] = []
+ # Validate/populate model exists
+ self._model = SSHTunnelDAO.find_by_id(self._model_id)
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+ if exceptions:
Review Comment:
I think the exceptions list may be copy pasta.. I don't see it being populated anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042271353
##########
tests/unit_tests/databases/ssh_tunnel/commands/create_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+def test_create_ssh_tunnel_command() -> None:
+ from superset.databases.ssh_tunnel.commands.create import CreateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ db = Database(id=1, database_name="my_database", sqlalchemy_uri="sqlite://")
+
+ properties = {
+ "database_id": db.id,
+ "server_address": "123.132.123.1",
+ "server_port": "3005",
+ "username": "foo",
+ "password": "bar",
+ }
Review Comment:
Hey @dpgaspar these tests are part of this PR: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1045114888
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
+ if ssh_tunnel_properties := self._properties.get("ssh_tunnel"):
+ ssh_tunnel = SSHTunnelDAO.create(
+ {
+ **ssh_tunnel_properties,
+ "database_id": database.id,
+ },
+ commit=False,
Review Comment:
This will be using the Commands now as shown in the API PR: https://github.com/apache/superset/pull/22199/files#diff-c3f1c5afb1b73f705f01bfa24cb4ca2ac66b84f82875abd3ddb73cb8147ea77eR84 . As for what's the different, I might be wrong but DAO's create is adding the model to the session so we can commit such session later, while just instantiating the model will create the model without it being added to any session so it never persists (unless you go and manually add it to the session I'd assume).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046042005
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
Review Comment:
Not sure following the pattern I saw for all the `AuditMixinNullable` columns in other migrations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046312573
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
Review Comment:
Agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] AAfghahi commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
AAfghahi commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1005834524
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import EncryptedType, UUIDType
+
+from superset import app
+
+app_config = app.config
+
+
+def upgrade():
Review Comment:
I know that previously we had established the pattern that migration and logic should be in separate PRs. I thought that made sense, since it helps with reversing the migration in case something happens. Thoughts on doing that for this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035416320
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ bind_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+
+ def parameters(self) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (self.bind_host, self.bind_port),
+ "local_bind_address": ("127.0.0.1",),
Review Comment:
bump.. can we make this a constant at least?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037432838
##########
superset/databases/commands/test_connection.py:
##########
@@ -90,6 +91,14 @@ def run(self) -> None: # pylint: disable=too-many-statements
database.set_sqlalchemy_uri(uri)
database.db_engine_spec.mutate_db_for_connection_test(database)
+ # Generate tunnel if present in the properties
+ ssh_tunnel = None
Review Comment:
Fixed in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> Dict[str, Any]:
Review Comment:
Fixed in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1052539301
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
Review Comment:
```suggestion
sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4, unique=True, index=True),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1332339765
/testenv up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1047431827
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
Review Comment:
no right now, but this is mostly for export/import if we wanted to have ssh tunnels exported with the db credentials
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1045116318
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
Review Comment:
we need the `ssh_tunnel` or we'll get a referenced before assigned error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026948801
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
Review Comment:
looks like this value is already checked in validate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026959506
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.update import UpdateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert 1 == result["ssh_tunnel"].database_id
+ assert "Test" == result["ssh_tunnel"].server_address
+
+ update_payload = {"server_address": "Test2"}
+ UpdateSSHTunnelCommand(1, update_payload).run()
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert "Test2" == result["ssh_tunnel"].server_address
Review Comment:
same thing here with unhappy path tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026959430
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_shh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert 1 == result["ssh_tunnel"].database_id
+
+ DeleteSSHTunnelCommand(1).run()
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert result["ssh_tunnel"] is None
Review Comment:
can we add some unhappy path tests like sshTunnel not found? And maybe some cascading deletes of databases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026956113
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
I see you have it on the migration.. that should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028078858
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
Hmm if the `update` behaves like a `PATCH` then user should be able to update any individual property of my tunnel with no need to provide anything extra, correct? Like, I don't need to send you the whole info of a tunnel to update just the password for example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026942987
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
we should probably put a length on these string/varchar types
##########
superset/databases/schemas.py:
##########
@@ -456,6 +456,26 @@ class Meta: # pylint: disable=too-few-public-methods
external_url = fields.String(allow_none=True)
+class DatabaseSSHTunnel(Schema):
+ id = fields.Integer()
+ database_id = fields.Integer()
+
+ server_address = fields.String()
+ server_port = fields.Integer()
+ username = fields.String()
+
+ # Basic Authentication
+ password = fields.String(required=False)
+
+ # password protected private key authentication
+ private_key = fields.String(required=False)
+ private_key_password = fields.String(required=False)
+
+ # remote binding port
+ bind_host = fields.String()
+ bind_port = fields.Integer()
Review Comment:
should we add some validation to some of these?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026939793
##########
superset/databases/schemas.py:
##########
@@ -482,6 +502,8 @@ class DatabaseTestConnectionSchema(Schema, DatabaseParametersSchemaMixin):
validate=[Length(1, 1024), sqlalchemy_uri_validator],
)
+ ssh_tunnel_credentials = fields.Nested(DatabaseSSHTunnel, allow_none=True)
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028071035
##########
superset/databases/ssh_tunnel/commands/exceptions.py:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask_babel import lazy_gettext as _
+
+from superset.commands.exceptions import (
+ CommandException,
+ CommandInvalidError,
+ DeleteFailedError,
+ UpdateFailedError,
+)
+
+
+class SSHTunnelDeleteFailedError(DeleteFailedError):
+ message = _("SSH Tunnel could not be deleted.")
+
+
+class SSHTunnelNotFoundError(CommandException):
+ status = 404
+ message = _("SSH Tunnel not found.")
+
+
+class SSHTunnelInvalidError(CommandInvalidError):
+ message = _("SSH Tunnel parameters are invalid.")
Review Comment:
The parent `CommandInvalidError` already has `status = 422`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037453755
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.update import UpdateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert 1 == result["ssh_tunnel"].database_id
+ assert "Test" == result["ssh_tunnel"].server_address
+
+ update_payload = {"server_address": "Test2"}
+ UpdateSSHTunnelCommand(1, update_payload).run()
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert "Test2" == result["ssh_tunnel"].server_address
Review Comment:
Tests included in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dpgaspar commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
dpgaspar commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042398475
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_ssh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
Review Comment:
that PR has the API tests, these are the command tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1043470221
##########
tests/unit_tests/databases/ssh_tunnel/commands/create_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+def test_create_ssh_tunnel_command() -> None:
+ from superset.databases.ssh_tunnel.commands.create import CreateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ db = Database(id=1, database_name="my_database", sqlalchemy_uri="sqlite://")
+
+ properties = {
+ "database_id": db.id,
+ "server_address": "123.132.123.1",
+ "server_port": "3005",
+ "username": "foo",
+ "password": "bar",
+ }
Review Comment:
Hey @dpgaspar I created this PR: https://github.com/apache/superset/pull/22372 with the required tests.
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_ssh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
Review Comment:
Hey @dpgaspar I created this PR: https://github.com/apache/superset/pull/22372 with the required tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044880665
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
Review Comment:
you can remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046312908
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
Review Comment:
Agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046323588
##########
superset/models/core.py:
##########
@@ -368,14 +372,37 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ override_ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params = {}
+ from superset.databases.dao import ( # pylint: disable=import-outside-toplevel
+ DatabaseDAO,
+ )
+
+ if ssh_tunnel := override_ssh_tunnel or DatabaseDAO.get_ssh_tunnel(
+ database_id=self.id
+ ):
+ # if ssh_tunnel is available build engine with information
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
Review Comment:
This block should be managed by a `SSHManager` that is configured via app_config. It's currently not possible for other folks to update the behavior of the tunneling mechanism
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1004977202
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
Review Comment:
nit, but naming-wise, I feel like a model _contains_ credentials, therefore the name of the model can be more generic, like just `SSHTunnel`. Open to other thoughts/ideas on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1004977202
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
Review Comment:
nit, but naming-wise, I feel like a model _contains_ credentials, therefore the name of the model can be more generic, like just `SSHTunnel`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1294008220
> I think it might be better to just add this model into the PR you're planning down the road that adds the SSH handling. It's a little hard to grok what's going on out of context. Also, you almost ALWAYS end up making model tweaks as you implement a new feature, which will require another migration π¬
IMHO the best approach is to write a single PR with logic + migration (to prevent the issues you raised), but then split it in two for review. Having self-contained migrations really helps with cherry-picking, because if you need to cherry-pick a PR that has a migration you have to also cherry-pick all previous PRs with migrations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042271353
##########
tests/unit_tests/databases/ssh_tunnel/commands/create_test.py:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+def test_create_ssh_tunnel_command() -> None:
+ from superset.databases.ssh_tunnel.commands.create import CreateSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ db = Database(id=1, database_name="my_database", sqlalchemy_uri="sqlite://")
+
+ properties = {
+ "database_id": db.id,
+ "server_address": "123.132.123.1",
+ "server_port": "3005",
+ "username": "foo",
+ "password": "bar",
+ }
Review Comment:
Hey @dpgaspar these tests are part of this PR: https://github.com/apache/superset/pull/22199
##########
tests/unit_tests/databases/ssh_tunnel/commands/update_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(database_id=db.id, database=db, server_address="Test")
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_update_shh_tunnel_command(session_with_data: Session) -> None:
Review Comment:
Hey @dpgaspar these tests are part of this PR: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1045116472
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
params is used populating the in ssh tunnel object, so `ssh_password` is the param in the `sshtunnel` class to be instantiated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] github-actions[bot] commented on pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1370285053
Ephemeral environment shutdown and build artifacts deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026939362
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> Dict[str, Any]:
+ ssh_tunnel = (
+ db.session.query(SSHTunnel)
+ .filter(SSHTunnel.database_id == database_id)
+ .one_or_none()
+ )
+
+ return dict(ssh_tunnel=ssh_tunnel)
Review Comment:
why return a dictionary instead of the original model instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026951908
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
This is prob a good place to do some validation on the args passed in, like do they have a username/pw or a private key/private key pw combination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028068834
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> Dict[str, Any]:
+ ssh_tunnel = (
+ db.session.query(SSHTunnel)
+ .filter(SSHTunnel.database_id == database_id)
+ .one_or_none()
+ )
+
+ return dict(ssh_tunnel=ssh_tunnel)
Review Comment:
Just wanted to wrap it in a dict since it's a property of the DB and not the DAO entity itself. But yeah, could just send the instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026958332
##########
superset/models/core.py:
##########
@@ -368,14 +372,41 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params = {}
+ if ssh_tunnel:
+ # build with override
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
+ ssh_tunnel.bind_host = url.host
+ ssh_tunnel.bind_port = url.port
+ ssh_params = ssh_tunnel.parameters()
+ try:
+ with sshtunnel.open_tunnel(**ssh_params) as server:
+ yield self._get_sqla_engine(
+ schema=schema,
+ nullpool=nullpool,
+ source=source,
+ ssh_tunnel_server=server,
+ )
+ except Exception as ex:
+ raise ex
+
+ else:
+ # do look up in table for using database_id
+ try:
+ yield self._get_sqla_engine(
+ schema=schema, nullpool=nullpool, source=source
+ )
+ except Exception as ex:
+ raise ex
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037480649
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
The API validates required fields (POST AND PUT call) and it throws errors if any of them is missing as evidenced in the API PR: https://github.com/apache/superset/pull/22199/files#diff-b94e273d2b41d3fe8a552cbf34cfa59ab9495154d16823b411470ae782cea38aR509
Other than that, the PUT call works like a PATCH behind the scene (based on experimentation while writing the API tests), so users would be able to send: `required field + any combination of optional fields` and the update will work.
Does this meet the validation you mention or do you want to validate also:
1. If `private_key_password` was sent, then `private_key` is required ? (the other way around I believe is not needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1050054331
##########
superset/models/core.py:
##########
@@ -66,6 +66,7 @@
from superset.utils.memoized import memoized
config = app.config
+ssh_manager = config["SSH_TUNNEL_MANAGER"]
Review Comment:
This should be exposed via `extensions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] ktmud commented on pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
ktmud commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1296547217
Not sure if this has been discussed, but why can't this be stored in the existing [encrypted_extra](https://github.com/apache/superset/blob/eac6fdcd29a8e0476b153d1452a0bd2a77b47fa4/superset/models/core.py#L150) field? It's already been used for things like [Google Sheets](https://github.com/apache/superset/blob/34a79add04d4525c164699248c30c9d6588c876d/tests/unit_tests/databases/api_test.py#L72-L91) and BigQuery connections, why would SSH tunnel credentials be different?
It seems to me you can just updated the `unmask_encrypted_extra` in `BaseEngineSpec` if you want to enable SSH tunnel for all databases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044891726
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
Review Comment:
This is a 1:1 relationship. Any reason why we're not using `uselist=False`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046277751
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
Review Comment:
do you ever look up by uuid?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046326524
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
Agreed - do we really want our DB models coupled to the `ssh` lib? In this case, I would prefer to just create a method in the "SSHManager" that will perform the mapping between whatever lives in our DB into what the underlying ssh lib expects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028081211
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
+ exceptions: List[ValidationError] = []
+ # Validate/populate model exists
+ self._model = SSHTunnelDAO.find_by_id(self._model_id)
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+ if exceptions:
Review Comment:
Good catch! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037480649
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
The API validates required fields (POST AND PUT call) and it throws errors if any of them is missing as evidenced in the API PR: https://github.com/apache/superset/pull/22199/files#diff-b94e273d2b41d3fe8a552cbf34cfa59ab9495154d16823b411470ae782cea38aR509
Other than that, the PUT call works like a PATCH behind the scene, so users would be able to send: `required field + any combination of optional fields` and the update will work.
Does this meet the validation you mention or do you want to validate also:
1. If `private_key_password` was sent, then `private_key` is required ? (the other way around I believe is not needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1038429757
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
Yeah, that's the use case. Since we don't require pkey and pkey password or password, can we validate like you said if they send private_key that we need a private_key_password, and no password, or just a password?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037608093
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
+ sa.Column("username", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column(
+ "password", encrypted_field_factory.create(sa.String(1024)), nullable=True
+ ),
+ sa.Column(
+ "private_key",
+ encrypted_field_factory.create(sa.String(1024)),
+ nullable=True,
+ ),
+ sa.Column(
+ "private_key_password",
+ encrypted_field_factory.create(sa.String(1024)),
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035267052
##########
superset/models/core.py:
##########
@@ -368,14 +372,41 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params = {}
+ if ssh_tunnel:
+ # build with override
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
+ ssh_tunnel.bind_host = url.host
Review Comment:
the bind_host and bind_port is coming from the sqla uri representing where on the VPC the database is available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035414183
##########
superset/databases/commands/test_connection.py:
##########
@@ -90,6 +91,14 @@ def run(self) -> None: # pylint: disable=too-many-statements
database.set_sqlalchemy_uri(uri)
database.db_engine_spec.mutate_db_for_connection_test(database)
+ # Generate tunnel if present in the properties
+ ssh_tunnel = None
Review Comment:
nit, but I don't think you need this assignment here. ssh_tunnel will be None even if the next line evaluates to false. It will assign the value to the local scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026958824
##########
superset/models/core.py:
##########
@@ -419,6 +450,13 @@ def _get_sqla_engine(
sqlalchemy_url, params, effective_username, security_manager, source
)
+ if ssh_tunnel_server:
+ # update sqlalchemy_url
+ url = make_url_safe(sqlalchemy_url)
+ sqlalchemy_url = url.set(
+ host="127.0.0.1", port=ssh_tunnel_server.local_bind_port
Review Comment:
can we define this host value as a constant somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1005047319
##########
superset/databases/models.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnelCredentials(
+ Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin
+):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel_credentials"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel_credentials", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
Review Comment:
This is more of a question than a suggestion, but I see that we're using `encrypted_field_factory` for database passwords. Is there any benefit to using the EncryptedType here instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1018095330
##########
requirements/development.txt:
##########
@@ -6,7 +6,7 @@
# pip-compile-multi
#
-r base.txt
--e file:.
+-e file:///Users/hugh/src/superset
Review Comment:
```suggestion
-e file:.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028072783
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
Review Comment:
Thanks! π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037558816
##########
setup.py:
##########
@@ -113,6 +113,7 @@ def get_git_sha() -> str:
"PyJWT>=2.4.0, <3.0",
"redis",
"selenium>=3.141.0",
+ "sshtunnel>=0.4.0",
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/models/core.py:
##########
@@ -419,6 +450,13 @@ def _get_sqla_engine(
sqlalchemy_url, params, effective_username, security_manager, source
)
+ if ssh_tunnel_server:
+ # update sqlalchemy_url
+ url = make_url_safe(sqlalchemy_url)
+ sqlalchemy_url = url.set(
+ host="127.0.0.1", port=ssh_tunnel_server.local_bind_port
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
tests/unit_tests/databases/ssh_tunnel/commands/__init__.py:
##########
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ bind_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+
+ def parameters(self) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (self.bind_host, self.bind_port),
+ "local_bind_address": ("127.0.0.1",),
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ bind_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+
+ def parameters(self) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (self.bind_host, self.bind_port),
+ "local_bind_address": ("127.0.0.1",),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
+ elif self.private_key:
+ params["ssh_pkey"] = self.private_key
+ params["ssh_private_key_password"] = self.private_key_password
Review Comment:
Should be fixed as part of: https://github.com/apache/superset/pull/22306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037453361
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_shh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert isinstance(result["ssh_tunnel"], SSHTunnel)
+ assert 1 == result["ssh_tunnel"].database_id
+
+ DeleteSSHTunnelCommand(1).run()
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
+
+ assert result
+ assert result["ssh_tunnel"] is None
Review Comment:
Tests included in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1044941189
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
Nit, use `password` instead of `ssh_password` for consistency.
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
Review Comment:
If this is a 1-tuple, does it need to be a tuple at all?
##########
superset/databases/schemas.py:
##########
@@ -365,6 +365,22 @@ class Meta: # pylint: disable=too-few-public-methods
)
+class DatabaseSSHTunnel(Schema):
+ id = fields.Integer()
+ database_id = fields.Integer()
Review Comment:
Yeah, if the `DatabaseSSHTunnel` is part of `DatabasePostSchema` then the database ID is redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1049990722
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
Maybe if we rename `params` to `sshtunnel_kwargs` it would be clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046041210
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: 4ce1d9b25135
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "4ce1d9b25135"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
Review Comment:
@betodealmeida do we need to put a constaint on these, in the ImportExportMixin columns i didn't a constraint being made so didn't feel like it was necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035415356
##########
tests/unit_tests/databases/ssh_tunnel/commands/__init__.py:
##########
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
Review Comment:
looks like you're missing an __init__file in the subfolder (commands) as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035410655
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
+ sa.Column("username", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column(
+ "password", encrypted_field_factory.create(sa.String(1024)), nullable=True
+ ),
+ sa.Column(
+ "private_key",
+ encrypted_field_factory.create(sa.String(1024)),
+ nullable=True,
+ ),
+ sa.Column(
+ "private_key_password",
+ encrypted_field_factory.create(sa.String(1024)),
Review Comment:
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035492098
##########
superset/databases/commands/test_connection.py:
##########
@@ -90,6 +91,14 @@ def run(self) -> None: # pylint: disable=too-many-statements
database.set_sqlalchemy_uri(uri)
database.db_engine_spec.mutate_db_for_connection_test(database)
+ # Generate tunnel if present in the properties
+ ssh_tunnel = None
Review Comment:
```suggestion
```
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> Dict[str, Any]:
Review Comment:
```suggestion
def get_ssh_tunnel(cls, database_id: int) -> Optional[SSHTunnel]:
```
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
+ sa.Column("username", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column(
+ "password", encrypted_field_factory.create(sa.String(1024)), nullable=True
+ ),
+ sa.Column(
+ "private_key",
+ encrypted_field_factory.create(sa.String(1024)),
+ nullable=True,
+ ),
+ sa.Column(
+ "private_key_password",
+ encrypted_field_factory.create(sa.String(1024)),
+ nullable=True,
+ ),
+ sa.Column("bind_host", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("bind_port", encrypted_field_factory.create(sa.INTEGER())),
Review Comment:
We don't need these, right? They'll be computed from the SQLAlchemy URI.
##########
superset/models/core.py:
##########
@@ -368,14 +372,38 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ override_ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params: Dict[str, Any] = {}
+ from superset.databases.dao import ( # pylint: disable=import-outside-toplevel
+ DatabaseDAO,
+ )
+
+ if ssh_tunnel := override_ssh_tunnel or DatabaseDAO.get_ssh_tunnel(
+ database_id=self.id
+ ):
+ # if ssh_tunnel is available build engine with information
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
+ ssh_tunnel.bind_host = url.host
+ ssh_tunnel.bind_port = url.port
+ ssh_params = ssh_tunnel.parameters()
+ with sshtunnel.open_tunnel(**ssh_params) as server:
+ yield self._get_sqla_engine(
+ schema=schema,
+ nullpool=nullpool,
+ source=source,
+ ssh_tunnel_server=server,
+ )
+
+ else:
+ yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
Review Comment:
You can write this a bit more cleanly:
```python
if ssh_tunnel := ...:
...
engine_context = sshtunnel.open_tunnel(**ssh_params)
else:
engine_context = contextlib.nullcontext()
with engine_context as context:
yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source, ssh_tunnel_server=context)
```
Also, ideally we wouldn't have to pass `ssh_tunnel_server=server` to `_get_sqla_engine`, since it would be nice to decouple the SSH tunnel from it. Maybe passing the SQL Alchemy URI directly here.
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
They also wouldn't need to be encrypted.
##########
superset/databases/dao.py:
##########
@@ -124,3 +125,13 @@ def get_related_objects(cls, database_id: int) -> Dict[str, Any]:
return dict(
charts=charts, dashboards=dashboards, sqllab_tab_states=sqllab_tab_states
)
+
+ @classmethod
+ def get_ssh_tunnel(cls, database_id: int) -> SSHTunnel:
Review Comment:
```suggestion
def get_ssh_tunnel(cls, database_id: int) -> Optional[SSHTunnel]:
```
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
Review Comment:
Unless I'm mistaken we don't need these encrypted.
##########
superset/databases/ssh_tunnel/commands/create.py:
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOCreateFailedError
+from superset.databases.dao import DatabaseDAO
+from superset.databases.ssh_tunnel.commands.exceptions import SSHTunnelCreateFailedError
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+
+logger = logging.getLogger(__name__)
+
+
+class CreateSSHTunnelCommand(BaseCommand):
+ def __init__(self, database_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._properties["database_id"] = database_id
+
+ def run(self) -> Model:
+ self.validate()
+
+ try:
+ tunnel = SSHTunnelDAO.create(self._properties, commit=False)
+ except DAOCreateFailedError as ex:
+ raise SSHTunnelCreateFailedError() from ex
+
+ return tunnel
+
+ def validate(self) -> None:
+ # TODO(hughhh): check to make sure the server port is not localhost
+ # using the config.SSH_TUNNEL_MANAGER
Review Comment:
If the command is called when the tunnel is added to the DB then we don't want to check the localhost here; we want to check every time the connection is made, since the value of the config might change after the ssh tunnel has been created.
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
Review Comment:
We don't need to encrypt these.
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import EncryptedType, UUIDType
+
+from superset import app
+
+app_config = app.config
+
+
+def upgrade():
Review Comment:
@craig-rueda I think we're always going to have this discussion... :)
Having migrations in separate PRs really helps when you need to cherry-pick a PR with a migration, since when that happens you also need to cherry-pick every PR that has a migration in between, regardless of what it is. If those intermediary PRs are harmless DB migrations the cherry-pick is **much** easier.
My recommendation has been: work on a single PR, since as you said the migrations tend to evolve during development. When the PR is ready, split it into 2. This of course assumes that the DB migration can live independently from the code (eg, it adds a new table or a new column).
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
Review Comment:
Me too, we can read these from the SQLAlchemy URI.
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ bind_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+
+ def parameters(self) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (self.bind_host, self.bind_port),
+ "local_bind_address": ("127.0.0.1",),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
+ elif self.private_key:
+ params["ssh_pkey"] = self.private_key
+ params["ssh_private_key_password"] = self.private_key_password
Review Comment:
Can we please use either `pkey` or `private_key` to indicate a private key?
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
Review Comment:
^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037432242
##########
superset/databases/commands/test_connection.py:
##########
@@ -90,6 +91,14 @@ def run(self) -> None: # pylint: disable=too-many-statements
database.set_sqlalchemy_uri(uri)
database.db_engine_spec.mutate_db_for_connection_test(database)
+ # Generate tunnel if present in the properties
+ ssh_tunnel = None
Review Comment:
Fixed in the API PR pointing to this branch: https://github.com/apache/superset/pull/22199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046038751
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
Review Comment:
the `sshtunnel` lib needs it to be a tuple or it throws this error:
```
118 else:
119 raise ValueError('ADDRESS is not a tuple, string, or character buffer '
120 '({0})'.format(type(address).__name__))
ValueError: ADDRESS not a valid socket domain socket (127.0.0.1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046273299
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
Review Comment:
my bad, I was reading line 77 as doing the assignment, but it's a slightly diff variable name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1045116472
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (bind_host, bind_port),
+ "local_bind_address": (SSH_TUNNELLING_LOCAL_BIND_ADDRESS,),
+ }
+
+ if self.password:
+ params["ssh_password"] = self.password
Review Comment:
params is used populating the in ssh tunnel object, so `ssh_password is the param in the `sshtunnel` class to be instantiated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1046274255
##########
superset/databases/commands/create.py:
##########
@@ -70,13 +71,25 @@ def run(self) -> Model:
try:
database = DatabaseDAO.create(self._properties, commit=False)
database.set_sqlalchemy_uri(database.sqlalchemy_uri)
+ db.session.flush()
+
+ ssh_tunnel = None
+ if ssh_tunnel_properties := self._properties.get("ssh_tunnel"):
+ ssh_tunnel = SSHTunnelDAO.create(
+ {
+ **ssh_tunnel_properties,
+ "database_id": database.id,
+ },
+ commit=False,
Review Comment:
got it, cool, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
betodealmeida commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1049992181
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask import current_app
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset.constants import SSH_TUNNELLING_LOCAL_BIND_ADDRESS
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = current_app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(
+ sa.Integer, sa.ForeignKey("dbs.id"), nullable=False, unique=True
+ )
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(sa.Text)
+ server_port = sa.Column(sa.Integer)
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ def parameters(self, bind_host: str, bind_port: int) -> Dict[str, Any]:
Review Comment:
Add a docstring saying that you're building kwargs for `sshtunnel`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1042409204
##########
tests/unit_tests/databases/ssh_tunnel/commands/delete_test.py:
##########
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterator
+
+import pytest
+from sqlalchemy.orm.session import Session
+
+
+@pytest.fixture
+def session_with_data(session: Session) -> Iterator[Session]:
+ from superset.connectors.sqla.models import SqlaTable
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+ from superset.models.core import Database
+
+ engine = session.get_bind()
+ SqlaTable.metadata.create_all(engine) # pylint: disable=no-member
+
+ db = Database(database_name="my_database", sqlalchemy_uri="sqlite://")
+ sqla_table = SqlaTable(
+ table_name="my_sqla_table",
+ columns=[],
+ metrics=[],
+ database=db,
+ )
+ ssh_tunnel = SSHTunnel(
+ database_id=db.id,
+ database=db,
+ )
+
+ session.add(db)
+ session.add(sqla_table)
+ session.add(ssh_tunnel)
+ session.flush()
+ yield session
+ session.rollback()
+
+
+def test_delete_ssh_tunnel_command(session_with_data: Session) -> None:
+ from superset.databases.dao import DatabaseDAO
+ from superset.databases.ssh_tunnel.commands.delete import DeleteSSHTunnelCommand
+ from superset.databases.ssh_tunnel.models import SSHTunnel
+
+ result = DatabaseDAO.get_ssh_tunnel(1)
Review Comment:
Ok cool, I was adding it there since the API uses the commands. But for sure we can add those here π Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] hughhhh merged pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
hughhhh merged PR #21912:
URL: https://github.com/apache/superset/pull/21912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1035416456
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
Review Comment:
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat(ssh-tunnelling): Setup SSH Tunneling Commands for Database Connections
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1037480649
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
The API validates required fields (POST AND PUT call) and it throws errors if any of them is missing as evidenced in the API PR: https://github.com/apache/superset/pull/22199/files#diff-b94e273d2b41d3fe8a552cbf34cfa59ab9495154d16823b411470ae782cea38aR514
Other than that, the PUT call works like a PATCH behind the scene (based on experimentation while writing the API tests), so users would be able to send: `required field + any combination of optional fields` and the update will work.
Does this meet the validation you mention or do you want to validate also:
1. If `private_key_password` was sent, then `private_key` is required ? (the other way around I believe is not needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on PR #21912:
URL: https://github.com/apache/superset/pull/21912#issuecomment-1293997397
I think it might be better to just add this model into the PR you're planning down the road that adds the SSH handling. It's a little hard to grok what's going on out of context. Also, you almost ALWAYS end up making model tweaks as you implement a new feature, which will require another migration π¬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #21912: feat: Create `ssh_tunnel_credentials` table
Posted by GitBox <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1019223551
##########
superset/databases/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnel"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnel_credentials", cascade="all, delete-orphan"),
Review Comment:
Do we need to rename this to be `ssh_tunnel` in this file?
##########
superset/databases/commands/test_connection.py:
##########
@@ -90,7 +90,13 @@ def run(self) -> None: # pylint: disable=too-many-statements
database.set_sqlalchemy_uri(uri)
database.db_engine_spec.mutate_db_for_connection_test(database)
- engine = database.get_sqla_engine()
+ from superset.databases.models import SSHTunnel
+
+ if self._properties.get("ssh_tunnel_credentials"):
Review Comment:
Do we need to rename this to be `ssh_tunnel` in this file?
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnel_credentials",
Review Comment:
Does this means user will still be creating a `ssh_tunnel_credentials` table?
##########
superset/databases/schemas.py:
##########
@@ -482,6 +502,8 @@ class DatabaseTestConnectionSchema(Schema, DatabaseParametersSchemaMixin):
validate=[Length(1, 1024), sqlalchemy_uri_validator],
)
+ ssh_tunnel_credentials = fields.Nested(DatabaseSSHTunnel, allow_none=True)
Review Comment:
Do we need to rename this to be `ssh_tunnel` in this file?
##########
superset/models/core.py:
##########
@@ -368,17 +372,36 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ ssh_tunnel_credentials: Optional["SSHTunnel"] = None
) -> Engine:
+ ssh_params = None
+ if ssh_tunnel_credentials:
Review Comment:
Do we need to rename this to be `ssh_tunnel` in this file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026955444
##########
superset/databases/ssh_tunnel/models.py:
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict
+
+import sqlalchemy as sa
+from flask_appbuilder import Model
+from sqlalchemy.orm import backref, relationship
+from sqlalchemy_utils import EncryptedType
+
+from superset import app
+from superset.models.core import Database
+from superset.models.helpers import (
+ AuditMixinNullable,
+ ExtraJSONMixin,
+ ImportExportMixin,
+)
+
+app_config = app.config
+
+
+class SSHTunnel(Model, AuditMixinNullable, ExtraJSONMixin, ImportExportMixin):
+ """
+ A ssh tunnel configuration in a database.
+ """
+
+ __tablename__ = "ssh_tunnels"
+
+ id = sa.Column(sa.Integer, primary_key=True)
+ database_id = sa.Column(sa.Integer, sa.ForeignKey("dbs.id"), nullable=False)
+ database: Database = relationship(
+ "Database",
+ backref=backref("ssh_tunnels", cascade="all, delete-orphan"),
+ foreign_keys=[database_id],
+ )
+
+ server_address = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ server_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+ username = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+
+ # basic authentication
+ password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ # password protected pkey authentication
+ private_key = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+ private_key_password = sa.Column(
+ EncryptedType(sa.String, app_config["SECRET_KEY"]), nullable=True
+ )
+
+ bind_host = sa.Column(EncryptedType(sa.String, app_config["SECRET_KEY"]))
+ bind_port = sa.Column(EncryptedType(sa.Integer, app_config["SECRET_KEY"]))
+
+ def parameters(self) -> Dict[str, Any]:
+ params = {
+ "ssh_address_or_host": self.server_address,
+ "ssh_port": self.server_port,
+ "ssh_username": self.username,
+ "remote_bind_address": (self.bind_host, self.bind_port),
+ "local_bind_address": ("127.0.0.1",),
Review Comment:
is this supposed to be a hard coded value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026950194
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
Review Comment:
I think we want to make sure that any database rollbacks will also roll back sshTunnel writes. Why are we doing this commit=True here? It seems that we should wait until the database changes are also committed so that they're in the same transaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028370629
##########
superset/databases/ssh_tunnel/commands/update.py:
##########
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Dict, List, Optional
+
+from flask_appbuilder.models.sqla import Model
+from marshmallow import ValidationError
+
+from superset.commands.base import BaseCommand
+from superset.dao.exceptions import DAOUpdateFailedError
+from superset.databases.ssh_tunnel.commands.exceptions import (
+ SSHTunnelDeleteFailedError,
+ SSHTunnelInvalidError,
+ SSHTunnelNotFoundError,
+ SSHTunnelUpdateFailedError,
+)
+from superset.databases.ssh_tunnel.dao import SSHTunnelDAO
+from superset.databases.ssh_tunnel.models import SSHTunnel
+
+logger = logging.getLogger(__name__)
+
+
+class UpdateSSHTunnelCommand(BaseCommand):
+ def __init__(self, model_id: int, data: Dict[str, Any]):
+ self._properties = data.copy()
+ self._model_id = model_id
+ self._model: Optional[SSHTunnel] = None
+
+ def run(self) -> Model:
+ self.validate()
+ if not self._model:
+ raise SSHTunnelNotFoundError()
+
+ try:
+ tunnel = SSHTunnelDAO.update(self._model, self._properties, commit=True)
+ except DAOUpdateFailedError as ex:
+ raise SSHTunnelUpdateFailedError() from ex
+ return tunnel
+
+ def validate(self) -> None:
Review Comment:
I believe our updates behave like a PUT, and send all the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1028368730
##########
superset/databases/ssh_tunnel/commands/exceptions.py:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from flask_babel import lazy_gettext as _
+
+from superset.commands.exceptions import (
+ CommandException,
+ CommandInvalidError,
+ DeleteFailedError,
+ UpdateFailedError,
+)
+
+
+class SSHTunnelDeleteFailedError(DeleteFailedError):
+ message = _("SSH Tunnel could not be deleted.")
+
+
+class SSHTunnelNotFoundError(CommandException):
+ status = 404
+ message = _("SSH Tunnel not found.")
+
+
+class SSHTunnelInvalidError(CommandInvalidError):
+ message = _("SSH Tunnel parameters are invalid.")
Review Comment:
Ok great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026956904
##########
superset/migrations/versions/2022-10-20_10-48_f3c2d8ec8595_create_ssh_tunnel_credentials_tbl.py:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""create_ssh_tunnel_credentials_tbl
+
+Revision ID: f3c2d8ec8595
+Revises: deb4c9d4a4ef
+Create Date: 2022-10-20 10:48:08.722861
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "f3c2d8ec8595"
+down_revision = "deb4c9d4a4ef"
+
+from uuid import uuid4
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy_utils import UUIDType
+
+from superset import app
+from superset.extensions import encrypted_field_factory
+
+app_config = app.config
+
+
+def upgrade():
+ op.create_table(
+ "ssh_tunnels",
+ # AuditMixinNullable
+ sa.Column("created_on", sa.DateTime(), nullable=True),
+ sa.Column("changed_on", sa.DateTime(), nullable=True),
+ sa.Column("created_by_fk", sa.Integer(), nullable=True),
+ sa.Column("changed_by_fk", sa.Integer(), nullable=True),
+ # ExtraJSONMixin
+ sa.Column("extra_json", sa.Text(), nullable=True),
+ # ImportExportMixin
+ sa.Column("uuid", UUIDType(binary=True), primary_key=False, default=uuid4),
+ # SSHTunnelCredentials
+ sa.Column("id", sa.Integer(), primary_key=True),
+ sa.Column("database_id", sa.INTEGER(), sa.ForeignKey("dbs.id")),
+ sa.Column("server_address", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column("server_port", encrypted_field_factory.create(sa.INTEGER())),
+ sa.Column("username", encrypted_field_factory.create(sa.String(1024))),
+ sa.Column(
+ "password", encrypted_field_factory.create(sa.String(1024)), nullable=True
+ ),
+ sa.Column(
+ "private_key",
+ encrypted_field_factory.create(sa.String(1024)),
+ nullable=True,
+ ),
+ sa.Column(
+ "private_key_password",
+ encrypted_field_factory.create(sa.String(1024)),
Review Comment:
1024 seems a bit high for these values.. especially server address, username and password.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] eschutho commented on a diff in pull request #21912: feat: Enable SSH Tunneling for Database Connections
Posted by GitBox <gi...@apache.org>.
eschutho commented on code in PR #21912:
URL: https://github.com/apache/superset/pull/21912#discussion_r1026958181
##########
superset/models/core.py:
##########
@@ -368,14 +372,41 @@ def get_sqla_engine_with_context(
schema: Optional[str] = None,
nullpool: bool = True,
source: Optional[utils.QuerySource] = None,
+ ssh_tunnel: Optional["SSHTunnel"] = None,
) -> Engine:
- yield self._get_sqla_engine(schema=schema, nullpool=nullpool, source=source)
+ ssh_params = {}
+ if ssh_tunnel:
+ # build with override
+ url = make_url_safe(self.sqlalchemy_uri_decrypted)
+ ssh_tunnel.bind_host = url.host
+ ssh_tunnel.bind_port = url.port
+ ssh_params = ssh_tunnel.parameters()
+ try:
+ with sshtunnel.open_tunnel(**ssh_params) as server:
+ yield self._get_sqla_engine(
+ schema=schema,
+ nullpool=nullpool,
+ source=source,
+ ssh_tunnel_server=server,
+ )
+ except Exception as ex:
+ raise ex
Review Comment:
why are we catching and reraising the same thing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org