You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "betodealmeida (via GitHub)" <gi...@apache.org> on 2023/08/09 01:52:13 UTC
[GitHub] [superset] betodealmeida opened a new pull request, #24921: feat(gsheets): file upload
betodealmeida opened a new pull request, #24921:
URL: https://github.com/apache/superset/pull/24921
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Add a `df_to_sql` method to the GSheets DB engine spec, so we can upload CSV files. When a file is uploaded, a new sheet is created, and the database catalog is updated to point the table name to the URL of the new sheet.
The user can also choose to replace an existing table, in which case the sheet gets cleared before the data is inserted.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
Inserting some data:

Querying the table:

Uploading a different CSV and replacing the table:

And querying it (note that the metadata is cached and shows the old schema):

And the sheet after the second upload:

### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida merged pull request #24921: feat(gsheets): file upload
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida merged PR #24921:
URL: https://github.com/apache/superset/pull/24921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #24921: feat(gsheets): file upload
Posted by "Antonio-RiveroMartnez (via GitHub)" <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1291171609
##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
class GSheetsParametersType(TypedDict):
service_account_info: str
- catalog: Optional[dict[str, str]]
+ catalog: dict[str, str] | None
Review Comment:
Oh cool! Good to know moving forward then. Thanks @betodealmeida
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #24921: feat(gsheets): file upload
Posted by "Antonio-RiveroMartnez (via GitHub)" <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1290878333
##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
class GSheetsParametersType(TypedDict):
service_account_info: str
- catalog: Optional[dict[str, str]]
+ catalog: dict[str, str] | None
Review Comment:
Just curious why moving from `Optional` ? Because I see `Optional` being used in several other `TypedDict` classes. Want to confirm if this is a pattern we aim to follow moving forward, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] betodealmeida commented on a diff in pull request #24921: feat(gsheets): file upload
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1290882849
##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
class GSheetsParametersType(TypedDict):
service_account_info: str
- catalog: Optional[dict[str, str]]
+ catalog: dict[str, str] | None
Review Comment:
We bumped the minimum Python version we support in Superset (I think to 3.9?), and in it we can use the syntax `foo | None` instead of `Optional[foo]`. This way we don't need to import `Optional`, and the type declaration looks more natural (for example, it's similar to Typescript's `foo | null` for an optional `foo`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #24921: feat(gsheets): file upload
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #24921:
URL: https://github.com/apache/superset/pull/24921#issuecomment-1671616077
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#24921](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bce295c) into [master](https://app.codecov.io/gh/apache/superset/commit/81bf2f01e237a8405f567115d577571c70702100?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (81bf2f0) will **decrease** coverage by `2.48%`.
> Report is 1 commits behind head on master.
> The diff coverage is `29.82%`.
> :exclamation: Current head bce295c differs from pull request most recent head c1b020a. Consider uploading reports for the commit c1b020a to get more accurate results
```diff
@@ Coverage Diff @@
## master #24921 +/- ##
==========================================
- Coverage 69.00% 66.52% -2.48%
==========================================
Files 1906 1906
Lines 74132 74154 +22
Branches 8208 8208
==========================================
- Hits 51153 49331 -1822
- Misses 20856 22700 +1844
Partials 2123 2123
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `54.13% <29.82%> (-0.05%)` | :arrow_down: |
| mysql | `?` | |
| postgres | `?` | |
| presto | `54.03% <29.82%> (-0.05%)` | :arrow_down: |
| python | `78.17% <29.82%> (-5.20%)` | :arrow_down: |
| sqlite | `77.81% <29.82%> (-0.09%)` | :arrow_down: |
| unit | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...tend/plugins/legacy-preset-chart-nvd3/src/utils.js](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9sZWdhY3ktcHJlc2V0LWNoYXJ0LW52ZDMvc3JjL3V0aWxzLmpz) | `16.02% <ø> (+0.10%)` | :arrow_up: |
| [superset/db\_engine\_specs/gsheets.py](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2dzaGVldHMucHk=) | `39.03% <29.82%> (-37.44%)` | :arrow_down: |
... and [137 files with indirect coverage changes](https://app.codecov.io/gh/apache/superset/pull/24921/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org