You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "betodealmeida (via GitHub)" <gi...@apache.org> on 2023/08/09 01:52:13 UTC

[GitHub] [superset] betodealmeida opened a new pull request, #24921: feat(gsheets): file upload

betodealmeida opened a new pull request, #24921:
URL: https://github.com/apache/superset/pull/24921

   <!---
   Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
   Example:
   fix(dashboard): load charts correctly
   -->
   
   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   
   Add a `df_to_sql` method to the GSheets DB engine spec, so we can upload CSV files. When a file is uploaded, a new sheet is created, and the database catalog is updated to point the table name to the URL of the new sheet.
   
   The user can also choose to replace an existing table, in which case the sheet gets cleared before the data is inserted.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   Inserting some data:
   
   ![Screenshot 2023-08-08 at 18-40-33 Superset](https://github.com/apache/superset/assets/1534870/a2b2edfe-dd8c-4860-9885-494fc518a9ce)
   
   Querying the table:
   
   ![Screenshot 2023-08-08 at 18-42-05 Superset](https://github.com/apache/superset/assets/1534870/1ec6429f-db88-479c-b984-b5da452050e8)
   
   Uploading a different CSV and replacing the table:
   
   ![Screenshot 2023-08-08 at 18-43-21 Superset](https://github.com/apache/superset/assets/1534870/df119ffb-bf58-4bfb-8f25-3db58fcac9a8)
   
   And querying it (note that the metadata is cached and shows the old schema):
   
   ![Screenshot 2023-08-08 at 18-44-24 Superset](https://github.com/apache/superset/assets/1534870/38dc15f6-86bd-4b41-bb02-013200d19d8b)
   
   And the sheet after the second upload:
   
   ![Screenshot 2023-08-08 at 18-44-46 upload_test](https://github.com/apache/superset/assets/1534870/82c02133-59ed-4bb3-8893-cbd445c21805)
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] betodealmeida merged pull request #24921: feat(gsheets): file upload

Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida merged PR #24921:
URL: https://github.com/apache/superset/pull/24921


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #24921: feat(gsheets): file upload

Posted by "Antonio-RiveroMartnez (via GitHub)" <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1291171609


##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
 
 class GSheetsParametersType(TypedDict):
     service_account_info: str
-    catalog: Optional[dict[str, str]]
+    catalog: dict[str, str] | None

Review Comment:
   Oh cool! Good to know moving forward then. Thanks @betodealmeida 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] Antonio-RiveroMartnez commented on a diff in pull request #24921: feat(gsheets): file upload

Posted by "Antonio-RiveroMartnez (via GitHub)" <gi...@apache.org>.
Antonio-RiveroMartnez commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1290878333


##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
 
 class GSheetsParametersType(TypedDict):
     service_account_info: str
-    catalog: Optional[dict[str, str]]
+    catalog: dict[str, str] | None

Review Comment:
   Just curious why moving from `Optional` ? Because I see `Optional` being used in several other `TypedDict` classes. Want to confirm if this is a pattern we aim to follow moving forward, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] betodealmeida commented on a diff in pull request #24921: feat(gsheets): file upload

Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on code in PR #24921:
URL: https://github.com/apache/superset/pull/24921#discussion_r1290882849


##########
superset/db_engine_specs/gsheets.py:
##########
@@ -57,7 +71,7 @@ class GSheetsParametersSchema(Schema):
 
 class GSheetsParametersType(TypedDict):
     service_account_info: str
-    catalog: Optional[dict[str, str]]
+    catalog: dict[str, str] | None

Review Comment:
   We bumped the minimum Python version we support in Superset (I think to 3.9?), and in it we can use the syntax `foo | None` instead of `Optional[foo]`. This way we don't need to import `Optional`, and the type declaration looks more natural (for example, it's similar to Typescript's `foo | null` for an optional `foo`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] codecov[bot] commented on pull request #24921: feat(gsheets): file upload

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #24921:
URL: https://github.com/apache/superset/pull/24921#issuecomment-1671616077

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#24921](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bce295c) into [master](https://app.codecov.io/gh/apache/superset/commit/81bf2f01e237a8405f567115d577571c70702100?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (81bf2f0) will **decrease** coverage by `2.48%`.
   > Report is 1 commits behind head on master.
   > The diff coverage is `29.82%`.
   
   > :exclamation: Current head bce295c differs from pull request most recent head c1b020a. Consider uploading reports for the commit c1b020a to get more accurate results
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #24921      +/-   ##
   ==========================================
   - Coverage   69.00%   66.52%   -2.48%     
   ==========================================
     Files        1906     1906              
     Lines       74132    74154      +22     
     Branches     8208     8208              
   ==========================================
   - Hits        51153    49331    -1822     
   - Misses      20856    22700    +1844     
     Partials     2123     2123              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hive | `54.13% <29.82%> (-0.05%)` | :arrow_down: |
   | mysql | `?` | |
   | postgres | `?` | |
   | presto | `54.03% <29.82%> (-0.05%)` | :arrow_down: |
   | python | `78.17% <29.82%> (-5.20%)` | :arrow_down: |
   | sqlite | `77.81% <29.82%> (-0.09%)` | :arrow_down: |
   | unit | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...tend/plugins/legacy-preset-chart-nvd3/src/utils.js](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQtZnJvbnRlbmQvcGx1Z2lucy9sZWdhY3ktcHJlc2V0LWNoYXJ0LW52ZDMvc3JjL3V0aWxzLmpz) | `16.02% <ø> (+0.10%)` | :arrow_up: |
   | [superset/db\_engine\_specs/gsheets.py](https://app.codecov.io/gh/apache/superset/pull/24921?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL2dzaGVldHMucHk=) | `39.03% <29.82%> (-37.44%)` | :arrow_down: |
   
   ... and [137 files with indirect coverage changes](https://app.codecov.io/gh/apache/superset/pull/24921/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org