[GitHub] [incubator-superset] mistercrunch commented on a change in pull request #7756: Add csv upload support for BigQuery

[GitBox <gi...@apache.org>]

mistercrunch commented on a change in pull request #7756: Add csv upload support for BigQuery
URL: https://github.com/apache/incubator-superset/pull/7756#discussion_r296450907
 
 

 ##########
 File path: superset/db_engine_specs/bigquery.py
 ##########
 @@ -141,3 +141,27 @@ def epoch_to_dttm(cls):
     @classmethod
     def epoch_ms_to_dttm(cls):
         return 'TIMESTAMP_MILLIS({col})'
+
+    @classmethod
+    def df_to_sql(cls, df: pd.DataFrame, **kwargs):
+        """
+        Upload data from a Pandas DataFrame to BigQuery. Calls
+        `DataFrame.to_gbq()` which requires `pandas_gbq` to be installed.
+
+        :param df: Dataframe with data to be uploaded
+        :param kwargs: kwargs to be passed to to_gbq() method. Requires both `schema
+        and ``name` to be present in kwargs, which are combined and passed to
+        `to_gbq()` as `destination_table`.
+        """
+        if not ('name' in kwargs and 'schema' in kwargs):
+            raise Exception('name and schema need to be defined in kwargs')
+        gbq_kwargs = {}
+        gbq_kwargs['project_id'] = kwargs['con'].engine.url.host
+        gbq_kwargs['destination_table'] = f"{kwargs.pop('schema')}.{kwargs.pop('name')}"
+
+        # Only pass through supported kwargs
+        supported_kwarg_keys = {'if_exists'}
+        for key in supported_kwarg_keys:
+            if key in kwargs:
+                gbq_kwargs[key] = kwargs[key]
+        df.to_gbq(**gbq_kwargs)
 
 Review comment:
   I'm wondering what the error raised will be if `pandas_gbq` isn't installed. I'm guessing it may be gibberish to most, as in `df object has no attribute to_gbq` which would be confusing to most users and admins.
   
   If that's the case, how about we assert that the lib is installed and if not raise an `ImportError` with a comprehensive message, as in: "Could not import the library `pandas_gbq`, which is required to be installed in your environment in order to load CSV files to BigQuery".

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org