[jira] [Commented] (AIRFLOW-3077) Mongo Hook Raise Error and Stop Migration Due to Bad Encoding from PyMongo

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723734#comment-16723734 ] 

ASF GitHub Bot commented on AIRFLOW-3077:
-----------------------------------------

stale[bot] closed pull request #3912: [AIRFLOW-3077] Default Not to Raise Error When PyMongo Contruct JSON Data
URL: https://github.com/apache/incubator-airflow/pull/3912
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/hooks/mongo_hook.py b/airflow/contrib/hooks/mongo_hook.py
index 80ceddec14..bd495eb294 100644
--- a/airflow/contrib/hooks/mongo_hook.py
+++ b/airflow/contrib/hooks/mongo_hook.py
@@ -70,6 +70,9 @@ def get_conn(self):
         if options.get('ssl', False):
             options.update({'ssl_cert_reqs': CERT_NONE})
 
+        if not options.get('unicode_decode_error_handler', False):
+            options.update({'unicode_decode_error_handler': 'ignore'})
+
         self.client = MongoClient(uri, **options)
 
         return self.client


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Mongo Hook Raise Error and Stop Migration Due to Bad Encoding from PyMongo
> --------------------------------------------------------------------------
>
>                 Key: AIRFLOW-3077
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3077
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: database, hooks
>    Affects Versions: 1.9.0, 1.10.0
>            Reporter: Bernie Chiu
>            Assignee: Bernie Chiu
>            Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Due to the fact that a single encoding problem should not stop the dataflow, provide the default with `ignore` option is best for default since PyMongo will still try to do its best to reconstruct the JSON data.
> [https://stackoverflow.com/questions/36314776/pymongo-error-bson-errors-invalidbson-utf8-codec-cant-decode-byte-0xa1-in-p]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)