[jira] [Commented] (AIRFLOW-4363) Encounter JSON Decode Error when using docker operator

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051787#comment-17051787 ] 

ASF GitHub Bot commented on AIRFLOW-4363:
-----------------------------------------

retornam commented on pull request #7628: [AIRFLOW-4363] Fix JSON encoding error
URL: https://github.com/apache/airflow/pull/7628
 
 
   From the docker-py code comments for APIClient pull,
   the decode parameter should be set to True, when the
   stream parameter is also set to True. This will allow
   decoding JSON data returned from the docker registry
   server into dicts
   
   Signed-off-by: Raymond Etornam <re...@users.noreply.github.com>
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x ] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Encounter JSON Decode Error when using docker operator
> ------------------------------------------------------
>
>                 Key: AIRFLOW-4363
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4363
>             Project: Apache Airflow
>          Issue Type: Bug
>         Environment: - Mac OS 10.14
> - python 3.6.8
> - airflow 1.10.2
>            Reporter: Ben Chen
>            Assignee: Ben Chen
>            Priority: Blocker
>
> *[Description]* . 
> When using the docker_operator, I experienced some issue while using Mac OS 10.14.4. The error was json.JSONDecodeError. After my investigation about this error, I found that there are several messages for logging aren't well separated, for example it contains \n inside one single message which should be split into 2 to more different messages.
> *[Update]*  
> Confirmed that issue came from the implementation in airflow, issue cannot be solved by just passing `decode` to parameter in docker.pull method in docker api.
> *[Solution]* . 
>  For now, I use try-catch to run the original implementation, and in the exception part I split the message to list and then parse it. Looking for simpler solution to this non critical but still blocking point.
> *[Logs]*  
> {docker_operator.py:188}
> INFO - Starting docker container from image hello-world
> {docker_operator.py:202}
> INFO - Pulling docker image hello-world
> {docker_operator.py:207}
> INFO - Pulling from library/hello-world
> {docker_operator.py:207}
> INFO - Pulling fs layer
> {docker_operator.py:207}
> INFO - Downloading
> {docker_operator.py:207}
> INFO - Downloading
> {docker_operator.py:207}
> INFO - Download complete
> {docker_operator.py:207}
> INFO - Extracting
> {docker_operator.py:207}
> INFO - Extracting
> {docker_operator.py:207}
> INFO - Pull complete
> {docker_operator.py:207}
> INFO - Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
> {docker_operator.py:207}
> INFO - Pulling from library/hello-world
> {docker_operator.py:207}
> INFO - Digest: sha256:1a67c1115b199aa9d964d5da5646917cbac2d5450c71a1deed7b1bfb79c2c82d
> {models.py:1788}
> ERROR - Extra data: line 2 column 1 (char 70)
>  Traceback (most recent call last):
>  line 1657, in _run_raw_task, result = task_copy.execute(context=context)
>  line 205, in execute output = json.loads(line)
>  line 354, in loads, return _default_decoder.decode(s)
>  line 342, in decode, raise JSONDecodeError("Extra data", s, end)
>  json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 70)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)