You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ben Chen (JIRA)" <ji...@apache.org> on 2019/04/20 07:54:00 UTC
[jira] [Updated] (AIRFLOW-4363) Encounter JSON Decode Error when
using docker operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben Chen updated AIRFLOW-4363:
------------------------------
Description:
*[Description]* .
When using the docker_operator, I experienced some issue while using Mac OS 10.14.4. The error was json.JSONDecodeError. After my investigation about this error, I found that there are several messages for logging aren't well separated, for example it contains \n inside one single message which should be split into 2 to more different messages.
*[Update]*
Confirmed that issue came from the implementation in airflow, issue cannot be solved by just passing `decode` to parameter in docker.pull method in docker api.
*[Solution]* .
For now, I use try-catch to run the original implementation, and in the exception part I split the message to list and then parse it. Looking for simpler solution to this non critical but still blocking point.
*[Logs]*
{docker_operator.py:188}
INFO - Starting docker container from image hello-world
{docker_operator.py:202}
INFO - Pulling docker image hello-world
{docker_operator.py:207}
INFO - Pulling from library/hello-world
{docker_operator.py:207}
INFO - Pulling fs layer
{docker_operator.py:207}
INFO - Downloading
{docker_operator.py:207}
INFO - Downloading
{docker_operator.py:207}
INFO - Download complete
{docker_operator.py:207}
INFO - Extracting
{docker_operator.py:207}
INFO - Extracting
{docker_operator.py:207}
INFO - Pull complete
{docker_operator.py:207}
INFO - Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
{docker_operator.py:207}
INFO - Pulling from library/hello-world
{docker_operator.py:207}
INFO - Digest: sha256:1a67c1115b199aa9d964d5da5646917cbac2d5450c71a1deed7b1bfb79c2c82d
{models.py:1788}
ERROR - Extra data: line 2 column 1 (char 70)
Traceback (most recent call last):
line 1657, in _run_raw_task, result = task_copy.execute(context=context)
line 205, in execute output = json.loads(line)
line 354, in loads, return _default_decoder.decode(s)
line 342, in decode, raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 70)
was:
## Description
When using the docker_operator, I experienced some issue while using Mac OS 10.14.4. The error was json.JSONDecodeError. After my investigation about this error, I found that there are several messages for logging aren't well separated, for example it contains \n inside one single message which should be split into 2 to more different messages.
[Update]
Confirmed that issue came from the implementation in airflow, issue cannot be solved by just passing `decode` to parameter in docker.pull method in docker api.
[Solution]
For now, I use try-catch to run the original implementation, and in the exception part I split the message to list and then parse it. Looking for simpler solution to this non critical but still blocking point.
## Logs
{docker_operator.py:188} INFO - Starting docker container from image hello-world
{docker_operator.py:202} INFO - Pulling docker image hello-world
{docker_operator.py:207} INFO - Pulling from library/hello-world
{docker_operator.py:207} INFO - Pulling fs layer
{docker_operator.py:207} INFO - Downloading
{docker_operator.py:207} INFO - Downloading
{docker_operator.py:207} INFO - Download complete
{docker_operator.py:207} INFO - Extracting
{docker_operator.py:207} INFO - Extracting
{docker_operator.py:207} INFO - Pull complete
{docker_operator.py:207} INFO - Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
{docker_operator.py:207} INFO - Pulling from library/hello-world
{docker_operator.py:207} INFO - Digest: sha256:1a67c1115b199aa9d964d5da5646917cbac2d5450c71a1deed7b1bfb79c2c82d
{models.py:1788} ERROR - Extra data: line 2 column 1 (char 70)
Traceback (most recent call last):
line 1657, in _run_raw_task, result = task_copy.execute(context=context)
line 205, in execute output = json.loads(line)
line 354, in loads, return _default_decoder.decode(s)
line 342, in decode, raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 70)
> Encounter JSON Decode Error when using docker operator
> ------------------------------------------------------
>
> Key: AIRFLOW-4363
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4363
> Project: Apache Airflow
> Issue Type: Bug
> Environment: - Mac OS 10.14
> - python 3.6.8
> - airflow 1.10.2
> Reporter: Ben Chen
> Assignee: Ben Chen
> Priority: Blocker
>
> *[Description]* .
> When using the docker_operator, I experienced some issue while using Mac OS 10.14.4. The error was json.JSONDecodeError. After my investigation about this error, I found that there are several messages for logging aren't well separated, for example it contains \n inside one single message which should be split into 2 to more different messages.
> *[Update]*
> Confirmed that issue came from the implementation in airflow, issue cannot be solved by just passing `decode` to parameter in docker.pull method in docker api.
> *[Solution]* .
> For now, I use try-catch to run the original implementation, and in the exception part I split the message to list and then parse it. Looking for simpler solution to this non critical but still blocking point.
> *[Logs]*
> {docker_operator.py:188}
> INFO - Starting docker container from image hello-world
> {docker_operator.py:202}
> INFO - Pulling docker image hello-world
> {docker_operator.py:207}
> INFO - Pulling from library/hello-world
> {docker_operator.py:207}
> INFO - Pulling fs layer
> {docker_operator.py:207}
> INFO - Downloading
> {docker_operator.py:207}
> INFO - Downloading
> {docker_operator.py:207}
> INFO - Download complete
> {docker_operator.py:207}
> INFO - Extracting
> {docker_operator.py:207}
> INFO - Extracting
> {docker_operator.py:207}
> INFO - Pull complete
> {docker_operator.py:207}
> INFO - Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
> {docker_operator.py:207}
> INFO - Pulling from library/hello-world
> {docker_operator.py:207}
> INFO - Digest: sha256:1a67c1115b199aa9d964d5da5646917cbac2d5450c71a1deed7b1bfb79c2c82d
> {models.py:1788}
> ERROR - Extra data: line 2 column 1 (char 70)
> Traceback (most recent call last):
> line 1657, in _run_raw_task, result = task_copy.execute(context=context)
> line 205, in execute output = json.loads(line)
> line 354, in loads, return _default_decoder.decode(s)
> line 342, in decode, raise JSONDecodeError("Extra data", s, end)
> json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 70)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)