You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/12 17:38:00 UTC
[jira] [Work logged] (BEAM-5788) wordcount_fnapi_it failed on
TestDataflowRunner because of JSON string decoding error
[ https://issues.apache.org/jira/browse/BEAM-5788?focusedWorklogId=165034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-165034 ]
ASF GitHub Bot logged work on BEAM-5788:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Nov/18 17:37
Start Date: 12/Nov/18 17:37
Worklog Time Spent: 10m
Work Description: markflyhigh opened a new pull request #7010: [BEAM-5788] Fix storage client encoding issue in Python3
URL: https://github.com/apache/beam/pull/7010
DataflowRunner needs to specify response encoding when running in Python 3. This is required when using `google-apitools`.
+R: @aaltay @tvalentyn
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 165034)
Time Spent: 10m
Remaining Estimate: 0h
> wordcount_fnapi_it failed on TestDataflowRunner because of JSON string decoding error
> -------------------------------------------------------------------------------------
>
> Key: BEAM-5788
> URL: https://issues.apache.org/jira/browse/BEAM-5788
> Project: Beam
> Issue Type: Sub-task
> Components: test-failures
> Reporter: Mark Liu
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Similar to BEAM-5785, wordcount_fnapi_it failed on Python 3 when running with TestDataflowRunner. Got TypeError: the JSON object must be str, not 'bytes'. This error cause infinite retry before job could submitted to service.
> More details about my env and test:
> Python version: 3.5.3
> Test: apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it
> Command:
> {code}
> python setup.py nosetests \
> --tests apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_fnapi_it \
> --nocapture \
> --nologcapture \
> --test-pipeline-options=" \
> --runner=TestDataflowRunner \
> --project=<my_project> \
> --staging_location=<my_staging> \
> --temp_location=<my_temp> \
> --output=<my_output> \
> --sdk_location=.../beam/sdks/python/dist/apache-beam-2.9.0.dev0.tar.gz \
> --num_workers=1"
> {code}
> Stacktrace:
> {code}
> WARNING:root:Retry with exponential backoff: waiting for 7.661876827680761 seconds before retrying exists because we caught exception: TypeError: the JSON object must be str, not 'bytes'
> Traceback for above exception (most recent call last):
> File ".../beam/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
> return fun(*args, **kwargs)
> File ".../beam/sdks/python/apache_beam/io/gcp/gcsio.py", line 375, in exists
> self.client.objects.Get(request) # metadata
> File ".../beam/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 955, in Get
> download=download)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 722, in _RunMethod
> return self.ProcessHttpResponse(method_config, http_response, request)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 728, in ProcessHttpResponse
> self.__ProcessHttpResponse(method_config, http_response, request))
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 611, in __ProcessHttpResponse
> response_type, http_response.content)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 442, in DeserializeMessage
> message = encoding.JsonToMessage(response_type, data)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py", line 104, in JsonToMessage
> return _ProtoJsonApiTools.Get().decode_message(message_type, message)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/py/encoding.py", line 290, in decode_message
> message_type, result)
> File ".../tmp/virtualenvs/py3-env/lib/python3.5/site-packages/apitools/base/protorpclite/protojson.py", line 210, in decode_message
> dictionary = json.loads(encoded_message)
> File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
> s.__class__.__name__))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)