You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Steven Ensslen (JIRA)" <ji...@apache.org> on 2019/06/16 22:22:00 UTC
[jira] [Comment Edited] (BEAM-5510) Records including datetime to
be saved as DATETIME or TIMESTAMP in BigQuery
[ https://issues.apache.org/jira/browse/BEAM-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865186#comment-16865186 ]
Steven Ensslen edited comment on BEAM-5510 at 6/16/19 10:21 PM:
----------------------------------------------------------------
And the same problem now with `int`s. Possibly due to upgrade to {{[google-cloud-bigquery|https://pypi.python.org/pypi/google-cloud-bigquery]==1.11.3}}
{quote}{color:#000000}WriteToBigQuery/NativeWrite failed., BigQuery import job "dataflow_job_2697474504339973542-B" failed., BigQuery job "dataflow_job_2697474504339973542-B" in project "poco-analytics-factory" finished with error(s): errorResult: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details., error: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details., error: Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: ciwr_id; Value: 160094{color}
{quote}
was (Author: ensslen):
And the same problem now with `int`s?
{quote}{color:#000000}WriteToBigQuery/NativeWrite failed., BigQuery import job "dataflow_job_2697474504339973542-B" failed., BigQuery job "dataflow_job_2697474504339973542-B" in project "poco-analytics-factory" finished with error(s): errorResult: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details., error: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details., error: Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: ciwr_id; Value: 160094{color}
{quote}
> Records including datetime to be saved as DATETIME or TIMESTAMP in BigQuery
> ---------------------------------------------------------------------------
>
> Key: BEAM-5510
> URL: https://issues.apache.org/jira/browse/BEAM-5510
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.6.0
> Reporter: Pascal Gula
> Priority: Major
>
> When trying to write some row in BigQuery that include a python datetime object, the marshaling used to save a row in BigQuery is impossible.
> {code:java}
> File "/home/pascal/Wks/GitHub/PEAT-AI/Albatros/venv/local/lib/python2.7/site-packages/apache_beam/internal/gcp/json_value.py", line 124, in to_json_value
> raise TypeError('Cannot convert %s to a JSON value.' % repr(obj))
> TypeError: Cannot convert datetime.datetime(2018, 9, 25, 18, 57, 18, 108579) to a JSON value. [while running 'save/WriteToBigQuery']
> {code}
> However, this is something perfectly feasible, as `google-cloud-python` supports it since this issue has been solved: [https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2957]
> thanks to this pull request: [https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3426/files]
> As similar approach could be taken for the `json_value.py` helper.
> Is there any workaround that can be applied to solve this issue?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)