You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Reuven Lax (JIRA)" <ji...@apache.org> on 2017/06/05 15:09:04 UTC
[jira] [Updated] (BEAM-2370) BigQuery Insert with Partition
Decorator throwing error
[ https://issues.apache.org/jira/browse/BEAM-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reuven Lax updated BEAM-2370:
-----------------------------
I believe this is a know issue. For now if you pass in null for the table
description (new TableDestination("my-project:dataset.mytable_orders$" +
dayString, null)) I believe this will work.
> BigQuery Insert with Partition Decorator throwing error
> -------------------------------------------------------
>
> Key: BEAM-2370
> URL: https://issues.apache.org/jira/browse/BEAM-2370
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
> Affects Versions: 2.0.0
> Environment: DirectRunner
> Reporter: Andre
> Assignee: Reuven Lax
>
> Running a DataFlow job with the DirectRunner which is inserting data into a partitioned table using decorators throws the following error multiple times BUT still inserts records into the right partition.
> {code:java|title=Error}
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> INFO: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
> {
> "code" : 400,
> "errors" : [ {
> "domain" : "global",
> "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used.",
> "reason" : "invalid"
> } ],
> "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."
> }
> at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> {code}
> {code:java|title=Code}
> // Write TableRows to BQ
> rows.apply("TransformationStep", ParDo.of(new Outputter()))
> .apply("WindowDaily", Window. <TableRow> into(CalendarWindows.days(1)))
> .apply("WriteToBQ", BigQueryIO.writeTableRows()
> .to(new SerializableFunction <ValueInSingleWindow<TableRow> , TableDestination> () {
> private static final long serialVersionUID = 8196602721734820219 L;
> @Override
> public TableDestination apply(ValueInSingleWindow <TableRow> value) {
> String dayString = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC)
> .print(((IntervalWindow) value.getWindow()).start());
> TableDestination td = new TableDestination("my-project:dataset.mytable_orders$" + dayString, "");
> return td;
> }
> }).withSchema(mySchema)
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
> .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)