[jira] [Created] (BEAM-2370) BigQuery Insert with Partition Decorator throwing error

["Andre (JIRA)" <ji...@apache.org>]

Andre created BEAM-2370:
---------------------------

             Summary: BigQuery Insert with Partition Decorator throwing error
                 Key: BEAM-2370
                 URL: https://issues.apache.org/jira/browse/BEAM-2370
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-gcp
    Affects Versions: 2.0.0
         Environment: DirectRunner
            Reporter: Andre
            Assignee: Daniel Halperin


Running a DataFlow job with the DirectRunner which is inserting data into a partitioned table using decorators throws the following error multiple times BUT still inserts records into the right partition.


{code:java|title=Error}
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
INFO: Ignore the error and retry the request.
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used.",
    "reason" : "invalid"
  } ],
  "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
{code}

{code:java|title=Code}
// Write TableRows to BQ
rows.apply("TransformationStep", ParDo.of(new Outputter()))
 .apply("WindowDaily", Window. <TableRow> into(CalendarWindows.days(1)))
 .apply("WriteToBQ", BigQueryIO.writeTableRows()
  .to(new SerializableFunction <ValueInSingleWindow<TableRow> , TableDestination> () {
   private static final long serialVersionUID = 8196602721734820219 L;

   @Override
   public TableDestination apply(ValueInSingleWindow <TableRow> value) {
    String dayString = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC)
     .print(((IntervalWindow) value.getWindow()).start());
    TableDestination td = new TableDestination("my-project:dataset.mytable_orders$" + dayString, "");
    return td;
   }
  }).withSchema(mySchema)
  .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
  .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)