[jira] [Updated] (BEAM-2122) Writing to partitioned BigQuery tables from Dataflow is causing errors

["Davor Bonaci (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davor Bonaci updated BEAM-2122:
-------------------------------
    Fix Version/s: First stable release

> Writing to partitioned BigQuery tables from Dataflow is causing errors
> ----------------------------------------------------------------------
>
>                 Key: BEAM-2122
>                 URL: https://issues.apache.org/jira/browse/BEAM-2122
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>         Environment: Running with Beam 0.7.0-SNAPSHOT version 48 for beam-sdks-java-io-google-cloud-platform, 49 for beam-sdks-java-core and beam-runners-google-cloud-dataflow-java in Eclipse using Dataflow service.
>            Reporter: Matthias Baetens
>            Assignee: Reuven Lax
>             Fix For: First stable release
>
>
> Using the latest Beam SNAPSHOT which has a new BigQuery connector and trying to write to partitioned tables according to the docs (or this Stackoverflow question http://stackoverflow.com/questions/43505534/writing-different-values-to-different-bigquery-tables-in-apache-beam/43655461#43655461):
> 	static class PartitionedTableGeneration
> 			implements SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination> {
> 		@ProcessElement
> 		public TableDestination apply(ValueInSingleWindow<TableRow> value) {
> 			// String dayString =
> 			// DateTimeFormat.forPattern("yyyy_MM_dd").withZone(DateTimeZone.UTC)
> 			String dayString = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC)
> 					.print(((IntervalWindow) value.getWindow()).start());
> 			TableDestination td = new TableDestination(
> 					"projecet:dataset.table + '$' dayString, "");
> 			return td;
> 		}
> 	}
> causes the following issues when running (depending on the specification of the dayString):
> 1. "Invalid table ID \"partitioned_sample$20150905\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used.",
>  2. java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Failed to create load job with id prefix 
> ...
>     "errorResult" : {
>       "message" : "Invalid date partitioned table suffix: 2015_11_26",
>       "reason" : "invalid"
>     }
> Writing to sharded tables (without the '$'-sign) is working fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)