You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Kaymak, Tobias" <to...@ricardo.ch> on 2020/07/08 14:40:19 UTC
Recommended way to generate a TableRow from Json without using
TableRowJsonCoder context OUTER?
As a workaround I am currently using the following code to generate a
TableRow object from a Java Protobuf class - as I am facing a problem with
Beam schemas (
https://www.mail-archive.com/user@beam.apache.org/msg05799.html).
It relies on the TableRowJsonCoder:
String json = JsonFormat.printer().omittingInsignificantWhitespace()
.preservingProtoFieldNames().print(article.toBuilder());
InputStream inputStream = new
ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8));
TableRow tableRow = TableRowJsonCoder.of().decode(inputStream,
Coder.Context.OUTER);
However, the usage of Coder.Context is deprecated - I've tried to simply
use the decode(), but that defaults to Context.NESTED.
What is the correct way of doing this?
Best,
Tobi
Re: Recommended way to generate a TableRow from Json without using
TableRowJsonCoder context OUTER?
Posted by "Kaymak, Tobias" <to...@ricardo.ch>.
Thank you Lars and thank you Luke!
On Wed, Jul 8, 2020 at 9:33 PM Luke Cwik <lc...@google.com> wrote:
> The deprecated method is not going to be removed anytime soon so I
> wouldn't worry about it being removed.
>
> If you really want to use non-deprecated methods, then the
> TableRowJsonCoder uses the StringUtf8Coder to parse strings so it is
> looking for a nested encoding using the StringUtf8Coder encoding. So
> something like this:
> ByteArrayOutputStream baos = new ...
> StringUtf8Coder.of().encode(jsonString, baos);
> TableRow row = TableRowJsonCoder.of().decode(new
> ByteArrayInputStream(baos.toByteArray()));
>
> But why use a coder at all? TableRowJsonCoder is a thin wrapper around
> using Jackson's ObjectMapper to perform the conversion. So you could do
> something like:
> ObjectMapper mapper = new
> ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
> TableRow row = mapper.readValue(strValue, TableRow.class);
>
>
> On Wed, Jul 8, 2020 at 7:57 AM Lars Almgren Schwartz <
> lars.almgren@tink.com> wrote:
>
>> Hey,
>>
>> Don't know if it's the official way but we have written our own proto to
>> BigQuery converter which works pretty well.
>>
>> public static TableRow convertEventToTableRow(TableRow tableRow, Message event) {
>> Map<Descriptors.FieldDescriptor, Object> fields = event.getAllFields();
>> for (Descriptors.FieldDescriptor field : fields.keySet()) {
>> tableRow = mapToBigQueryField(tableRow, field, fields.get(field));
>> }
>>
>> return tableRow;
>> }
>>
>> private static TableRow mapToBigQueryField(
>> TableRow tableRow, Descriptors.FieldDescriptor field, Object value) {
>> Descriptors.FieldDescriptor.JavaType fieldType = field.getJavaType();
>> switch (fieldType) {
>> case INT:
>> case LONG:
>> case FLOAT:
>> case DOUBLE:
>> case BOOLEAN:
>> return tableRow.set(field.getName(), value);
>> case BYTE_STRING:
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value,
>> x ->
>> Base64.getEncoder()
>> .encodeToString(
>> ((ByteString) x).toByteArray())));
>> } else {
>> return tableRow.set(
>> field.getName(),
>> Base64.getEncoder().encodeToString(((ByteString) value).toByteArray()));
>> }
>> case ENUM:
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(), processRepeatedField(value, x -> x.toString()));
>> } else {
>> return tableRow.set(field.getName(), value.toString());
>> }
>> case STRING:
>> if (isUUIDField(field.getName())) {
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value, x -> UUIDUtil.getBase64FromUUID((String) x)));
>> } else {
>> return tableRow.set(
>> field.getName(), UUIDUtil.getBase64FromUUID((String) value));
>> }
>> } else {
>> return tableRow.set(field.getName(), value);
>> }
>> case MESSAGE:
>> switch (field.getMessageType().getFullName()) {
>> // Map well known message types that we have a specific mapping for.
>> case "google.protobuf.Timestamp":
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value,
>> x ->
>> com.google.cloud.Timestamp.fromProto(
>> (Timestamp) x)
>> .toString()));
>> } else {
>> return tableRow.set(
>> field.getName(),
>> com.google.cloud.Timestamp.fromProto((Timestamp) value)
>> .toString());
>> }
>> case "xxx.xxx.ExactNumber":
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value, x -> NumberUtils.toString((ExactNumber) x)));
>> } else {
>> return tableRow.set(
>> field.getName(), NumberUtils.toString((ExactNumber) value));
>> }
>> case "google.protobuf.UInt64Value":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.UInt64Value.getDefaultInstance())) {
>> value = 0;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((UInt64Value) value).getValue());
>> }
>> case "google.protobuf.Int32Value":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.Int32Value.getDefaultInstance())) {
>> value = 0;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((Int32Value) value).getValue());
>> }
>> case "google.protobuf.DoubleValue":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.DoubleValue.getDefaultInstance())) {
>> value = 0;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((DoubleValue) value).getValue());
>> }
>> case "google.protobuf.FloatValue":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.FloatValue.getDefaultInstance())) {
>> value = 0;
>>
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((FloatValue) value).getValue());
>> }
>> case "google.protobuf.Int64Value":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.Int64Value.getDefaultInstance())) {
>> value = 0;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((Int64Value) value).getValue());
>> }
>> case "google.protobuf.UInt32Value":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.UInt32Value.getDefaultInstance())) {
>> value = 0;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((UInt32Value) value).getValue());
>> }
>> case "google.protobuf.BoolValue":
>> if (field.hasDefaultValue()) break;
>> else if (value.equals(com.google.protobuf.BoolValue.getDefaultInstance())) {
>> value = false;
>> return tableRow.set(field.getName(), value);
>> } else {
>> return tableRow.set(field.getName(), ((BoolValue) value).getValue());
>> }
>> case "google.protobuf.StringValue":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.StringValue.getDefaultInstance())) {
>> value = "";
>> return tableRow.set(field.getName(), value);
>> } else if (isUUIDField(field.getName())) {
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value,
>> x ->
>> UUIDUtil.getBase64FromUUID(
>> ((StringValue) x).getValue())));
>> } else {
>> return tableRow.set(
>> field.getName(),
>> UUIDUtil.getBase64FromUUID(
>> ((StringValue) value).getValue()));
>> }
>> } else if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(value, x -> ((StringValue) x).getValue()));
>> } else {
>> return tableRow.set(field.getName(), ((StringValue) value).getValue());
>> }
>> case "google.protobuf.BytesValue":
>> if (field.hasDefaultValue()) {
>> break;
>> } else if (value.equals(
>> com.google.protobuf.BytesValue.getDefaultInstance())) {
>> value = ByteString.EMPTY;
>> return tableRow.set(field.getName(), value);
>> } else if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value,
>> x ->
>> Base64.getEncoder()
>> .encodeToString(
>> ((BytesValue) x)
>> .getValue()
>> .toByteArray())));
>> } else {
>> return tableRow.set(
>> field.getName(),
>> Base64.getEncoder()
>> .encodeToString(
>> ((BytesValue) value).getValue().toByteArray()));
>> }
>> default:
>> if (field.isRepeated()) {
>> return tableRow.set(
>> field.getName(),
>> processRepeatedField(
>> value,
>> x ->
>> convertEventToTableRow(
>> new TableRow(), (Message) x)));
>> } else {
>> return tableRow.set(
>> field.getName(),
>> convertEventToTableRow(new TableRow(), (Message) value));
>> }
>> }
>> default:
>> throw new IllegalArgumentException(
>> field.getFullName() + " has an unsupported type " + field.getType());
>> }
>> }
>>
>>
>> On Wed, Jul 8, 2020 at 4:40 PM Kaymak, Tobias <to...@ricardo.ch>
>> wrote:
>>
>>> As a workaround I am currently using the following code to generate a
>>> TableRow object from a Java Protobuf class - as I am facing a problem with
>>> Beam schemas (
>>> https://www.mail-archive.com/user@beam.apache.org/msg05799.html).
>>>
>>> It relies on the TableRowJsonCoder:
>>>
>>> String json =
>>> JsonFormat.printer().omittingInsignificantWhitespace()
>>> .preservingProtoFieldNames().print(article.toBuilder());
>>> InputStream inputStream = new
>>> ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8));
>>>
>>> TableRow tableRow = TableRowJsonCoder.of().decode(inputStream,
>>> Coder.Context.OUTER);
>>>
>>> However, the usage of Coder.Context is deprecated - I've tried to
>>> simply use the decode(), but that defaults to Context.NESTED.
>>>
>>> What is the correct way of doing this?
>>>
>>> Best,
>>> Tobi
>>>
>>
Re: Recommended way to generate a TableRow from Json without using
TableRowJsonCoder context OUTER?
Posted by Luke Cwik <lc...@google.com>.
The deprecated method is not going to be removed anytime soon so I wouldn't
worry about it being removed.
If you really want to use non-deprecated methods, then the
TableRowJsonCoder uses the StringUtf8Coder to parse strings so it is
looking for a nested encoding using the StringUtf8Coder encoding. So
something like this:
ByteArrayOutputStream baos = new ...
StringUtf8Coder.of().encode(jsonString, baos);
TableRow row = TableRowJsonCoder.of().decode(new
ByteArrayInputStream(baos.toByteArray()));
But why use a coder at all? TableRowJsonCoder is a thin wrapper around
using Jackson's ObjectMapper to perform the conversion. So you could do
something like:
ObjectMapper mapper = new
ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
TableRow row = mapper.readValue(strValue, TableRow.class);
On Wed, Jul 8, 2020 at 7:57 AM Lars Almgren Schwartz <la...@tink.com>
wrote:
> Hey,
>
> Don't know if it's the official way but we have written our own proto to
> BigQuery converter which works pretty well.
>
> public static TableRow convertEventToTableRow(TableRow tableRow, Message event) {
> Map<Descriptors.FieldDescriptor, Object> fields = event.getAllFields();
> for (Descriptors.FieldDescriptor field : fields.keySet()) {
> tableRow = mapToBigQueryField(tableRow, field, fields.get(field));
> }
>
> return tableRow;
> }
>
> private static TableRow mapToBigQueryField(
> TableRow tableRow, Descriptors.FieldDescriptor field, Object value) {
> Descriptors.FieldDescriptor.JavaType fieldType = field.getJavaType();
> switch (fieldType) {
> case INT:
> case LONG:
> case FLOAT:
> case DOUBLE:
> case BOOLEAN:
> return tableRow.set(field.getName(), value);
> case BYTE_STRING:
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value,
> x ->
> Base64.getEncoder()
> .encodeToString(
> ((ByteString) x).toByteArray())));
> } else {
> return tableRow.set(
> field.getName(),
> Base64.getEncoder().encodeToString(((ByteString) value).toByteArray()));
> }
> case ENUM:
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(), processRepeatedField(value, x -> x.toString()));
> } else {
> return tableRow.set(field.getName(), value.toString());
> }
> case STRING:
> if (isUUIDField(field.getName())) {
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value, x -> UUIDUtil.getBase64FromUUID((String) x)));
> } else {
> return tableRow.set(
> field.getName(), UUIDUtil.getBase64FromUUID((String) value));
> }
> } else {
> return tableRow.set(field.getName(), value);
> }
> case MESSAGE:
> switch (field.getMessageType().getFullName()) {
> // Map well known message types that we have a specific mapping for.
> case "google.protobuf.Timestamp":
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value,
> x ->
> com.google.cloud.Timestamp.fromProto(
> (Timestamp) x)
> .toString()));
> } else {
> return tableRow.set(
> field.getName(),
> com.google.cloud.Timestamp.fromProto((Timestamp) value)
> .toString());
> }
> case "xxx.xxx.ExactNumber":
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value, x -> NumberUtils.toString((ExactNumber) x)));
> } else {
> return tableRow.set(
> field.getName(), NumberUtils.toString((ExactNumber) value));
> }
> case "google.protobuf.UInt64Value":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.UInt64Value.getDefaultInstance())) {
> value = 0;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((UInt64Value) value).getValue());
> }
> case "google.protobuf.Int32Value":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.Int32Value.getDefaultInstance())) {
> value = 0;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((Int32Value) value).getValue());
> }
> case "google.protobuf.DoubleValue":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.DoubleValue.getDefaultInstance())) {
> value = 0;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((DoubleValue) value).getValue());
> }
> case "google.protobuf.FloatValue":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.FloatValue.getDefaultInstance())) {
> value = 0;
>
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((FloatValue) value).getValue());
> }
> case "google.protobuf.Int64Value":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.Int64Value.getDefaultInstance())) {
> value = 0;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((Int64Value) value).getValue());
> }
> case "google.protobuf.UInt32Value":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.UInt32Value.getDefaultInstance())) {
> value = 0;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((UInt32Value) value).getValue());
> }
> case "google.protobuf.BoolValue":
> if (field.hasDefaultValue()) break;
> else if (value.equals(com.google.protobuf.BoolValue.getDefaultInstance())) {
> value = false;
> return tableRow.set(field.getName(), value);
> } else {
> return tableRow.set(field.getName(), ((BoolValue) value).getValue());
> }
> case "google.protobuf.StringValue":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.StringValue.getDefaultInstance())) {
> value = "";
> return tableRow.set(field.getName(), value);
> } else if (isUUIDField(field.getName())) {
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value,
> x ->
> UUIDUtil.getBase64FromUUID(
> ((StringValue) x).getValue())));
> } else {
> return tableRow.set(
> field.getName(),
> UUIDUtil.getBase64FromUUID(
> ((StringValue) value).getValue()));
> }
> } else if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(value, x -> ((StringValue) x).getValue()));
> } else {
> return tableRow.set(field.getName(), ((StringValue) value).getValue());
> }
> case "google.protobuf.BytesValue":
> if (field.hasDefaultValue()) {
> break;
> } else if (value.equals(
> com.google.protobuf.BytesValue.getDefaultInstance())) {
> value = ByteString.EMPTY;
> return tableRow.set(field.getName(), value);
> } else if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value,
> x ->
> Base64.getEncoder()
> .encodeToString(
> ((BytesValue) x)
> .getValue()
> .toByteArray())));
> } else {
> return tableRow.set(
> field.getName(),
> Base64.getEncoder()
> .encodeToString(
> ((BytesValue) value).getValue().toByteArray()));
> }
> default:
> if (field.isRepeated()) {
> return tableRow.set(
> field.getName(),
> processRepeatedField(
> value,
> x ->
> convertEventToTableRow(
> new TableRow(), (Message) x)));
> } else {
> return tableRow.set(
> field.getName(),
> convertEventToTableRow(new TableRow(), (Message) value));
> }
> }
> default:
> throw new IllegalArgumentException(
> field.getFullName() + " has an unsupported type " + field.getType());
> }
> }
>
>
> On Wed, Jul 8, 2020 at 4:40 PM Kaymak, Tobias <to...@ricardo.ch>
> wrote:
>
>> As a workaround I am currently using the following code to generate a
>> TableRow object from a Java Protobuf class - as I am facing a problem with
>> Beam schemas (
>> https://www.mail-archive.com/user@beam.apache.org/msg05799.html).
>>
>> It relies on the TableRowJsonCoder:
>>
>> String json = JsonFormat.printer().omittingInsignificantWhitespace()
>> .preservingProtoFieldNames().print(article.toBuilder());
>> InputStream inputStream = new
>> ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8));
>>
>> TableRow tableRow = TableRowJsonCoder.of().decode(inputStream,
>> Coder.Context.OUTER);
>>
>> However, the usage of Coder.Context is deprecated - I've tried to simply
>> use the decode(), but that defaults to Context.NESTED.
>>
>> What is the correct way of doing this?
>>
>> Best,
>> Tobi
>>
>
Re: Recommended way to generate a TableRow from Json without using
TableRowJsonCoder context OUTER?
Posted by Lars Almgren Schwartz <la...@tink.com>.
Hey,
Don't know if it's the official way but we have written our own proto to
BigQuery converter which works pretty well.
public static TableRow convertEventToTableRow(TableRow tableRow,
Message event) {
Map<Descriptors.FieldDescriptor, Object> fields = event.getAllFields();
for (Descriptors.FieldDescriptor field : fields.keySet()) {
tableRow = mapToBigQueryField(tableRow, field, fields.get(field));
}
return tableRow;
}
private static TableRow mapToBigQueryField(
TableRow tableRow, Descriptors.FieldDescriptor field, Object value) {
Descriptors.FieldDescriptor.JavaType fieldType = field.getJavaType();
switch (fieldType) {
case INT:
case LONG:
case FLOAT:
case DOUBLE:
case BOOLEAN:
return tableRow.set(field.getName(), value);
case BYTE_STRING:
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value,
x ->
Base64.getEncoder()
.encodeToString(
((ByteString)
x).toByteArray())));
} else {
return tableRow.set(
field.getName(),
Base64.getEncoder().encodeToString(((ByteString)
value).toByteArray()));
}
case ENUM:
if (field.isRepeated()) {
return tableRow.set(
field.getName(), processRepeatedField(value, x
-> x.toString()));
} else {
return tableRow.set(field.getName(), value.toString());
}
case STRING:
if (isUUIDField(field.getName())) {
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value, x ->
UUIDUtil.getBase64FromUUID((String) x)));
} else {
return tableRow.set(
field.getName(),
UUIDUtil.getBase64FromUUID((String) value));
}
} else {
return tableRow.set(field.getName(), value);
}
case MESSAGE:
switch (field.getMessageType().getFullName()) {
// Map well known message types that we have a
specific mapping for.
case "google.protobuf.Timestamp":
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value,
x ->
com.google.cloud.Timestamp.fromProto(
(Timestamp) x)
.toString()));
} else {
return tableRow.set(
field.getName(),
com.google.cloud.Timestamp.fromProto((Timestamp) value)
.toString());
}
case "xxx.xxx.ExactNumber":
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value, x ->
NumberUtils.toString((ExactNumber) x)));
} else {
return tableRow.set(
field.getName(),
NumberUtils.toString((ExactNumber) value));
}
case "google.protobuf.UInt64Value":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.UInt64Value.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((UInt64Value) value).getValue());
}
case "google.protobuf.Int32Value":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.Int32Value.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((Int32Value) value).getValue());
}
case "google.protobuf.DoubleValue":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.DoubleValue.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((DoubleValue) value).getValue());
}
case "google.protobuf.FloatValue":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.FloatValue.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((FloatValue) value).getValue());
}
case "google.protobuf.Int64Value":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.Int64Value.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((Int64Value) value).getValue());
}
case "google.protobuf.UInt32Value":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.UInt32Value.getDefaultInstance())) {
value = 0;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((UInt32Value) value).getValue());
}
case "google.protobuf.BoolValue":
if (field.hasDefaultValue()) break;
else if
(value.equals(com.google.protobuf.BoolValue.getDefaultInstance())) {
value = false;
return tableRow.set(field.getName(), value);
} else {
return tableRow.set(field.getName(),
((BoolValue) value).getValue());
}
case "google.protobuf.StringValue":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.StringValue.getDefaultInstance())) {
value = "";
return tableRow.set(field.getName(), value);
} else if (isUUIDField(field.getName())) {
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value,
x ->
UUIDUtil.getBase64FromUUID(
((StringValue) x).getValue())));
} else {
return tableRow.set(
field.getName(),
UUIDUtil.getBase64FromUUID(
((StringValue) value).getValue()));
}
} else if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(value, x ->
((StringValue) x).getValue()));
} else {
return tableRow.set(field.getName(),
((StringValue) value).getValue());
}
case "google.protobuf.BytesValue":
if (field.hasDefaultValue()) {
break;
} else if (value.equals(
com.google.protobuf.BytesValue.getDefaultInstance())) {
value = ByteString.EMPTY;
return tableRow.set(field.getName(), value);
} else if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value,
x ->
Base64.getEncoder()
.encodeToString(
((BytesValue) x)
.getValue()
.toByteArray())));
} else {
return tableRow.set(
field.getName(),
Base64.getEncoder()
.encodeToString(
((BytesValue)
value).getValue().toByteArray()));
}
default:
if (field.isRepeated()) {
return tableRow.set(
field.getName(),
processRepeatedField(
value,
x ->
convertEventToTableRow(
new
TableRow(), (Message) x)));
} else {
return tableRow.set(
field.getName(),
convertEventToTableRow(new TableRow(),
(Message) value));
}
}
default:
throw new IllegalArgumentException(
field.getFullName() + " has an unsupported type "
+ field.getType());
}
}
On Wed, Jul 8, 2020 at 4:40 PM Kaymak, Tobias <to...@ricardo.ch>
wrote:
> As a workaround I am currently using the following code to generate a
> TableRow object from a Java Protobuf class - as I am facing a problem with
> Beam schemas (
> https://www.mail-archive.com/user@beam.apache.org/msg05799.html).
>
> It relies on the TableRowJsonCoder:
>
> String json = JsonFormat.printer().omittingInsignificantWhitespace()
> .preservingProtoFieldNames().print(article.toBuilder());
> InputStream inputStream = new
> ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8));
>
> TableRow tableRow = TableRowJsonCoder.of().decode(inputStream,
> Coder.Context.OUTER);
>
> However, the usage of Coder.Context is deprecated - I've tried to simply
> use the decode(), but that defaults to Context.NESTED.
>
> What is the correct way of doing this?
>
> Best,
> Tobi
>