You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/01/22 13:02:28 UTC
[GitHub] [kafka] sknop opened a new pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
sknop opened a new pull request #9950:
URL: https://github.com/apache/kafka/pull/9950
Cast SMT transformation for bytes -> string.
Without this fix, the conversion becomes
ByteBuffer.toString(), which always gives one of these useless results:
"java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]"
"[B@5ec0a365" (for byte array)
With this change, the byte array is converted into a hex string of the
byte buffer content, for example
"FEDCBA9876543210"
Completed with test case and successfully tried out in a
real database conversion.
### Committer Checklist (excluded from commit message)
- [X ] Verify design and implementation
- [ X] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison merged pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
mimaison merged pull request #9950:
URL: https://github.com/apache/kafka/pull/9950
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r580882032
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,31 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ if (byteBuffer.hasArray()) {
+ return castByteArrayToString(byteBuffer.array());
+ }
+ else {
+ byte[] array = new byte[byteBuffer.remaining()];
+ byteBuffer.get(array);
+ return castByteArrayToString(array);
+ }
Review comment:
This sounds like an excellent plan and passes the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-783634110
Before this fix, the changes made in #4820 ([KAFKA-6684](https://issues.apache.org/jira/browse/KAFKA-6684)) resulted in `toString()` being called on `byte[]` and `ByteBuffer`. As highlighted in this PR and issue that `ByteBuffer.toString()` is not useful, but the `toString()` on `byte[]` still works. This PR seems to change that behavior, which would not be backward compatible.
The discussion on PR #4820 also talked about making this compatible with `Values.convertToString(...)`, which for `byte[]` and `ByteBuffer` results in a base 64 encoded string (with `ISO-8859-1` encoding). See the [Values code for details](https://github.com/apache/kafka/blob/95f51539c8d0b88bd7f285011d42e2d1117107de/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L673-L682).
~Because of this, it seems much more sensible to also use base64 here for `ByteBuffer` so it matches the existing behavior with `byte[]`.~
~I agree that it maybe doesn't suffice in all user situations, so if we also want to support other encodings we'd need other config changes that will require using the KIP mechanism to propose such enhancements.~
Actually, the existing code simply outputs the `toString()` result of a `byte[]` object, which is of the form `[B@22ef9844`, which is the `Object.toString()` implementation that includes the class alias (e.g., `[B` is a byte array) and the hex representation of the object's `hashCode()`.
However, given that none of the cast forms of `byte[]` or `ByteBuffer[]` are useful, then perhaps it's worth adding the previous discussion on #4820 mentioning that:
> As @ewencp suggested, for consistency perhaps we can use the same string formats as SimpleHeaderConverter: https://github.com/apache/kafka/blob/trunk/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java,
As mentioned above, the `Values.convertToString(...)` uses a base 64 encoding. And this aligns with @kkonstantine's suggestion:
> Why are we selecting hex here and not base64 for example? Hex of course is less efficient.
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r580884849
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
If base64 is the better choice we shall go with that. Would we need to adjust the JIRA ticket to reflect this?
I just ran a little test in the JShell:
jshell> byte[] byteArray = new byte[] {(byte) 0xFE, (byte) 0xDC, (byte) 0xBA, (byte) 0x98, 0x76, 0x54, 0x32, 0x10};
byteArray ==> byte[8] { -2, -36, -70, -104, 118, 84, 50, 16 }
jshell> Base64.getEncoder().encode(byteArray)
$2 ==> byte[12] { 47, 116, 121, 54, 109, 72, 90, 85, 77, 104, 65, 61 }
jshell> Base64.getEncoder().encodeToString(byteArray)
$3 ==> "/ty6mHZUMhA="
jshell> Base64.getEncoder().withoutPadding().encodeToString(byteArray)
$4 ==> "/ty6mHZUMhA"
Would you suggest using padding or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-766939861
> @sknop hello , It seems like there are some style error .
> You can use `./gradlew checkstyleMain checkstyleTest` to cehck .
Thank you @g1geordie for your feedback. I have adjusted the code and it now passes the style check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] g1geordie commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
g1geordie commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-766586309
@sknop hello , It seems like there are some style error .
You can use `./gradlew checkstyleMain checkstyleTest` to cehck .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786911483
The documentation is auto-generated from the code, via [TransformationDoc](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/tools/TransformationDoc.java).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r581206410
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
Connect's `Values` code just uses `Base64.getEncoder().encodeToString((byte[]) value)` and does not disable padding. Seems like we should follow the same pattern here. Thoughts, @kkonstantine ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r583514614
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -39,12 +40,8 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.util.EnumSet;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
+import java.nio.ByteBuffer;
+import java.util.*;
Review comment:
IntelliJ did this automatically for me - I have changed the code accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-787084819
Thank you @rhauch. Just to make sure I understand this correctly - the types mentioned in the cast string are the _target_ casts, not the source tasks. The current code does not allow casting to a byte array, only casting _from_ a byte array.
As I understand it, therefore no documentation changes are required (except maybe for release notes)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-789161208
Yes of course, totally agree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kkonstantine commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
kkonstantine commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r578823263
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
Review comment:
formally we should almost always check that a `ByteBuffer` is indeed backed by an array with `hasArray` or this part my throw exceptions now: https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html#array--
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
Also, I'm a bit curious. Why are we selecting hex here and not base64 for example?
Hex of course is less efficient.
https://issues.apache.org/jira/browse/KAFKA-12170 does not mention anything about the rational, so I thought I'd ask.
@rhauch wdyt about the representation here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786911483
The documentation is auto-generated from the transforms code, via [TransformationDoc](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/tools/TransformationDoc.java). Since this is an existing SMT, there are no changes that that `TransformationDoc` class, but we still have to change the `Cast` class [here](https://github.com/apache/kafka/blob/958f90e710f0de164dce1dda5f45d75a8b8fb8d4/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L57-L61) and [here](https://github.com/apache/kafka/blob/958f90e710f0de164dce1dda5f45d75a8b8fb8d4/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L83-L85) to mention that byte arrays can be cast to strings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786537716
What about documentation?
Where is the documentation stored and what is the process to change it? It might be useful to point out that the Cast SMT will create base64 strings for byte arrays.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r582985853
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -39,12 +40,8 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.util.EnumSet;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
+import java.nio.ByteBuffer;
+import java.util.*;
Review comment:
Wildcard imports should be caught by checkstyle, and should fail the build. In any case, please replace with non-wildcard imports.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-790111206
Test failures are not related:
```
JDK 8 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testOneWayReplicationWithAutoOffsetSync()
JDK 11 / org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testMultipleWorkersRejoining
```
Merging to trunk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r579839255
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
The reason I suggested Hex representation was that in the use case I encountered - database PK field encoded as BINARY retrieved via CDC Source Connector - the representation in the SQL CLI was Hex, so choosing the same format made it easy to compare source and target.
I would suggest that base64 encoding would validate its own dedicated transformer, with an ability to choose padding, for example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r579838511
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
Review comment:
That is a good point, but what should happen in that case?
I assume we start with hasArray() and only process the content if an array is available, as I understand it? ByteBuffer.remaining() would contain the number of bytes available ... I'll go and write some test cases to understand this better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-766939861
> @sknop hello , It seems like there are some style error .
> You can use `./gradlew checkstyleMain checkstyleTest` to cehck .
Thank you for your feedback. I have adjusted the code and it now passes the style check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
mimaison commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r578550844
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -360,15 +361,30 @@ else if (value instanceof String)
throw new DataException("Unexpected type in Cast transformation: " + value.getClass());
}
+
private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value.getClass() == byte[].class) {
Review comment:
Let's use `value instanceof byte[]` so it's like the other cases
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -360,15 +361,30 @@ else if (value instanceof String)
throw new DataException("Unexpected type in Cast transformation: " + value.getClass());
}
+
Review comment:
We don't need this extra blank line
##########
File path: connect/transforms/src/test/java/org/apache/kafka/connect/transforms/CastTest.java
##########
@@ -425,7 +426,21 @@ public void castLogicalToString() {
@Test
public void castFieldsWithSchema() {
Date day = new Date(MILLIS_PER_DAY);
- xformValue.configure(Collections.singletonMap(Cast.SPEC_CONFIG, "int8:int16,int16:int32,int32:int64,int64:boolean,float32:float64,float64:boolean,boolean:int8,string:int32,bigdecimal:string,date:string,optional:int32"));
+ ByteBuffer byteBuffer = ByteBuffer.allocate(8);
+ byteBuffer.put((byte) 0xFE);
+ byteBuffer.put((byte) 0xDC);
+ byteBuffer.put((byte) 0xBA);
+ byteBuffer.put((byte) 0x98);
+ byteBuffer.put((byte) 0x76);
+ byteBuffer.put((byte) 0x54);
+ byteBuffer.put((byte) 0x32);
+ byteBuffer.put((byte) 0x10);
+ byteBuffer.flip();
+
+ byte[] byteArray = Arrays.copyOf(byteBuffer.array(), byteBuffer.array().length);
Review comment:
What about doing something like:
```java
byte[] byteArray = new byte[] {(byte) 0xFE, (byte) 0xDC, (byte) 0xBA, (byte) 0x98, 0x76, 0x54, 0x32, 0x10};
ByteBuffer byteBuffer = ByteBuffer.wrap(Arrays.copyOf(byteArray, byteArray.length));
```
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -360,15 +361,30 @@ else if (value instanceof String)
throw new DataException("Unexpected type in Cast transformation: " + value.getClass());
}
+
private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
Review comment:
We can inline this like you've done for the byte[] case below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r580881343
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,31 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ if (byteBuffer.hasArray()) {
+ return castByteArrayToString(byteBuffer.array());
+ }
+ else {
+ byte[] array = new byte[byteBuffer.remaining()];
+ byteBuffer.get(array);
+ return castByteArrayToString(array);
+ }
Review comment:
That sounds like an excellent plan, I did not know this class existed. Changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-783634110
Before this fix, the changes made in #4820 ([KAFKA-6684](https://issues.apache.org/jira/browse/KAFKA-6684)) resulted in `toString()` being called on `byte[]` and `ByteBuffer`. As highlighted in this PR and issue that `ByteBuffer.toString()` is not useful, but the `toString()` on `byte[]` still works. This PR seems to change that behavior, which would not be backward compatible.
The discussion on PR #4820 also talked about making this compatible with `Values.convertToString(...)`, which for `byte[]` and `ByteBuffer` results in a base 64 encoded string (with `ISO-8859-1` encoding). See the [Values code for details](https://github.com/apache/kafka/blob/95f51539c8d0b88bd7f285011d42e2d1117107de/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L673-L682).
Because of this, it seems much more sensible to also use base64 here for `ByteBuffer` so it matches the existing behavior with `byte[]`.
I agree that it maybe doesn't suffice in all user situations, so if we also want to support other encodings we'd need other config changes that will require using the KIP mechanism to propose such enhancements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786911483
The documentation is auto-generated from the transforms code, via [TransformationDoc](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/tools/TransformationDoc.java). Since this is an existing SMT, there are no changes required for documentation (other than changing the overview and config docs in `Cast.java`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-789100759
If we don't clarify the documentation, then I think users will be very confused. Can you take a stab at improving/expanding the documentation a bit to clarify the input and output types?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-766939861
> @sknop hello , It seems like there are some style error .
> You can use `./gradlew checkstyleMain checkstyleTest` to cehck .
Thank you for your feedback. I have adjusted the code and it now passes the style check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786911483
The documentation is auto-generated from the transforms code, via [TransformationDoc](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/tools/TransformationDoc.java).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-783634110
Before this fix, the changes made in #4820 ([KAFKA-6684](https://issues.apache.org/jira/browse/KAFKA-6684)) resulted in `toString()` being called on `byte[]` and `ByteBuffer`. As highlighted in this PR and issue that `ByteBuffer.toString()` is not useful, but the `toString()` on `byte[]` still works. This PR seems to change that behavior, which would not be backward compatible.
The discussion on PR #4820 also talked about making this compatible with `Values.convertToString(...)`, which for `byte[]` and `ByteBuffer` results in a base 64 encoded string (with `ISO-8859-1` encoding). See the [Values code for details](https://github.com/apache/kafka/blob/95f51539c8d0b88bd7f285011d42e2d1117107de/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L673-L682).
~Because of this, it seems much more sensible to also use base64 here for `ByteBuffer` so it matches the existing behavior with `byte[]`.~
Actually, the existing code simply outputs the `toString()` result of a `byte[]` object, which is of the form `[B@22ef9844`, which is the `Object.toString()` implementation that includes the class alias (e.g., `[B` is a byte array) and the hex representation of the object's `hashCode()`.
However, given that none of the cast forms of `byte[]` or `ByteBuffer[]` are useful, then perhaps it's worth adding the previous discussion on #4820 mentioning that:
> As @ewencp suggested, for consistency perhaps we can use the same string formats as SimpleHeaderConverter: https://github.com/apache/kafka/blob/trunk/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java,
As mentioned above, the `Values.convertToString(...)` uses a base 64 encoding. And this aligns with @kkonstantine's suggestion:
> Why are we selecting hex here and not base64 for example? Hex of course is less efficient.
I agree that it maybe doesn't suffice in all user situations, so if we also want to support other encodings we'd need other config changes that will require using the KIP mechanism to propose such enhancements.
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch edited a comment on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch edited a comment on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-786911483
The documentation is auto-generated from the transforms code, via [TransformationDoc](https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/tools/TransformationDoc.java). Since this is an existing SMT, there are no changes required for documentation (other than changing the overview and config docs in `Cast.java`. Those changes should be made [here](https://github.com/apache/kafka/blob/958f90e710f0de164dce1dda5f45d75a8b8fb8d4/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L57-L61) and [here](https://github.com/apache/kafka/blob/958f90e710f0de164dce1dda5f45d75a8b8fb8d4/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java#L83-L85).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rhauch commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
rhauch commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r580584221
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
As noted in my previous comment, I agree with @kkonstantine that base64 would be preferable. Doing that would align better with the existing `Values` class used in Connect's header converter mechanism.
If we want to add support for multiple encodings, we would need to have a KIP since it would likely mean changing the SMT configuration.
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,31 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ if (byteBuffer.hasArray()) {
+ return castByteArrayToString(byteBuffer.array());
+ }
+ else {
+ byte[] array = new byte[byteBuffer.remaining()];
+ byteBuffer.get(array);
+ return castByteArrayToString(array);
+ }
Review comment:
The use of `ByteBuffer.get(...)` here does not account for the fact that it may not be positioned at the beginning. Kafka has two `Utils.readBytes(...)` methods that we should probably use. This code would then simplify to:
```suggestion
byte[] rawBytes = Utils.readBytes(byteBuffer);
return castByteArrayToString(rawBytes);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] g1geordie commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
g1geordie commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-766586309
@sknop hello , It seems like there are some style error .
You can use `./gradlew checkstyleMain checkstyleTest` to cehck .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kkonstantine commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
kkonstantine commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r581622982
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
+ } else if (value instanceof byte[]) {
+ return castByteArrayToString((byte[]) value);
} else {
return value.toString();
}
}
+ private static String castByteArrayToString(byte[] array) {
+ StringBuilder sbuf = new StringBuilder();
+ for (byte b : array) {
+ sbuf.append(String.format("%02X", b));
Review comment:
Yes, I agree. `encodeToString` is the method I had in mind. Used in `Values` and JsonConverter for a conversion of bytes to string. It's also more efficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on a change in pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on a change in pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#discussion_r580112242
##########
File path: connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java
##########
@@ -364,11 +365,24 @@ private static String castToString(Object value) {
if (value instanceof java.util.Date) {
java.util.Date dateValue = (java.util.Date) value;
return Values.dateFormatFor(dateValue).format(dateValue);
+ } else if (value instanceof ByteBuffer) {
+ ByteBuffer byteBuffer = (ByteBuffer) value;
+ return castByteArrayToString(byteBuffer.array());
Review comment:
I added code that checks for the existence of the array and otherwise, I extract the content through ByteBuffer.get(byte[] array).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] sknop commented on pull request #9950: KAFKA-12170: Fix for Connect Cast SMT to correctly transform a Byte array into a string
Posted by GitBox <gi...@apache.org>.
sknop commented on pull request #9950:
URL: https://github.com/apache/kafka/pull/9950#issuecomment-781618683
Hi Mickael
Thank you so much for your code review. I have adjusted the code according to your suggestions, they all make sense to me.
Cheers
Sven Erik
seknop@gmail.com
On 18 Feb 2021, 16:40 +0000, apache/kafka <re...@reply.github.com>, wrote:
>
> @mimaison
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org