You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Magalhães <sp...@gmail.com> on 2020/02/26 11:56:08 UTC
Scala string interpolation weird behaviour with Flink Streaming Java
tests dependency.
I'm testing a custom sink that uses TwoPhaseCommit with the test harness
provided by flink-streaming-java.
"org.apache.flink" %% "flink-streaming-java" % flinkVersion % "test"
classifier "tests"
Using this, in some tests that I use scala string interpolation, the string
output have a strange behaviour, like it changes the place where values
goes.
Example:
val account_id = "account0"
val partitionDate = "202002"
val fileName = "2020-02-26_11-09-46.parquet"
s"account_id=$accountId/partition_date=$partitionDate/$fileName"
Should be:
account_id=account0/partition_date=202002/2020-02-26_11-09-46.parquet
Actual result:
account_id=account0/partition_date=2020-02-26_11-09-46.parquet/202002
The variables values after the string interpolation do change values.
Concat behaviour is not affected:
"account_id=".concat(accountId).concat("/partition_date=").concat(partitionDate).concat("/").concat(fileName)
If I remove the flink-streaming-java dependency is works as expected.
Any thoughts why is behaving this way ?
Re: Scala string interpolation weird behaviour with Flink Streaming
Java tests dependency.
Posted by Piotr Nowojski <pi...@ververica.com>.
Good to hear that it’s working. I would doubt that this was a Flink issue, but if it comes back, let us know.
Piotrek
> On 28 Feb 2020, at 16:48, David Magalhães <sp...@gmail.com> wrote:
>
> Hi Piotr, the typo was on writing the example here, not on the code it self.
>
> Regarding to the mix of Scala versions, I'm using 2.12 in every place. My Java version is 1.8.0_221.
>
> Currently it is working, but not sure what happened here.
>
> Thanks!
>
> On Fri, Feb 28, 2020 at 10:50 AM Piotr Nowojski <piotr@ververica.com <ma...@ververica.com>> wrote:
> Also, don’t you have a typo in your pattern? In your pattern you are using `$accountId`, while the variable is `account_id`? (Maybe I don’t understand it as I don’t know Scala very well).
>
> Piotrek
>
>> On 28 Feb 2020, at 11:45, Piotr Nowojski <pnowojski@apache.org <ma...@apache.org>> wrote:
>>
>> Hey,
>>
>> What Java versions are you using?
>>
>> Also, could you check, if you are not mixing Scala versions somewhere? There are two different Flink binaries for Scala 2.11 and Scala 2.12. I guess if you mix them, of if you use incorrect Scala runtime not matching the supported version of the binaries that you have downloaded, bad things could happen.
>>
>> Piotrek
>>
>>> On 26 Feb 2020, at 12:56, David Magalhães <speeddragon@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> I'm testing a custom sink that uses TwoPhaseCommit with the test harness provided by flink-streaming-java.
>>>
>>> "org.apache.flink" %% "flink-streaming-java" % flinkVersion % "test" classifier "tests"
>>>
>>> Using this, in some tests that I use scala string interpolation, the string output have a strange behaviour, like it changes the place where values goes.
>>>
>>> Example:
>>>
>>> val account_id = "account0"
>>> val partitionDate = "202002"
>>> val fileName = "2020-02-26_11-09-46.parquet"
>>>
>>> s"account_id=$accountId/partition_date=$partitionDate/$fileName"
>>>
>>> Should be: account_id=account0/partition_date=202002/2020-02-26_11-09-46.parquet
>>> Actual result: account_id=account0/partition_date=2020-02-26_11-09-46.parquet/202002
>>>
>>> The variables values after the string interpolation do change values.
>>>
>>> Concat behaviour is not affected:
>>>
>>> "account_id=".concat(accountId).concat("/partition_date=").concat(partitionDate).concat("/").concat(fileName)
>>>
>>> If I remove the flink-streaming-java dependency is works as expected.
>>>
>>> Any thoughts why is behaving this way ?
>>
>
Re: Scala string interpolation weird behaviour with Flink Streaming
Java tests dependency.
Posted by David Magalhães <sp...@gmail.com>.
Hi Piotr, the typo was on writing the example here, not on the code it self.
Regarding to the mix of Scala versions, I'm using 2.12 in every place. My
Java version is 1.8.0_221.
Currently it is working, but not sure what happened here.
Thanks!
On Fri, Feb 28, 2020 at 10:50 AM Piotr Nowojski <pi...@ververica.com> wrote:
> Also, don’t you have a typo in your pattern? In your pattern you are using
> `$accountId`, while the variable is `account_id`? (Maybe I don’t understand
> it as I don’t know Scala very well).
>
> Piotrek
>
> On 28 Feb 2020, at 11:45, Piotr Nowojski <pn...@apache.org> wrote:
>
> Hey,
>
> What Java versions are you using?
>
> Also, could you check, if you are not mixing Scala versions somewhere?
> There are two different Flink binaries for Scala 2.11 and Scala 2.12. I
> guess if you mix them, of if you use incorrect Scala runtime not matching
> the supported version of the binaries that you have downloaded, bad things
> could happen.
>
> Piotrek
>
> On 26 Feb 2020, at 12:56, David Magalhães <sp...@gmail.com> wrote:
>
> I'm testing a custom sink that uses TwoPhaseCommit with the test harness
> provided by flink-streaming-java.
>
> "org.apache.flink" %% "flink-streaming-java" % flinkVersion % "test"
> classifier "tests"
>
> Using this, in some tests that I use scala string interpolation, the
> string output have a strange behaviour, like it changes the place where
> values goes.
>
> Example:
>
> val account_id = "account0"
> val partitionDate = "202002"
> val fileName = "2020-02-26_11-09-46.parquet"
>
> s"account_id=$accountId/partition_date=$partitionDate/$fileName"
>
> Should be:
> account_id=account0/partition_date=202002/2020-02-26_11-09-46.parquet
> Actual result:
> account_id=account0/partition_date=2020-02-26_11-09-46.parquet/202002
>
> The variables values after the string interpolation do change values.
>
> Concat behaviour is not affected:
>
>
> "account_id=".concat(accountId).concat("/partition_date=").concat(partitionDate).concat("/").concat(fileName)
>
> If I remove the flink-streaming-java dependency is works as expected.
>
> Any thoughts why is behaving this way ?
>
>
>
>
Re: Scala string interpolation weird behaviour with Flink Streaming
Java tests dependency.
Posted by Piotr Nowojski <pi...@ververica.com>.
Also, don’t you have a typo in your pattern? In your pattern you are using `$accountId`, while the variable is `account_id`? (Maybe I don’t understand it as I don’t know Scala very well).
Piotrek
> On 28 Feb 2020, at 11:45, Piotr Nowojski <pn...@apache.org> wrote:
>
> Hey,
>
> What Java versions are you using?
>
> Also, could you check, if you are not mixing Scala versions somewhere? There are two different Flink binaries for Scala 2.11 and Scala 2.12. I guess if you mix them, of if you use incorrect Scala runtime not matching the supported version of the binaries that you have downloaded, bad things could happen.
>
> Piotrek
>
>> On 26 Feb 2020, at 12:56, David Magalhães <speeddragon@gmail.com <ma...@gmail.com>> wrote:
>>
>> I'm testing a custom sink that uses TwoPhaseCommit with the test harness provided by flink-streaming-java.
>>
>> "org.apache.flink" %% "flink-streaming-java" % flinkVersion % "test" classifier "tests"
>>
>> Using this, in some tests that I use scala string interpolation, the string output have a strange behaviour, like it changes the place where values goes.
>>
>> Example:
>>
>> val account_id = "account0"
>> val partitionDate = "202002"
>> val fileName = "2020-02-26_11-09-46.parquet"
>>
>> s"account_id=$accountId/partition_date=$partitionDate/$fileName"
>>
>> Should be: account_id=account0/partition_date=202002/2020-02-26_11-09-46.parquet
>> Actual result: account_id=account0/partition_date=2020-02-26_11-09-46.parquet/202002
>>
>> The variables values after the string interpolation do change values.
>>
>> Concat behaviour is not affected:
>>
>> "account_id=".concat(accountId).concat("/partition_date=").concat(partitionDate).concat("/").concat(fileName)
>>
>> If I remove the flink-streaming-java dependency is works as expected.
>>
>> Any thoughts why is behaving this way ?
>
Re: Scala string interpolation weird behaviour with Flink Streaming
Java tests dependency.
Posted by Piotr Nowojski <pi...@ververica.com>.
Hey,
What Java versions are you using?
Also, could you check, if you are not mixing Scala versions somewhere? There are two different Flink binaries for Scala 2.11 and Scala 2.12. I guess if you mix them, of if you use incorrect Scala runtime not matching the supported version of the binaries that you have downloaded, bad things could happen.
Piotrek
> On 26 Feb 2020, at 12:56, David Magalhães <sp...@gmail.com> wrote:
>
> I'm testing a custom sink that uses TwoPhaseCommit with the test harness provided by flink-streaming-java.
>
> "org.apache.flink" %% "flink-streaming-java" % flinkVersion % "test" classifier "tests"
>
> Using this, in some tests that I use scala string interpolation, the string output have a strange behaviour, like it changes the place where values goes.
>
> Example:
>
> val account_id = "account0"
> val partitionDate = "202002"
> val fileName = "2020-02-26_11-09-46.parquet"
>
> s"account_id=$accountId/partition_date=$partitionDate/$fileName"
>
> Should be: account_id=account0/partition_date=202002/2020-02-26_11-09-46.parquet
> Actual result: account_id=account0/partition_date=2020-02-26_11-09-46.parquet/202002
>
> The variables values after the string interpolation do change values.
>
> Concat behaviour is not affected:
>
> "account_id=".concat(accountId).concat("/partition_date=").concat(partitionDate).concat("/").concat(fileName)
>
> If I remove the flink-streaming-java dependency is works as expected.
>
> Any thoughts why is behaving this way ?