You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Clay Teeter <cl...@maalka.com> on 2021/02/16 15:42:04 UTC
How do i register a streaming table sink in 1.12?
Hey all. Hopefully this is an easy question. I'm porting my JDBC postgres
sink from 1.10 to 1.12
I'm using:
* StreamTableEnvironment
* JdbcUpsertTableSink
What I'm having difficulty with is how to register the sink with the
streaming table environment.
In 1.10:
tableEnv.registerTableSink(
> s"${interval}_maalka_jdbc_output_table",
> jdbcTableSink)
This method doesn't exist in 1.12, what is the equivalent?
Thanks!
Clay
Re: How do i register a streaming table sink in 1.12?
Posted by Till Rohrmann <tr...@apache.org>.
Great to hear that it is now working and thanks for letting the community
know :-)
On Wed, Feb 17, 2021 at 2:48 PM Clay Teeter <cl...@maalka.com> wrote:
> Yep, that was it! thanks! And to complete the thread, this is the working
> revision.
>
> package com.maalka.flink.sinks
>
> import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
> import com.typesafe.scalalogging.LazyLogging
> import org.apache.flink.api.common.functions.RuntimeContext
> import org.apache.flink.connector.jdbc.{JdbcConnectionOptions, JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
> import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction, JdbcBatchingOutputFormat}
> import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
> import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
> import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
> import org.apache.flink.streaming.api.functions.sink.SinkFunction
> import org.apache.flink.streaming.api.scala._
> import com.maalka.flink.typeInformation.Implicits._
>
> import java.sql.PreparedStatement
> import java.util.function.Function
>
> object TestSink extends LazyLogging {
> // START HERE
> def process(messageStream: DataStream[MaalkaDataRecord],
> signableUpdateStream: Option[DataStream[SignableUpdate]],
> streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
>
> insertAnalyticData("raw",
> "insert into analytic_table ... ",
> messageStream.map(_ => "A"))
> }
>
> // it is required that you explicitly create a new JDBCStatementBuilder
> val statementBuilder: JdbcStatementBuilder[String] =
> new JdbcStatementBuilder[String] {
> override def accept(ps: PreparedStatement, t: String): Unit = {
> ps.setString(1, t)
> }
> }
>
>
> private def insertAnalyticData(
> interval: String,
> insertStatement: String,
> messageStream: DataStream[String]): Unit = {
> val connectionString = s"jdbc:postgresql://localhost/db"
> val sink: SinkFunction[String] = JdbcSink.sink(
> insertStatement,
> statementBuilder,
> JdbcExecutionOptions.builder()
> .withBatchIntervalMs(1000)
> .withBatchSize(1000)
> .withMaxRetries(10)
> .build,
> JdbcOptions.builder()
> .setDBUrl(connectionString)
> .setTableName("analytic_table")
> .build
> )
>
> messageStream
> .addSink(sink)
> }
> }
>
>
>
>
> On Wed, Feb 17, 2021 at 2:24 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> I am not 100% sure but maybe (_, _) => {} captures a reference to object
>> TestSink which is not serializable. Maybe try to simply define a no
>> op JdbcStatementBuilder and pass such an instance to JdbcSink.sink().
>>
>> Cheers,
>> Till
>>
>> On Wed, Feb 17, 2021 at 2:06 PM Clay Teeter <cl...@maalka.com>
>> wrote:
>>
>>> Ok, this is about as simple as I can get.
>>>
>>> package com.maalka.flink.sinks
>>>
>>> import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
>>> import com.typesafe.scalalogging.LazyLogging
>>> import org.apache.flink.api.common.functions.RuntimeContext
>>> import org.apache.flink.connector.jdbc.{JdbcConnectionOptions, JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
>>> import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction, JdbcBatchingOutputFormat}
>>> import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
>>> import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
>>> import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
>>> import org.apache.flink.streaming.api.functions.sink.SinkFunction
>>> import org.apache.flink.streaming.api.scala._
>>> import com.maalka.flink.typeInformation.Implicits._
>>>
>>> import java.util.function.Function
>>>
>>> object TestSink extends LazyLogging {
>>> // START HERE
>>> def process(messageStream: DataStream[MaalkaDataRecord],
>>> signableUpdateStream: Option[DataStream[SignableUpdate]],
>>> streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
>>>
>>> insertAnalyticData("raw",
>>> "insert into analytic_table ... ",
>>> messageStream.map(_ => "A"))
>>> }
>>>
>>> private def insertAnalyticData(
>>> interval: String,
>>> insertStatement: String,
>>> messageStream: DataStream[String]): Unit = {
>>> val connectionString = s"jdbc:postgresql://localhost/db"
>>> val sink: SinkFunction[String] = JdbcSink.sink(
>>> insertStatement,
>>> (_, _) => {}, // I have a feeling that this is the lambda that can't serialize
>>> JdbcExecutionOptions.builder()
>>> .withBatchIntervalMs(1000)
>>> .withBatchSize(1000)
>>> .withMaxRetries(10)
>>> .build,
>>> JdbcOptions.builder()
>>> .setDBUrl(connectionString)
>>> .setTableName("analytic_table")
>>> .build
>>> )
>>>
>>> messageStream
>>> .addSink(sink)
>>> }
>>> }
>>>
>>>
>>> On Wed, Feb 17, 2021 at 9:34 AM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Hi Clay,
>>>>
>>>> could you maybe share the source code of
>>>> com.maalka.flink.sinks.MaalkaPostgresSink with us? It seems that this
>>>> sink uses a lambda which is not serializable. Maybe it holds a reference to
>>>> some non Serializable class as part of its closure.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Tue, Feb 16, 2021 at 8:58 PM Clay Teeter <cl...@maalka.com>
>>>> wrote:
>>>>
>>>>> Thanks Till, the tickets and links were immensely useful. With that i
>>>>> was able to make progress and even get things to compile. However, when i
>>>>> run things a serializable exception is thrown. (see below)
>>>>>
>>>>> .addSink(JdbcSink.sink[SignableTableSchema](
>>>>>> addIntervalToInsertStatement(insertStatement, interval),
>>>>>> (ps: PreparedStatement, rd: SignableTableSchema) => {
>>>>>> ps.setString(1, rd.data_processing_id)
>>>>>> ps.setTimestamp(2, rd.crc)
>>>>>> ps.setString(3, rd.command)
>>>>>> ps.setString(4, rd.result)
>>>>>> ps.setOptionalString(5, rd.message)
>>>>>> ps.setString(6, rd.arguments)
>>>>>> ps.setOptionalString(7, rd.validatorUUID)
>>>>>> },
>>>>>> getJdbcExecutionOptions,
>>>>>> getJdbcOptions(interval, insertStatement) // <-- This is line 376
>>>>>> ))
>>>>>>
>>>>>> Where i set the executionOptions to behave in a bachfull way.
>>>>>
>>>>> def getJdbcExecutionOptions: JdbcExecutionOptions = {
>>>>> JdbcExecutionOptions.builder()
>>>>> .withBatchIntervalMs(1000)
>>>>> .withBatchSize(1000)
>>>>> .withMaxRetries(10)
>>>>> .build
>>>>> }
>>>>>
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> [info] org.apache.flink.api.common.InvalidProgramException: The
>>>>>> implementation of the AbstractJdbcOutputFormat is not serializable. The
>>>>>> object probably contains or references non serializable fields.
>>>>>> [info] at
>>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
>>>>>> [info] at
>>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
>>>>>> [info] at
>>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>>>>>> [info] at
>>>>>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
>>>>>> [info] at
>>>>>> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
>>>>>> [info] at
>>>>>> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
>>>>>> [info] at
>>>>>> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
>>>>>> [info] at
>>>>>> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
>>>>>> [info] at
>>>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
>>>>>> [info] at
>>>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
>>>>>> [info] ...
>>>>>> [info] Cause: java.io.NotSerializableException: Non-serializable
>>>>>> lambda
>>>>>> [info] at
>>>>>> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
>>>>>> Source)
>>>>>> [info] at
>>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>> Method)
>>>>>> [info] at
>>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> [info] at
>>>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>>> [info] at
>>>>>> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
>>>>>> [info] at
>>>>>> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
>>>>>> [info] at
>>>>>> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
>>>>>> [info] at
>>>>>> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
>>>>>> [info] at
>>>>>> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Clay,
>>>>>>
>>>>>> I am not a Table API expert but let me try to answer your question:
>>>>>>
>>>>>> With FLINK-17748 [1] the community removed the registerTableSink in
>>>>>> favour of the connect API. The connect API has been deprecated [2] because
>>>>>> it was not well maintained. Now the recommended way for specifying sinks is
>>>>>> to use Flink's DDL [3]. Unfortunately, I couldn't find an easy example on
>>>>>> how to use the DDL. Maybe Timo or Jark can point you towards a good guide
>>>>>> on how to register your jdbc table sink.
>>>>>>
>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-17748
>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-18416
>>>>>> [3]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>>>>>>> postgres sink from 1.10 to 1.12
>>>>>>>
>>>>>>> I'm using:
>>>>>>> * StreamTableEnvironment
>>>>>>> * JdbcUpsertTableSink
>>>>>>>
>>>>>>> What I'm having difficulty with is how to register the sink with the
>>>>>>> streaming table environment.
>>>>>>>
>>>>>>> In 1.10:
>>>>>>>
>>>>>>> tableEnv.registerTableSink(
>>>>>>>> s"${interval}_maalka_jdbc_output_table",
>>>>>>>> jdbcTableSink)
>>>>>>>
>>>>>>>
>>>>>>> This method doesn't exist in 1.12, what is the equivalent?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Clay
>>>>>>>
>>>>>>>
Re: How do i register a streaming table sink in 1.12?
Posted by Clay Teeter <cl...@maalka.com>.
Yep, that was it! thanks! And to complete the thread, this is the working
revision.
package com.maalka.flink.sinks
import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
import com.typesafe.scalalogging.LazyLogging
import org.apache.flink.api.common.functions.RuntimeContext
import org.apache.flink.connector.jdbc.{JdbcConnectionOptions,
JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction,
JdbcBatchingOutputFormat}
import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
import org.apache.flink.streaming.api.functions.sink.SinkFunction
import org.apache.flink.streaming.api.scala._
import com.maalka.flink.typeInformation.Implicits._
import java.sql.PreparedStatement
import java.util.function.Function
object TestSink extends LazyLogging {
// START HERE
def process(messageStream: DataStream[MaalkaDataRecord],
signableUpdateStream: Option[DataStream[SignableUpdate]],
streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
insertAnalyticData("raw",
"insert into analytic_table ... ",
messageStream.map(_ => "A"))
}
// it is required that you explicitly create a new JDBCStatementBuilder
val statementBuilder: JdbcStatementBuilder[String] =
new JdbcStatementBuilder[String] {
override def accept(ps: PreparedStatement, t: String): Unit = {
ps.setString(1, t)
}
}
private def insertAnalyticData(
interval: String,
insertStatement: String,
messageStream: DataStream[String]): Unit = {
val connectionString = s"jdbc:postgresql://localhost/db"
val sink: SinkFunction[String] = JdbcSink.sink(
insertStatement,
statementBuilder,
JdbcExecutionOptions.builder()
.withBatchIntervalMs(1000)
.withBatchSize(1000)
.withMaxRetries(10)
.build,
JdbcOptions.builder()
.setDBUrl(connectionString)
.setTableName("analytic_table")
.build
)
messageStream
.addSink(sink)
}
}
On Wed, Feb 17, 2021 at 2:24 PM Till Rohrmann <tr...@apache.org> wrote:
> I am not 100% sure but maybe (_, _) => {} captures a reference to object
> TestSink which is not serializable. Maybe try to simply define a no
> op JdbcStatementBuilder and pass such an instance to JdbcSink.sink().
>
> Cheers,
> Till
>
> On Wed, Feb 17, 2021 at 2:06 PM Clay Teeter <cl...@maalka.com>
> wrote:
>
>> Ok, this is about as simple as I can get.
>>
>> package com.maalka.flink.sinks
>>
>> import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
>> import com.typesafe.scalalogging.LazyLogging
>> import org.apache.flink.api.common.functions.RuntimeContext
>> import org.apache.flink.connector.jdbc.{JdbcConnectionOptions, JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
>> import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction, JdbcBatchingOutputFormat}
>> import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
>> import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
>> import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
>> import org.apache.flink.streaming.api.functions.sink.SinkFunction
>> import org.apache.flink.streaming.api.scala._
>> import com.maalka.flink.typeInformation.Implicits._
>>
>> import java.util.function.Function
>>
>> object TestSink extends LazyLogging {
>> // START HERE
>> def process(messageStream: DataStream[MaalkaDataRecord],
>> signableUpdateStream: Option[DataStream[SignableUpdate]],
>> streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
>>
>> insertAnalyticData("raw",
>> "insert into analytic_table ... ",
>> messageStream.map(_ => "A"))
>> }
>>
>> private def insertAnalyticData(
>> interval: String,
>> insertStatement: String,
>> messageStream: DataStream[String]): Unit = {
>> val connectionString = s"jdbc:postgresql://localhost/db"
>> val sink: SinkFunction[String] = JdbcSink.sink(
>> insertStatement,
>> (_, _) => {}, // I have a feeling that this is the lambda that can't serialize
>> JdbcExecutionOptions.builder()
>> .withBatchIntervalMs(1000)
>> .withBatchSize(1000)
>> .withMaxRetries(10)
>> .build,
>> JdbcOptions.builder()
>> .setDBUrl(connectionString)
>> .setTableName("analytic_table")
>> .build
>> )
>>
>> messageStream
>> .addSink(sink)
>> }
>> }
>>
>>
>> On Wed, Feb 17, 2021 at 9:34 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Clay,
>>>
>>> could you maybe share the source code of
>>> com.maalka.flink.sinks.MaalkaPostgresSink with us? It seems that this
>>> sink uses a lambda which is not serializable. Maybe it holds a reference to
>>> some non Serializable class as part of its closure.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Feb 16, 2021 at 8:58 PM Clay Teeter <cl...@maalka.com>
>>> wrote:
>>>
>>>> Thanks Till, the tickets and links were immensely useful. With that i
>>>> was able to make progress and even get things to compile. However, when i
>>>> run things a serializable exception is thrown. (see below)
>>>>
>>>> .addSink(JdbcSink.sink[SignableTableSchema](
>>>>> addIntervalToInsertStatement(insertStatement, interval),
>>>>> (ps: PreparedStatement, rd: SignableTableSchema) => {
>>>>> ps.setString(1, rd.data_processing_id)
>>>>> ps.setTimestamp(2, rd.crc)
>>>>> ps.setString(3, rd.command)
>>>>> ps.setString(4, rd.result)
>>>>> ps.setOptionalString(5, rd.message)
>>>>> ps.setString(6, rd.arguments)
>>>>> ps.setOptionalString(7, rd.validatorUUID)
>>>>> },
>>>>> getJdbcExecutionOptions,
>>>>> getJdbcOptions(interval, insertStatement) // <-- This is line 376
>>>>> ))
>>>>>
>>>>> Where i set the executionOptions to behave in a bachfull way.
>>>>
>>>> def getJdbcExecutionOptions: JdbcExecutionOptions = {
>>>> JdbcExecutionOptions.builder()
>>>> .withBatchIntervalMs(1000)
>>>> .withBatchSize(1000)
>>>> .withMaxRetries(10)
>>>> .build
>>>> }
>>>>
>>>>
>>>> Any suggestions?
>>>>
>>>> [info] org.apache.flink.api.common.InvalidProgramException: The
>>>>> implementation of the AbstractJdbcOutputFormat is not serializable. The
>>>>> object probably contains or references non serializable fields.
>>>>> [info] at
>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
>>>>> [info] at
>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
>>>>> [info] at
>>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>>>>> [info] at
>>>>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
>>>>> [info] at
>>>>> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
>>>>> [info] at
>>>>> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
>>>>> [info] at
>>>>> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
>>>>> [info] at
>>>>> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
>>>>> [info] at
>>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
>>>>> [info] at
>>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
>>>>> [info] ...
>>>>> [info] Cause: java.io.NotSerializableException: Non-serializable
>>>>> lambda
>>>>> [info] at
>>>>> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
>>>>> Source)
>>>>> [info] at
>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> [info] at
>>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> [info] at
>>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> [info] at
>>>>> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
>>>>> [info] at
>>>>> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
>>>>> [info] at
>>>>> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
>>>>> [info] at
>>>>> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
>>>>> [info] at
>>>>> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>>>>>
>>>>
>>>>
>>>> On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Clay,
>>>>>
>>>>> I am not a Table API expert but let me try to answer your question:
>>>>>
>>>>> With FLINK-17748 [1] the community removed the registerTableSink in
>>>>> favour of the connect API. The connect API has been deprecated [2] because
>>>>> it was not well maintained. Now the recommended way for specifying sinks is
>>>>> to use Flink's DDL [3]. Unfortunately, I couldn't find an easy example on
>>>>> how to use the DDL. Maybe Timo or Jark can point you towards a good guide
>>>>> on how to register your jdbc table sink.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-17748
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-18416
>>>>> [3]
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
>>>>> wrote:
>>>>>
>>>>>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>>>>>> postgres sink from 1.10 to 1.12
>>>>>>
>>>>>> I'm using:
>>>>>> * StreamTableEnvironment
>>>>>> * JdbcUpsertTableSink
>>>>>>
>>>>>> What I'm having difficulty with is how to register the sink with the
>>>>>> streaming table environment.
>>>>>>
>>>>>> In 1.10:
>>>>>>
>>>>>> tableEnv.registerTableSink(
>>>>>>> s"${interval}_maalka_jdbc_output_table",
>>>>>>> jdbcTableSink)
>>>>>>
>>>>>>
>>>>>> This method doesn't exist in 1.12, what is the equivalent?
>>>>>>
>>>>>> Thanks!
>>>>>> Clay
>>>>>>
>>>>>>
Re: How do i register a streaming table sink in 1.12?
Posted by Till Rohrmann <tr...@apache.org>.
I am not 100% sure but maybe (_, _) => {} captures a reference to object
TestSink which is not serializable. Maybe try to simply define a no
op JdbcStatementBuilder and pass such an instance to JdbcSink.sink().
Cheers,
Till
On Wed, Feb 17, 2021 at 2:06 PM Clay Teeter <cl...@maalka.com> wrote:
> Ok, this is about as simple as I can get.
>
> package com.maalka.flink.sinks
>
> import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
> import com.typesafe.scalalogging.LazyLogging
> import org.apache.flink.api.common.functions.RuntimeContext
> import org.apache.flink.connector.jdbc.{JdbcConnectionOptions, JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
> import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction, JdbcBatchingOutputFormat}
> import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
> import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
> import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
> import org.apache.flink.streaming.api.functions.sink.SinkFunction
> import org.apache.flink.streaming.api.scala._
> import com.maalka.flink.typeInformation.Implicits._
>
> import java.util.function.Function
>
> object TestSink extends LazyLogging {
> // START HERE
> def process(messageStream: DataStream[MaalkaDataRecord],
> signableUpdateStream: Option[DataStream[SignableUpdate]],
> streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
>
> insertAnalyticData("raw",
> "insert into analytic_table ... ",
> messageStream.map(_ => "A"))
> }
>
> private def insertAnalyticData(
> interval: String,
> insertStatement: String,
> messageStream: DataStream[String]): Unit = {
> val connectionString = s"jdbc:postgresql://localhost/db"
> val sink: SinkFunction[String] = JdbcSink.sink(
> insertStatement,
> (_, _) => {}, // I have a feeling that this is the lambda that can't serialize
> JdbcExecutionOptions.builder()
> .withBatchIntervalMs(1000)
> .withBatchSize(1000)
> .withMaxRetries(10)
> .build,
> JdbcOptions.builder()
> .setDBUrl(connectionString)
> .setTableName("analytic_table")
> .build
> )
>
> messageStream
> .addSink(sink)
> }
> }
>
>
> On Wed, Feb 17, 2021 at 9:34 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Clay,
>>
>> could you maybe share the source code of
>> com.maalka.flink.sinks.MaalkaPostgresSink with us? It seems that this
>> sink uses a lambda which is not serializable. Maybe it holds a reference to
>> some non Serializable class as part of its closure.
>>
>> Cheers,
>> Till
>>
>> On Tue, Feb 16, 2021 at 8:58 PM Clay Teeter <cl...@maalka.com>
>> wrote:
>>
>>> Thanks Till, the tickets and links were immensely useful. With that i
>>> was able to make progress and even get things to compile. However, when i
>>> run things a serializable exception is thrown. (see below)
>>>
>>> .addSink(JdbcSink.sink[SignableTableSchema](
>>>> addIntervalToInsertStatement(insertStatement, interval),
>>>> (ps: PreparedStatement, rd: SignableTableSchema) => {
>>>> ps.setString(1, rd.data_processing_id)
>>>> ps.setTimestamp(2, rd.crc)
>>>> ps.setString(3, rd.command)
>>>> ps.setString(4, rd.result)
>>>> ps.setOptionalString(5, rd.message)
>>>> ps.setString(6, rd.arguments)
>>>> ps.setOptionalString(7, rd.validatorUUID)
>>>> },
>>>> getJdbcExecutionOptions,
>>>> getJdbcOptions(interval, insertStatement) // <-- This is line 376
>>>> ))
>>>>
>>>> Where i set the executionOptions to behave in a bachfull way.
>>>
>>> def getJdbcExecutionOptions: JdbcExecutionOptions = {
>>> JdbcExecutionOptions.builder()
>>> .withBatchIntervalMs(1000)
>>> .withBatchSize(1000)
>>> .withMaxRetries(10)
>>> .build
>>> }
>>>
>>>
>>> Any suggestions?
>>>
>>> [info] org.apache.flink.api.common.InvalidProgramException: The
>>>> implementation of the AbstractJdbcOutputFormat is not serializable. The
>>>> object probably contains or references non serializable fields.
>>>> [info] at
>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
>>>> [info] at
>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
>>>> [info] at
>>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>>>> [info] at
>>>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
>>>> [info] at
>>>> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
>>>> [info] at
>>>> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
>>>> [info] at
>>>> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
>>>> [info] at
>>>> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
>>>> [info] at
>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
>>>> [info] at
>>>> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
>>>> [info] ...
>>>> [info] Cause: java.io.NotSerializableException: Non-serializable
>>>> lambda
>>>> [info] at
>>>> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
>>>> Source)
>>>> [info] at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> [info] at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> [info] at
>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> [info] at
>>>> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
>>>> [info] at
>>>> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
>>>> [info] at
>>>> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
>>>> [info] at
>>>> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
>>>> [info] at
>>>> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>>>>
>>>
>>>
>>> On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Hi Clay,
>>>>
>>>> I am not a Table API expert but let me try to answer your question:
>>>>
>>>> With FLINK-17748 [1] the community removed the registerTableSink in
>>>> favour of the connect API. The connect API has been deprecated [2] because
>>>> it was not well maintained. Now the recommended way for specifying sinks is
>>>> to use Flink's DDL [3]. Unfortunately, I couldn't find an easy example on
>>>> how to use the DDL. Maybe Timo or Jark can point you towards a good guide
>>>> on how to register your jdbc table sink.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-17748
>>>> [2] https://issues.apache.org/jira/browse/FLINK-18416
>>>> [3]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
>>>> wrote:
>>>>
>>>>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>>>>> postgres sink from 1.10 to 1.12
>>>>>
>>>>> I'm using:
>>>>> * StreamTableEnvironment
>>>>> * JdbcUpsertTableSink
>>>>>
>>>>> What I'm having difficulty with is how to register the sink with the
>>>>> streaming table environment.
>>>>>
>>>>> In 1.10:
>>>>>
>>>>> tableEnv.registerTableSink(
>>>>>> s"${interval}_maalka_jdbc_output_table",
>>>>>> jdbcTableSink)
>>>>>
>>>>>
>>>>> This method doesn't exist in 1.12, what is the equivalent?
>>>>>
>>>>> Thanks!
>>>>> Clay
>>>>>
>>>>>
Re: How do i register a streaming table sink in 1.12?
Posted by Clay Teeter <cl...@maalka.com>.
Ok, this is about as simple as I can get.
package com.maalka.flink.sinks
import com.maalka.flink.models.{MaalkaDataRecord, SignableUpdate}
import com.typesafe.scalalogging.LazyLogging
import org.apache.flink.api.common.functions.RuntimeContext
import org.apache.flink.connector.jdbc.{JdbcConnectionOptions,
JdbcExecutionOptions, JdbcSink, JdbcStatementBuilder}
import org.apache.flink.connector.jdbc.internal.{GenericJdbcSinkFunction,
JdbcBatchingOutputFormat}
import org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider
import org.apache.flink.connector.jdbc.internal.executor.JdbcBatchStatementExecutor
import org.apache.flink.connector.jdbc.internal.options.JdbcOptions
import org.apache.flink.streaming.api.functions.sink.SinkFunction
import org.apache.flink.streaming.api.scala._
import com.maalka.flink.typeInformation.Implicits._
import java.util.function.Function
object TestSink extends LazyLogging {
// START HERE
def process(messageStream: DataStream[MaalkaDataRecord],
signableUpdateStream: Option[DataStream[SignableUpdate]],
streamExecutionEnvironment: StreamExecutionEnvironment): Unit = {
insertAnalyticData("raw",
"insert into analytic_table ... ",
messageStream.map(_ => "A"))
}
private def insertAnalyticData(
interval: String,
insertStatement: String,
messageStream: DataStream[String]): Unit = {
val connectionString = s"jdbc:postgresql://localhost/db"
val sink: SinkFunction[String] = JdbcSink.sink(
insertStatement,
(_, _) => {}, // I have a feeling that this is the lambda
that can't serialize
JdbcExecutionOptions.builder()
.withBatchIntervalMs(1000)
.withBatchSize(1000)
.withMaxRetries(10)
.build,
JdbcOptions.builder()
.setDBUrl(connectionString)
.setTableName("analytic_table")
.build
)
messageStream
.addSink(sink)
}
}
On Wed, Feb 17, 2021 at 9:34 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Clay,
>
> could you maybe share the source code of
> com.maalka.flink.sinks.MaalkaPostgresSink with us? It seems that this
> sink uses a lambda which is not serializable. Maybe it holds a reference to
> some non Serializable class as part of its closure.
>
> Cheers,
> Till
>
> On Tue, Feb 16, 2021 at 8:58 PM Clay Teeter <cl...@maalka.com>
> wrote:
>
>> Thanks Till, the tickets and links were immensely useful. With that i
>> was able to make progress and even get things to compile. However, when i
>> run things a serializable exception is thrown. (see below)
>>
>> .addSink(JdbcSink.sink[SignableTableSchema](
>>> addIntervalToInsertStatement(insertStatement, interval),
>>> (ps: PreparedStatement, rd: SignableTableSchema) => {
>>> ps.setString(1, rd.data_processing_id)
>>> ps.setTimestamp(2, rd.crc)
>>> ps.setString(3, rd.command)
>>> ps.setString(4, rd.result)
>>> ps.setOptionalString(5, rd.message)
>>> ps.setString(6, rd.arguments)
>>> ps.setOptionalString(7, rd.validatorUUID)
>>> },
>>> getJdbcExecutionOptions,
>>> getJdbcOptions(interval, insertStatement) // <-- This is line 376
>>> ))
>>>
>>> Where i set the executionOptions to behave in a bachfull way.
>>
>> def getJdbcExecutionOptions: JdbcExecutionOptions = {
>> JdbcExecutionOptions.builder()
>> .withBatchIntervalMs(1000)
>> .withBatchSize(1000)
>> .withMaxRetries(10)
>> .build
>> }
>>
>>
>> Any suggestions?
>>
>> [info] org.apache.flink.api.common.InvalidProgramException: The
>>> implementation of the AbstractJdbcOutputFormat is not serializable. The
>>> object probably contains or references non serializable fields.
>>> [info] at
>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
>>> [info] at
>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
>>> [info] at
>>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>>> [info] at
>>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
>>> [info] at
>>> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
>>> [info] at
>>> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
>>> [info] at
>>> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
>>> [info] at
>>> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
>>> [info] at
>>> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
>>> [info] at
>>> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
>>> [info] ...
>>> [info] Cause: java.io.NotSerializableException: Non-serializable lambda
>>> [info] at
>>> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
>>> Source)
>>> [info] at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> [info] at
>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> [info] at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> [info] at
>>> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
>>> [info] at
>>> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
>>> [info] at
>>> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
>>> [info] at
>>> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
>>> [info] at
>>> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>>>
>>
>>
>> On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Clay,
>>>
>>> I am not a Table API expert but let me try to answer your question:
>>>
>>> With FLINK-17748 [1] the community removed the registerTableSink in
>>> favour of the connect API. The connect API has been deprecated [2] because
>>> it was not well maintained. Now the recommended way for specifying sinks is
>>> to use Flink's DDL [3]. Unfortunately, I couldn't find an easy example on
>>> how to use the DDL. Maybe Timo or Jark can point you towards a good guide
>>> on how to register your jdbc table sink.
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-17748
>>> [2] https://issues.apache.org/jira/browse/FLINK-18416
>>> [3]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
>>> wrote:
>>>
>>>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>>>> postgres sink from 1.10 to 1.12
>>>>
>>>> I'm using:
>>>> * StreamTableEnvironment
>>>> * JdbcUpsertTableSink
>>>>
>>>> What I'm having difficulty with is how to register the sink with the
>>>> streaming table environment.
>>>>
>>>> In 1.10:
>>>>
>>>> tableEnv.registerTableSink(
>>>>> s"${interval}_maalka_jdbc_output_table",
>>>>> jdbcTableSink)
>>>>
>>>>
>>>> This method doesn't exist in 1.12, what is the equivalent?
>>>>
>>>> Thanks!
>>>> Clay
>>>>
>>>>
Re: How do i register a streaming table sink in 1.12?
Posted by Till Rohrmann <tr...@apache.org>.
Hi Clay,
could you maybe share the source code of
com.maalka.flink.sinks.MaalkaPostgresSink with us? It seems that this
sink uses a lambda which is not serializable. Maybe it holds a reference to
some non Serializable class as part of its closure.
Cheers,
Till
On Tue, Feb 16, 2021 at 8:58 PM Clay Teeter <cl...@maalka.com> wrote:
> Thanks Till, the tickets and links were immensely useful. With that i was
> able to make progress and even get things to compile. However, when i run
> things a serializable exception is thrown. (see below)
>
> .addSink(JdbcSink.sink[SignableTableSchema](
>> addIntervalToInsertStatement(insertStatement, interval),
>> (ps: PreparedStatement, rd: SignableTableSchema) => {
>> ps.setString(1, rd.data_processing_id)
>> ps.setTimestamp(2, rd.crc)
>> ps.setString(3, rd.command)
>> ps.setString(4, rd.result)
>> ps.setOptionalString(5, rd.message)
>> ps.setString(6, rd.arguments)
>> ps.setOptionalString(7, rd.validatorUUID)
>> },
>> getJdbcExecutionOptions,
>> getJdbcOptions(interval, insertStatement) // <-- This is line 376
>> ))
>>
>> Where i set the executionOptions to behave in a bachfull way.
>
> def getJdbcExecutionOptions: JdbcExecutionOptions = {
> JdbcExecutionOptions.builder()
> .withBatchIntervalMs(1000)
> .withBatchSize(1000)
> .withMaxRetries(10)
> .build
> }
>
>
> Any suggestions?
>
> [info] org.apache.flink.api.common.InvalidProgramException: The
>> implementation of the AbstractJdbcOutputFormat is not serializable. The
>> object probably contains or references non serializable fields.
>> [info] at
>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
>> [info] at
>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
>> [info] at
>> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>> [info] at
>> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
>> [info] at
>> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
>> [info] at
>> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
>> [info] at
>> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
>> [info] at
>> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
>> [info] at
>> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
>> [info] at
>> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
>> [info] ...
>> [info] Cause: java.io.NotSerializableException: Non-serializable lambda
>> [info] at
>> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
>> Source)
>> [info] at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> [info] at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> [info] at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> [info] at
>> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
>> [info] at
>> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
>> [info] at
>> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
>> [info] at
>> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
>> [info] at
>> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>>
>
>
> On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Clay,
>>
>> I am not a Table API expert but let me try to answer your question:
>>
>> With FLINK-17748 [1] the community removed the registerTableSink in
>> favour of the connect API. The connect API has been deprecated [2] because
>> it was not well maintained. Now the recommended way for specifying sinks is
>> to use Flink's DDL [3]. Unfortunately, I couldn't find an easy example on
>> how to use the DDL. Maybe Timo or Jark can point you towards a good guide
>> on how to register your jdbc table sink.
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-17748
>> [2] https://issues.apache.org/jira/browse/FLINK-18416
>> [3]
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>>
>> Cheers,
>> Till
>>
>> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
>> wrote:
>>
>>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>>> postgres sink from 1.10 to 1.12
>>>
>>> I'm using:
>>> * StreamTableEnvironment
>>> * JdbcUpsertTableSink
>>>
>>> What I'm having difficulty with is how to register the sink with the
>>> streaming table environment.
>>>
>>> In 1.10:
>>>
>>> tableEnv.registerTableSink(
>>>> s"${interval}_maalka_jdbc_output_table",
>>>> jdbcTableSink)
>>>
>>>
>>> This method doesn't exist in 1.12, what is the equivalent?
>>>
>>> Thanks!
>>> Clay
>>>
>>>
Re: How do i register a streaming table sink in 1.12?
Posted by Clay Teeter <cl...@maalka.com>.
Thanks Till, the tickets and links were immensely useful. With that i was
able to make progress and even get things to compile. However, when i run
things a serializable exception is thrown. (see below)
.addSink(JdbcSink.sink[SignableTableSchema](
> addIntervalToInsertStatement(insertStatement, interval),
> (ps: PreparedStatement, rd: SignableTableSchema) => {
> ps.setString(1, rd.data_processing_id)
> ps.setTimestamp(2, rd.crc)
> ps.setString(3, rd.command)
> ps.setString(4, rd.result)
> ps.setOptionalString(5, rd.message)
> ps.setString(6, rd.arguments)
> ps.setOptionalString(7, rd.validatorUUID)
> },
> getJdbcExecutionOptions,
> getJdbcOptions(interval, insertStatement) // <-- This is line 376
> ))
>
> Where i set the executionOptions to behave in a bachfull way.
def getJdbcExecutionOptions: JdbcExecutionOptions = {
JdbcExecutionOptions.builder()
.withBatchIntervalMs(1000)
.withBatchSize(1000)
.withMaxRetries(10)
.build
}
Any suggestions?
[info] org.apache.flink.api.common.InvalidProgramException: The
> implementation of the AbstractJdbcOutputFormat is not serializable. The
> object probably contains or references non serializable fields.
> [info] at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
> [info] at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
> [info] at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
> [info] at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000)
> [info] at
> org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203)
> [info] at
> org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243)
> [info] at
> org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1110)
> [info] at
> com.maalka.flink.sinks.MaalkaPostgresSink$.insertAnalyticData(MaalkaPostgresSink.scala:376)
> [info] at
> com.maalka.flink.sinks.MaalkaPostgresSink.process(MaalkaPostgresSink.scala:262)
> [info] at
> com.maalka.flink.sinks.MaalkaPostgresSink.process$(MaalkaPostgresSink.scala:250)
> [info] ...
> [info] Cause: java.io.NotSerializableException: Non-serializable lambda
> [info] at
> com.maalka.flink.sinks.MaalkaPostgresSink$$$Lambda$22459/0x0000000809678c40.writeObject(Unknown
> Source)
> [info] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> [info] at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [info] at
> java.base/java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
> [info] at
> java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
> [info] at
> java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
> [info] at
> java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
> [info] at
> java.base/java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1379)
>
On Tue, Feb 16, 2021 at 6:11 PM Till Rohrmann <tr...@apache.org> wrote:
> Hi Clay,
>
> I am not a Table API expert but let me try to answer your question:
>
> With FLINK-17748 [1] the community removed the registerTableSink in favour
> of the connect API. The connect API has been deprecated [2] because it was
> not well maintained. Now the recommended way for specifying sinks is to use
> Flink's DDL [3]. Unfortunately, I couldn't find an easy example on how to
> use the DDL. Maybe Timo or Jark can point you towards a good guide on how
> to register your jdbc table sink.
>
> [1] https://issues.apache.org/jira/browse/FLINK-17748
> [2] https://issues.apache.org/jira/browse/FLINK-18416
> [3]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
>
> Cheers,
> Till
>
> On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com>
> wrote:
>
>> Hey all. Hopefully this is an easy question. I'm porting my JDBC
>> postgres sink from 1.10 to 1.12
>>
>> I'm using:
>> * StreamTableEnvironment
>> * JdbcUpsertTableSink
>>
>> What I'm having difficulty with is how to register the sink with the
>> streaming table environment.
>>
>> In 1.10:
>>
>> tableEnv.registerTableSink(
>>> s"${interval}_maalka_jdbc_output_table",
>>> jdbcTableSink)
>>
>>
>> This method doesn't exist in 1.12, what is the equivalent?
>>
>> Thanks!
>> Clay
>>
>>
Re: How do i register a streaming table sink in 1.12?
Posted by Till Rohrmann <tr...@apache.org>.
Hi Clay,
I am not a Table API expert but let me try to answer your question:
With FLINK-17748 [1] the community removed the registerTableSink in favour
of the connect API. The connect API has been deprecated [2] because it was
not well maintained. Now the recommended way for specifying sinks is to use
Flink's DDL [3]. Unfortunately, I couldn't find an easy example on how to
use the DDL. Maybe Timo or Jark can point you towards a good guide on how
to register your jdbc table sink.
[1] https://issues.apache.org/jira/browse/FLINK-17748
[2] https://issues.apache.org/jira/browse/FLINK-18416
[3]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sourceSinks.html
Cheers,
Till
On Tue, Feb 16, 2021 at 4:42 PM Clay Teeter <cl...@maalka.com> wrote:
> Hey all. Hopefully this is an easy question. I'm porting my JDBC
> postgres sink from 1.10 to 1.12
>
> I'm using:
> * StreamTableEnvironment
> * JdbcUpsertTableSink
>
> What I'm having difficulty with is how to register the sink with the
> streaming table environment.
>
> In 1.10:
>
> tableEnv.registerTableSink(
>> s"${interval}_maalka_jdbc_output_table",
>> jdbcTableSink)
>
>
> This method doesn't exist in 1.12, what is the equivalent?
>
> Thanks!
> Clay
>
>