You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Leon Papke <le...@student.hpi.de> on 2021/01/13 11:16:51 UTC
InfluxDB Source Sink Connector
Hi all,
we want to contribute a connector for InfluxDB v2.0. In the following, we
will briefly describe our planned architecture. We happily appreciate any
comments on our plans.
Background
InfluxDB is a time-series database created by InfluxData. InfluxData offers
an open source, enterprise, and cloud version. The open source version can
be deployed on a single node, whereas the enterprise version supports
horizontal scaling. The connector focuses on the open source version of
InfluxDB. In particular, InfluxDB underwent significant architecture changes
from v1 to v2, and we'll focus on the current v2 only.
Architecture
The following we propose a source and sink architecture using the new Flink
connector APIs.
Source
The Flink InfluxDB source connector will implement an InfluxDB write
endpoint that accepts line protocol-formatted data points for an InfluxDB
instance, i.e., existing tools like Telegraf will be compatible with this
source connector out of the box. In conclusion, we replicate the InfluxDB
write endpoint in our source by providing an HTTP REST API with one
endpoint, i.e., `POST /api/v2/write`.
Sink
The Flink InfluxDB sink connector builds upon the principles of InfluxDB.
Although InfluxDB does not support any transactions, it does not store
duplicate data points. InfluxDB identifies unique data points by their
measurement, tag set, and timestamp. Hence, to guarantee an exactly-once
semantic, we can simply write to InfluxDB.
Finally, we are highly interested in specific use case ideas. Yet, we are
thinking of real-time IoT sensor analytics but please reach out to us if you
have any in your mind.
Best regards,Ramin Gharib, Felix Seidel, and Leon Papke
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: InfluxDB Source Sink Connector
Posted by Leon Papke <le...@student.hpi.de>.
The current connector is based on the Flink's old connector API and InfluxDB
v1.X. InfluxDB's architecture change was quite an impact, hence, changes
were made on both sides. Though, we are aware of the current implementation
and investigate what we can reuse.
Best
Leon
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
Re: InfluxDB Source Sink Connector
Posted by Yun Tang <my...@live.com>.
I wonder what's relationship between current influxdb connector in bahir-flink [1]?
[1] https://github.com/apache/bahir-flink/tree/master/flink-connector-influxdb
Best
Yun Tang
________________________________
From: Leon Papke <le...@student.hpi.de>
Sent: Wednesday, January 13, 2021 19:16
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: InfluxDB Source Sink Connector
Hi all,
we want to contribute a connector for InfluxDB v2.0. In the following, we
will briefly describe our planned architecture. We happily appreciate any
comments on our plans.
Background
InfluxDB is a time-series database created by InfluxData. InfluxData offers
an open source, enterprise, and cloud version. The open source version can
be deployed on a single node, whereas the enterprise version supports
horizontal scaling. The connector focuses on the open source version of
InfluxDB. In particular, InfluxDB underwent significant architecture changes
from v1 to v2, and we'll focus on the current v2 only.
Architecture
The following we propose a source and sink architecture using the new Flink
connector APIs.
Source
The Flink InfluxDB source connector will implement an InfluxDB write
endpoint that accepts line protocol-formatted data points for an InfluxDB
instance, i.e., existing tools like Telegraf will be compatible with this
source connector out of the box. In conclusion, we replicate the InfluxDB
write endpoint in our source by providing an HTTP REST API with one
endpoint, i.e., `POST /api/v2/write`.
Sink
The Flink InfluxDB sink connector builds upon the principles of InfluxDB.
Although InfluxDB does not support any transactions, it does not store
duplicate data points. InfluxDB identifies unique data points by their
measurement, tag set, and timestamp. Hence, to guarantee an exactly-once
semantic, we can simply write to InfluxDB.
Finally, we are highly interested in specific use case ideas. Yet, we are
thinking of real-time IoT sensor analytics but please reach out to us if you
have any in your mind.
Best regards,Ramin Gharib, Felix Seidel, and Leon Papke
--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/