You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Dhruv Garg <dg...@gmail.com> on 2021/04/08 05:36:01 UTC
IoTDB write performance
Hello all,
In the past month I have been using the JDBC client of IoTDB to write data
from CSV into IoTDB and also query on the data. Looking at the CSV code in
ImportCsv.java
<https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java>,
it seems that csv itself is again parsed into an IoTDB-friendly structure
and then ingested. I would like to avoid the csv-parsing time and directly
provide the data as needed to IoTDB. This should improve the ingestion
performance.
What I am talking about is similar to InfluxDB where we can also parse from
CSV to InfluxDB's native line protocol and then ingest the data into DB.
However, for better performance, InfluxDB also provides write APIs
<https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java>
to take records in line protocol as input and directly ingest those.
I have three questions:
1. I see that IoTDB is getting newer write APIs like InsertTablets and
it seems that it is designed to be faster than insertRecords. Approximately
how much performance improvement have you seen with InsertTablets?
2. Is there a way to create a data file such that it is easy to use
InsertTablets with it? This is to know if I can create an IoTDB-friendly
and IoTDB-specific input file and then directly use InsertTablets to ingest
the data.
3. As a preliminary check, I am also trying out IoTDB on Raspberry Pi 4B
devices. However, the ingestion time with CSV on the Raspberry Pi is taking
10 times of what it is on the desktop (amd64). This ratio should have
ideally been closer to 4X, based on other applications that I have
benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB earlier?
I would be awaiting your response. Thanks!
Regards,
dgargcs
Re: IoTDB write performance
Posted by Jialin Qiao <qi...@apache.org>.
Hi Dhruv,
You are welcome:)
There is a comparison with other TSDBs [1], which is generated from [2].
Welcome to add more test cases or improve this page~
[1] http://iotdb.apache.org/UserGuide/Master/Comparison/TSDB-Comparison.html
[2]
https://github.com/apache/iotdb/blob/master/docs/UserGuide/Comparison/TSDB-Comparison.md
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Dhruv Garg <dg...@gmail.com> 于2021年4月11日周日 下午2:46写道：
> Hi Jialin,
>
> Thanks a lot for your inputs. Will test v0.12 out then and also start using
> insertTablets().
>
> I had seen the original IoTDB paper
> <https://www.vldb.org/pvldb/vol13/p2901-wang.pdf> - on which you are a
> co-author :) - had shown IoTDB working on the Raspberry Pi. I have been
> testing the IoTDB data ingestion/ querying on desktop and Raspberry Pi, in
> comparison with InfluxDB v2.0. Do let me know if there are any plans of
> benchmarking / comparative study that your team intends to do with IoTDB,
> InfluxDB or similar TSDBs. I could contribute to that.
>
> Regards,
> dgargcs
>
> On Sun, 11 Apr 2021 at 11:55, Jialin Qiao <qi...@apache.org> wrote:
>
> > Hi,
> >
> > 1. *Can I use the API on v0.11.2 (I am currently using this) or do I
> > need v0.11.3?*
> >
> > You can use 0.11.2, we do not change the RPC API in a minor(bug-fix)
> > version.
> >
> > 2. *Is there any limit to the number of rows that can
> > be inserted at once using insertTablet()? Or, is there an optimal number
> of
> > rows per insertTablet() to get best performance?*
> >
> > It depends on the columns of the Tablet. The default rpc size limit is
> > 64MB, so rows * columns * 8Byte < 64MB.
> > I usually set rows to 1000 when I have 1000 columns.
> >
> > 3. *Since insertTablet() also takes rows for one
> > device, what exactly is the difference between insertTablet() and
> > insertRecordsOfOneDevice()? Are they for different use-cases? Which
> > performs better?*
> >
> > insertRecordsOfOneDevice is actually an improved version of
> > insertRecords(): all records belong to one device, so we only acquire one
> > writelock in the writing process.
> >
> > InsertTablet requires that each row has all measurements(using primitive
> > data types to store data), insertRecordsOfOneDevice allows each row has
> > different measurements(using Object to store data).
> >
> > Performance: InsertTablet > insertRecordsOfOneDevice > insertRecords >
> > insertRecord
> >
> > InsertTablet is always the fastest :)
> >
> > 4. *I suppose IoTDB does not have benchmark numbers for such devices? *
> >
> > We haven't tested these cases.
> >
> > 5. *How soon can we expect it to come out? Or can I test it out even now
> > since I eventually
> > plan to start working with it? Is it stable enough?*
> >
> > The release is already got more than 3 binding votes, it will come out in
> > one or two days.
> > You can get it from
> > https://dist.apache.org/repos/dist/dev/iotdb/0.12.0/rc1
> >
> > We fixed nearly all important known bugs in 0.12.0. The single-node
> version
> > is stable enough.
> > The data migration in the cluster version is not supported, and the test
> of
> > the cluster version is not very much.
> > Welcome to test it and give feedback :)
> >
> > 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
> > improve DB data ingestion/query performance?*
> >
> > It removes some redundant fields in the previous version (decreasing disk
> > occupation) and optimizes the performance of the raw data query.
> >
> > Thanks,
> > —————————————————
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> >
> > Dhruv Garg <dg...@gmail.com> 于2021年4月11日周日 下午1:38写道：
> >
> > > Hello Jialin,
> > >
> > > Thanks for your response.
> > >
> > > 1. Alright I will definitely move from insertRecords() to
> insertTablet()
> > > then. *Can I use the API on v0.11.2 (I am currently using this) or do I
> > > need v0.11.3?*
> > >
> > > 2. The info you provided on insertTablet() is very helpful. I could
> start
> > > with that. I think specifying the data types at the top of the file
> will
> > > also help reduce the data-type inference time for the api and possibly
> > > reduce ingestion time. *Is there any limit to the number of rows that
> can
> > > be inserted at once using insertTablet()? Or, is there an optimal
> number
> > of
> > > rows per insertTablet() to get best performance?*
> > >
> > > 3. Also, I see that there is a new API in v0.12 called
> > > insertRecordsOfOneDevice(). *Since insertTablet() also takes rows for
> one
> > > device, what exactly is the difference between insertTablet() and
> > > insertRecordsOfOneDevice()? Are they for different use-cases? Which
> > > performs better?*
> > >
> > > 4. So the Raspberry Pi are low-end devices. They run on lower RAM
> (2GB),
> > > have ARM processors and use SD cards for persistent storage. *I suppose
> > > IoTDB does not have benchmark numbers for such devices? *
> > >
> > > A couple of additional questions:
> > > 5. I see that the IoTDB team is nearing release of v0.12. *How soon can
> > we
> > > expect it to come out? Or can I test it out even now since I eventually
> > > plan to start working with it? Is it stable enough?*
> > >
> > > 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
> > > improve DB data ingestion/query performance?*
> > >
> > > Thanks in advance!
> > >
> > > On Thu, 8 Apr 2021 at 17:26, Jialin Qiao <qi...@apache.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > 1. InsertTablets could reach more than 3 times faster than
> > insertRecords.
> > > >
> > > > 2. Yes, Tablet is actually a small table with some columns and rows.
> > > > It has a time column and many value columns. In each row, all columns
> > > must
> > > > have a value.
> > > >
> > > > An example:
> > > >
> > > > time, root.sg.d1.s1, root.sg.d1.s2
> > > > 1, 1, 2.2
> > > > 2, 1, 2.2
> > > > 3, 1, 2.2
> > > >
> > > > Tablets do not allow to have null values.
> > > >
> > > > However, the tablet uses an array of primitive types to store data
> in a
> > > > columnar format, e.g., long[], int[], double[].
> > > > If you store data in a text file, the data type indicator is needed,
> > like
> > > > this:
> > > >
> > > > time, root.sg.d1.s1, root.sg.d1.s2
> > > > long, int, double
> > > > 1, 1, 2.2
> > > > 2, 1, 2.2
> > > > 3, 1, 2.2
> > > >
> > > > Then you could generate a tablet from the data and then use the
> > > > insertTablet.
> > > >
> > > > 3. You could check the memory allocated in Raspberry pi. Is it the
> same
> > > as
> > > > on desktop? This may impact the write throughput.
> > > >
> > > > Thanks,
> > > > —————————————————
> > > > Jialin Qiao
> > > > School of Software, Tsinghua University
> > > >
> > > > 乔嘉林
> > > > 清华大学 软件学院
> > > >
> > > >
> > > > Dhruv Garg <dg...@gmail.com> 于2021年4月8日周四 下午1:36写道：
> > > >
> > > > > Hello all,
> > > > >
> > > > > In the past month I have been using the JDBC client of IoTDB to
> write
> > > > data
> > > > > from CSV into IoTDB and also query on the data. Looking at the CSV
> > code
> > > > in
> > > > > ImportCsv.java
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
> > > > > >,
> > > > > it seems that csv itself is again parsed into an IoTDB-friendly
> > > structure
> > > > > and then ingested. I would like to avoid the csv-parsing time and
> > > > directly
> > > > > provide the data as needed to IoTDB. This should improve the
> > ingestion
> > > > > performance.
> > > > >
> > > > > What I am talking about is similar to InfluxDB where we can also
> > parse
> > > > from
> > > > > CSV to InfluxDB's native line protocol and then ingest the data
> into
> > > DB.
> > > > > However, for better performance, InfluxDB also provides write APIs
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java
> > > > > >
> > > > > to take records in line protocol as input and directly ingest
> those.
> > > > >
> > > > > I have three questions:
> > > > >
> > > > > 1. I see that IoTDB is getting newer write APIs like
> InsertTablets
> > > and
> > > > > it seems that it is designed to be faster than insertRecords.
> > > > > Approximately
> > > > > how much performance improvement have you seen with
> InsertTablets?
> > > > > 2. Is there a way to create a data file such that it is easy to
> > use
> > > > > InsertTablets with it? This is to know if I can create an
> > > > IoTDB-friendly
> > > > > and IoTDB-specific input file and then directly use
> InsertTablets
> > to
> > > > > ingest
> > > > > the data.
> > > > > 3. As a preliminary check, I am also trying out IoTDB on
> Raspberry
> > > Pi
> > > > 4B
> > > > > devices. However, the ingestion time with CSV on the Raspberry
> Pi
> > is
> > > > > taking
> > > > > 10 times of what it is on the desktop (amd64). This ratio should
> > > have
> > > > > ideally been closer to 4X, based on other applications that I
> have
> > > > > benchmarked. Have you all run any Raspberry Pi benchmarks for
> > IoTDB
> > > > > earlier?
> > > > >
> > > > > I would be awaiting your response. Thanks!
> > > > >
> > > > > Regards,
> > > > > dgargcs
> > > > >
> > > >
> > >
> >
>
Re: IoTDB write performance
Posted by Dhruv Garg <dg...@gmail.com>.
Hi Jialin,
Thanks a lot for your inputs. Will test v0.12 out then and also start using
insertTablets().
I had seen the original IoTDB paper
<https://www.vldb.org/pvldb/vol13/p2901-wang.pdf> - on which you are a
co-author :) - had shown IoTDB working on the Raspberry Pi. I have been
testing the IoTDB data ingestion/ querying on desktop and Raspberry Pi, in
comparison with InfluxDB v2.0. Do let me know if there are any plans of
benchmarking / comparative study that your team intends to do with IoTDB,
InfluxDB or similar TSDBs. I could contribute to that.
Regards,
dgargcs
On Sun, 11 Apr 2021 at 11:55, Jialin Qiao <qi...@apache.org> wrote:
> Hi,
>
> 1. *Can I use the API on v0.11.2 (I am currently using this) or do I
> need v0.11.3?*
>
> You can use 0.11.2, we do not change the RPC API in a minor(bug-fix)
> version.
>
> 2. *Is there any limit to the number of rows that can
> be inserted at once using insertTablet()? Or, is there an optimal number of
> rows per insertTablet() to get best performance?*
>
> It depends on the columns of the Tablet. The default rpc size limit is
> 64MB, so rows * columns * 8Byte < 64MB.
> I usually set rows to 1000 when I have 1000 columns.
>
> 3. *Since insertTablet() also takes rows for one
> device, what exactly is the difference between insertTablet() and
> insertRecordsOfOneDevice()? Are they for different use-cases? Which
> performs better?*
>
> insertRecordsOfOneDevice is actually an improved version of
> insertRecords(): all records belong to one device, so we only acquire one
> writelock in the writing process.
>
> InsertTablet requires that each row has all measurements(using primitive
> data types to store data), insertRecordsOfOneDevice allows each row has
> different measurements(using Object to store data).
>
> Performance: InsertTablet > insertRecordsOfOneDevice > insertRecords >
> insertRecord
>
> InsertTablet is always the fastest :)
>
> 4. *I suppose IoTDB does not have benchmark numbers for such devices? *
>
> We haven't tested these cases.
>
> 5. *How soon can we expect it to come out? Or can I test it out even now
> since I eventually
> plan to start working with it? Is it stable enough?*
>
> The release is already got more than 3 binding votes, it will come out in
> one or two days.
> You can get it from
> https://dist.apache.org/repos/dist/dev/iotdb/0.12.0/rc1
>
> We fixed nearly all important known bugs in 0.12.0. The single-node version
> is stable enough.
> The data migration in the cluster version is not supported, and the test of
> the cluster version is not very much.
> Welcome to test it and give feedback :)
>
> 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
> improve DB data ingestion/query performance?*
>
> It removes some redundant fields in the previous version (decreasing disk
> occupation) and optimizes the performance of the raw data query.
>
> Thanks,
> —————————————————
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> Dhruv Garg <dg...@gmail.com> 于2021年4月11日周日 下午1:38写道：
>
> > Hello Jialin,
> >
> > Thanks for your response.
> >
> > 1. Alright I will definitely move from insertRecords() to insertTablet()
> > then. *Can I use the API on v0.11.2 (I am currently using this) or do I
> > need v0.11.3?*
> >
> > 2. The info you provided on insertTablet() is very helpful. I could start
> > with that. I think specifying the data types at the top of the file will
> > also help reduce the data-type inference time for the api and possibly
> > reduce ingestion time. *Is there any limit to the number of rows that can
> > be inserted at once using insertTablet()? Or, is there an optimal number
> of
> > rows per insertTablet() to get best performance?*
> >
> > 3. Also, I see that there is a new API in v0.12 called
> > insertRecordsOfOneDevice(). *Since insertTablet() also takes rows for one
> > device, what exactly is the difference between insertTablet() and
> > insertRecordsOfOneDevice()? Are they for different use-cases? Which
> > performs better?*
> >
> > 4. So the Raspberry Pi are low-end devices. They run on lower RAM (2GB),
> > have ARM processors and use SD cards for persistent storage. *I suppose
> > IoTDB does not have benchmark numbers for such devices? *
> >
> > A couple of additional questions:
> > 5. I see that the IoTDB team is nearing release of v0.12. *How soon can
> we
> > expect it to come out? Or can I test it out even now since I eventually
> > plan to start working with it? Is it stable enough?*
> >
> > 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
> > improve DB data ingestion/query performance?*
> >
> > Thanks in advance!
> >
> > On Thu, 8 Apr 2021 at 17:26, Jialin Qiao <qi...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > 1. InsertTablets could reach more than 3 times faster than
> insertRecords.
> > >
> > > 2. Yes, Tablet is actually a small table with some columns and rows.
> > > It has a time column and many value columns. In each row, all columns
> > must
> > > have a value.
> > >
> > > An example:
> > >
> > > time, root.sg.d1.s1, root.sg.d1.s2
> > > 1, 1, 2.2
> > > 2, 1, 2.2
> > > 3, 1, 2.2
> > >
> > > Tablets do not allow to have null values.
> > >
> > > However, the tablet uses an array of primitive types to store data in a
> > > columnar format, e.g., long[], int[], double[].
> > > If you store data in a text file, the data type indicator is needed,
> like
> > > this:
> > >
> > > time, root.sg.d1.s1, root.sg.d1.s2
> > > long, int, double
> > > 1, 1, 2.2
> > > 2, 1, 2.2
> > > 3, 1, 2.2
> > >
> > > Then you could generate a tablet from the data and then use the
> > > insertTablet.
> > >
> > > 3. You could check the memory allocated in Raspberry pi. Is it the same
> > as
> > > on desktop? This may impact the write throughput.
> > >
> > > Thanks,
> > > —————————————————
> > > Jialin Qiao
> > > School of Software, Tsinghua University
> > >
> > > 乔嘉林
> > > 清华大学 软件学院
> > >
> > >
> > > Dhruv Garg <dg...@gmail.com> 于2021年4月8日周四 下午1:36写道：
> > >
> > > > Hello all,
> > > >
> > > > In the past month I have been using the JDBC client of IoTDB to write
> > > data
> > > > from CSV into IoTDB and also query on the data. Looking at the CSV
> code
> > > in
> > > > ImportCsv.java
> > > > <
> > > >
> > >
> >
> https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
> > > > >,
> > > > it seems that csv itself is again parsed into an IoTDB-friendly
> > structure
> > > > and then ingested. I would like to avoid the csv-parsing time and
> > > directly
> > > > provide the data as needed to IoTDB. This should improve the
> ingestion
> > > > performance.
> > > >
> > > > What I am talking about is similar to InfluxDB where we can also
> parse
> > > from
> > > > CSV to InfluxDB's native line protocol and then ingest the data into
> > DB.
> > > > However, for better performance, InfluxDB also provides write APIs
> > > > <
> > > >
> > >
> >
> https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java
> > > > >
> > > > to take records in line protocol as input and directly ingest those.
> > > >
> > > > I have three questions:
> > > >
> > > > 1. I see that IoTDB is getting newer write APIs like InsertTablets
> > and
> > > > it seems that it is designed to be faster than insertRecords.
> > > > Approximately
> > > > how much performance improvement have you seen with InsertTablets?
> > > > 2. Is there a way to create a data file such that it is easy to
> use
> > > > InsertTablets with it? This is to know if I can create an
> > > IoTDB-friendly
> > > > and IoTDB-specific input file and then directly use InsertTablets
> to
> > > > ingest
> > > > the data.
> > > > 3. As a preliminary check, I am also trying out IoTDB on Raspberry
> > Pi
> > > 4B
> > > > devices. However, the ingestion time with CSV on the Raspberry Pi
> is
> > > > taking
> > > > 10 times of what it is on the desktop (amd64). This ratio should
> > have
> > > > ideally been closer to 4X, based on other applications that I have
> > > > benchmarked. Have you all run any Raspberry Pi benchmarks for
> IoTDB
> > > > earlier?
> > > >
> > > > I would be awaiting your response. Thanks!
> > > >
> > > > Regards,
> > > > dgargcs
> > > >
> > >
> >
>
Re: IoTDB write performance
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
1. *Can I use the API on v0.11.2 (I am currently using this) or do I
need v0.11.3?*
You can use 0.11.2, we do not change the RPC API in a minor(bug-fix)
version.
2. *Is there any limit to the number of rows that can
be inserted at once using insertTablet()? Or, is there an optimal number of
rows per insertTablet() to get best performance?*
It depends on the columns of the Tablet. The default rpc size limit is
64MB, so rows * columns * 8Byte < 64MB.
I usually set rows to 1000 when I have 1000 columns.
3. *Since insertTablet() also takes rows for one
device, what exactly is the difference between insertTablet() and
insertRecordsOfOneDevice()? Are they for different use-cases? Which
performs better?*
insertRecordsOfOneDevice is actually an improved version of
insertRecords(): all records belong to one device, so we only acquire one
writelock in the writing process.
InsertTablet requires that each row has all measurements(using primitive
data types to store data), insertRecordsOfOneDevice allows each row has
different measurements(using Object to store data).
Performance: InsertTablet > insertRecordsOfOneDevice > insertRecords >
insertRecord
InsertTablet is always the fastest :)
4. *I suppose IoTDB does not have benchmark numbers for such devices? *
We haven't tested these cases.
5. *How soon can we expect it to come out? Or can I test it out even now
since I eventually
plan to start working with it? Is it stable enough?*
The release is already got more than 3 binding votes, it will come out in
one or two days.
You can get it from https://dist.apache.org/repos/dist/dev/iotdb/0.12.0/rc1
We fixed nearly all important known bugs in 0.12.0. The single-node version
is stable enough.
The data migration in the cluster version is not supported, and the test of
the cluster version is not very much.
Welcome to test it and give feedback :)
6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
improve DB data ingestion/query performance?*
It removes some redundant fields in the previous version (decreasing disk
occupation) and optimizes the performance of the raw data query.
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Dhruv Garg <dg...@gmail.com> 于2021年4月11日周日 下午1:38写道：
> Hello Jialin,
>
> Thanks for your response.
>
> 1. Alright I will definitely move from insertRecords() to insertTablet()
> then. *Can I use the API on v0.11.2 (I am currently using this) or do I
> need v0.11.3?*
>
> 2. The info you provided on insertTablet() is very helpful. I could start
> with that. I think specifying the data types at the top of the file will
> also help reduce the data-type inference time for the api and possibly
> reduce ingestion time. *Is there any limit to the number of rows that can
> be inserted at once using insertTablet()? Or, is there an optimal number of
> rows per insertTablet() to get best performance?*
>
> 3. Also, I see that there is a new API in v0.12 called
> insertRecordsOfOneDevice(). *Since insertTablet() also takes rows for one
> device, what exactly is the difference between insertTablet() and
> insertRecordsOfOneDevice()? Are they for different use-cases? Which
> performs better?*
>
> 4. So the Raspberry Pi are low-end devices. They run on lower RAM (2GB),
> have ARM processors and use SD cards for persistent storage. *I suppose
> IoTDB does not have benchmark numbers for such devices? *
>
> A couple of additional questions:
> 5. I see that the IoTDB team is nearing release of v0.12. *How soon can we
> expect it to come out? Or can I test it out even now since I eventually
> plan to start working with it? Is it stable enough?*
>
> 6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
> improve DB data ingestion/query performance?*
>
> Thanks in advance!
>
> On Thu, 8 Apr 2021 at 17:26, Jialin Qiao <qi...@apache.org> wrote:
>
> > Hi,
> >
> > 1. InsertTablets could reach more than 3 times faster than insertRecords.
> >
> > 2. Yes, Tablet is actually a small table with some columns and rows.
> > It has a time column and many value columns. In each row, all columns
> must
> > have a value.
> >
> > An example:
> >
> > time, root.sg.d1.s1, root.sg.d1.s2
> > 1, 1, 2.2
> > 2, 1, 2.2
> > 3, 1, 2.2
> >
> > Tablets do not allow to have null values.
> >
> > However, the tablet uses an array of primitive types to store data in a
> > columnar format, e.g., long[], int[], double[].
> > If you store data in a text file, the data type indicator is needed, like
> > this:
> >
> > time, root.sg.d1.s1, root.sg.d1.s2
> > long, int, double
> > 1, 1, 2.2
> > 2, 1, 2.2
> > 3, 1, 2.2
> >
> > Then you could generate a tablet from the data and then use the
> > insertTablet.
> >
> > 3. You could check the memory allocated in Raspberry pi. Is it the same
> as
> > on desktop? This may impact the write throughput.
> >
> > Thanks,
> > —————————————————
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> >
> > Dhruv Garg <dg...@gmail.com> 于2021年4月8日周四 下午1:36写道：
> >
> > > Hello all,
> > >
> > > In the past month I have been using the JDBC client of IoTDB to write
> > data
> > > from CSV into IoTDB and also query on the data. Looking at the CSV code
> > in
> > > ImportCsv.java
> > > <
> > >
> >
> https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
> > > >,
> > > it seems that csv itself is again parsed into an IoTDB-friendly
> structure
> > > and then ingested. I would like to avoid the csv-parsing time and
> > directly
> > > provide the data as needed to IoTDB. This should improve the ingestion
> > > performance.
> > >
> > > What I am talking about is similar to InfluxDB where we can also parse
> > from
> > > CSV to InfluxDB's native line protocol and then ingest the data into
> DB.
> > > However, for better performance, InfluxDB also provides write APIs
> > > <
> > >
> >
> https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java
> > > >
> > > to take records in line protocol as input and directly ingest those.
> > >
> > > I have three questions:
> > >
> > > 1. I see that IoTDB is getting newer write APIs like InsertTablets
> and
> > > it seems that it is designed to be faster than insertRecords.
> > > Approximately
> > > how much performance improvement have you seen with InsertTablets?
> > > 2. Is there a way to create a data file such that it is easy to use
> > > InsertTablets with it? This is to know if I can create an
> > IoTDB-friendly
> > > and IoTDB-specific input file and then directly use InsertTablets to
> > > ingest
> > > the data.
> > > 3. As a preliminary check, I am also trying out IoTDB on Raspberry
> Pi
> > 4B
> > > devices. However, the ingestion time with CSV on the Raspberry Pi is
> > > taking
> > > 10 times of what it is on the desktop (amd64). This ratio should
> have
> > > ideally been closer to 4X, based on other applications that I have
> > > benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB
> > > earlier?
> > >
> > > I would be awaiting your response. Thanks!
> > >
> > > Regards,
> > > dgargcs
> > >
> >
>
Re: IoTDB write performance
Posted by Dhruv Garg <dg...@gmail.com>.
Hello Jialin,
Thanks for your response.
1. Alright I will definitely move from insertRecords() to insertTablet()
then. *Can I use the API on v0.11.2 (I am currently using this) or do I
need v0.11.3?*
2. The info you provided on insertTablet() is very helpful. I could start
with that. I think specifying the data types at the top of the file will
also help reduce the data-type inference time for the api and possibly
reduce ingestion time. *Is there any limit to the number of rows that can
be inserted at once using insertTablet()? Or, is there an optimal number of
rows per insertTablet() to get best performance?*
3. Also, I see that there is a new API in v0.12 called
insertRecordsOfOneDevice(). *Since insertTablet() also takes rows for one
device, what exactly is the difference between insertTablet() and
insertRecordsOfOneDevice()? Are they for different use-cases? Which
performs better?*
4. So the Raspberry Pi are low-end devices. They run on lower RAM (2GB),
have ARM processors and use SD cards for persistent storage. *I suppose
IoTDB does not have benchmark numbers for such devices? *
A couple of additional questions:
5. I see that the IoTDB team is nearing release of v0.12. *How soon can we
expect it to come out? Or can I test it out even now since I eventually
plan to start working with it? Is it stable enough?*
6. *What benefits does the new Tsfile structure in v0.12 bring? Does it
improve DB data ingestion/query performance?*
Thanks in advance!
On Thu, 8 Apr 2021 at 17:26, Jialin Qiao <qi...@apache.org> wrote:
> Hi,
>
> 1. InsertTablets could reach more than 3 times faster than insertRecords.
>
> 2. Yes, Tablet is actually a small table with some columns and rows.
> It has a time column and many value columns. In each row, all columns must
> have a value.
>
> An example:
>
> time, root.sg.d1.s1, root.sg.d1.s2
> 1, 1, 2.2
> 2, 1, 2.2
> 3, 1, 2.2
>
> Tablets do not allow to have null values.
>
> However, the tablet uses an array of primitive types to store data in a
> columnar format, e.g., long[], int[], double[].
> If you store data in a text file, the data type indicator is needed, like
> this:
>
> time, root.sg.d1.s1, root.sg.d1.s2
> long, int, double
> 1, 1, 2.2
> 2, 1, 2.2
> 3, 1, 2.2
>
> Then you could generate a tablet from the data and then use the
> insertTablet.
>
> 3. You could check the memory allocated in Raspberry pi. Is it the same as
> on desktop? This may impact the write throughput.
>
> Thanks,
> —————————————————
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> Dhruv Garg <dg...@gmail.com> 于2021年4月8日周四 下午1:36写道：
>
> > Hello all,
> >
> > In the past month I have been using the JDBC client of IoTDB to write
> data
> > from CSV into IoTDB and also query on the data. Looking at the CSV code
> in
> > ImportCsv.java
> > <
> >
> https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
> > >,
> > it seems that csv itself is again parsed into an IoTDB-friendly structure
> > and then ingested. I would like to avoid the csv-parsing time and
> directly
> > provide the data as needed to IoTDB. This should improve the ingestion
> > performance.
> >
> > What I am talking about is similar to InfluxDB where we can also parse
> from
> > CSV to InfluxDB's native line protocol and then ingest the data into DB.
> > However, for better performance, InfluxDB also provides write APIs
> > <
> >
> https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java
> > >
> > to take records in line protocol as input and directly ingest those.
> >
> > I have three questions:
> >
> > 1. I see that IoTDB is getting newer write APIs like InsertTablets and
> > it seems that it is designed to be faster than insertRecords.
> > Approximately
> > how much performance improvement have you seen with InsertTablets?
> > 2. Is there a way to create a data file such that it is easy to use
> > InsertTablets with it? This is to know if I can create an
> IoTDB-friendly
> > and IoTDB-specific input file and then directly use InsertTablets to
> > ingest
> > the data.
> > 3. As a preliminary check, I am also trying out IoTDB on Raspberry Pi
> 4B
> > devices. However, the ingestion time with CSV on the Raspberry Pi is
> > taking
> > 10 times of what it is on the desktop (amd64). This ratio should have
> > ideally been closer to 4X, based on other applications that I have
> > benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB
> > earlier?
> >
> > I would be awaiting your response. Thanks!
> >
> > Regards,
> > dgargcs
> >
>
Re: IoTDB write performance
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
1. InsertTablets could reach more than 3 times faster than insertRecords.
2. Yes, Tablet is actually a small table with some columns and rows.
It has a time column and many value columns. In each row, all columns must
have a value.
An example:
time, root.sg.d1.s1, root.sg.d1.s2
1, 1, 2.2
2, 1, 2.2
3, 1, 2.2
Tablets do not allow to have null values.
However, the tablet uses an array of primitive types to store data in a
columnar format, e.g., long[], int[], double[].
If you store data in a text file, the data type indicator is needed, like
this:
time, root.sg.d1.s1, root.sg.d1.s2
long, int, double
1, 1, 2.2
2, 1, 2.2
3, 1, 2.2
Then you could generate a tablet from the data and then use the
insertTablet.
3. You could check the memory allocated in Raspberry pi. Is it the same as
on desktop? This may impact the write throughput.
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Dhruv Garg <dg...@gmail.com> 于2021年4月8日周四 下午1:36写道：
> Hello all,
>
> In the past month I have been using the JDBC client of IoTDB to write data
> from CSV into IoTDB and also query on the data. Looking at the CSV code in
> ImportCsv.java
> <
> https://github.com/apache/iotdb/blob/master/cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
> >,
> it seems that csv itself is again parsed into an IoTDB-friendly structure
> and then ingested. I would like to avoid the csv-parsing time and directly
> provide the data as needed to IoTDB. This should improve the ingestion
> performance.
>
> What I am talking about is similar to InfluxDB where we can also parse from
> CSV to InfluxDB's native line protocol and then ingest the data into DB.
> However, for better performance, InfluxDB also provides write APIs
> <
> https://github.com/influxdata/influxdb-client-java/blob/master/client/src/main/java/com/influxdb/client/WriteApi.java
> >
> to take records in line protocol as input and directly ingest those.
>
> I have three questions:
>
> 1. I see that IoTDB is getting newer write APIs like InsertTablets and
> it seems that it is designed to be faster than insertRecords.
> Approximately
> how much performance improvement have you seen with InsertTablets?
> 2. Is there a way to create a data file such that it is easy to use
> InsertTablets with it? This is to know if I can create an IoTDB-friendly
> and IoTDB-specific input file and then directly use InsertTablets to
> ingest
> the data.
> 3. As a preliminary check, I am also trying out IoTDB on Raspberry Pi 4B
> devices. However, the ingestion time with CSV on the Raspberry Pi is
> taking
> 10 times of what it is on the desktop (amd64). This ratio should have
> ideally been closer to 4X, based on other applications that I have
> benchmarked. Have you all run any Raspberry Pi benchmarks for IoTDB
> earlier?
>
> I would be awaiting your response. Thanks!
>
> Regards,
> dgargcs
>