You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@community.apache.org by "Xiangdong Huang (Jira)" <ji...@apache.org> on 2021/03/11 15:05:00 UTC

[jira] [Created] (COMDEV-401) Apache IoTDB: Metadata (Schema) Storage Engine

Xiangdong Huang created COMDEV-401:
--------------------------------------

             Summary: Apache IoTDB: Metadata (Schema) Storage Engine
                 Key: COMDEV-401
                 URL: https://issues.apache.org/jira/browse/COMDEV-401
             Project: Community Development
          Issue Type: Task
          Components: GSoC/Mentoring ideas
            Reporter: Xiangdong Huang


Apache IoTDB [1] is an Open Source IoT database designed to meet the rigorous data, storage, and analytics requirements of large-scale Internet of Things (IoT) and Industrial Internet of Things (IIoT) applications. 

Different with traditional relational databases, IoTDB uses a tree-based structure to store its schema (a.k.a, metadata) in memory, and only persists the structure on disk in a write ahead log form.

In memory, the memory cost of each time series wil take about 300 bytes.

However, when the tree is too large (e.g., there are 100 million time series), IoTDB will take too much memory.

Therefore, we'd like to develop a metadata storage engine module. The engine will do the following tasks:

1. data file: Persist the tree on disk, like a b-tree structure. (need to support CRUD)
2. WAL: to accelerate the IO operation, we can save schema modification operations as Write Ahead Log, and then write back to the tree file in an atomic way.
3. Cache: If there is no much memory, just load a part of the tree into memory. 

What you need:
1. Java language
2. Know some basic design idea of database data file, e.g., slot file.[2]

[1] https://iotdb.apache.org
[2] http://pages.cs.wisc.edu/~dbbook/openAccess/firstEdition/slides/pdfslides/mod2l1.pdf



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
For additional commands, e-mail: dev-help@community.apache.org

Re: [jira] [Created] (COMDEV-401) Apache IoTDB: Metadata (Schema) Storage Engine

Posted by Xiangdong Huang <sa...@gmail.com>.
Hi Garcia,

Welcome to join.

This issue is moved to https://issues.apache.org/jira/browse/COMDEV-403.

Best,
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Isidoro Garcia Gutierrez <ig...@uoc.edu> 于2021年3月11日周四 下午11:42写道：

> Hello,
>
> My name is Isidoro Garcia and next june I will finish my Software
> Engineering degree. I've some experience in java programming and in my
> final project to the university I have to desing a new DB.
>
> I find interesting IoTDB and I would like to contribute to this project for
> GSoC 2021. If there is something I can do, please, let me know.
>
> Regards.
>
>
> El jue, 11 mar 2021 a las 16:05, Xiangdong Huang (Jira) (<jira@apache.org
> >)
> escribió:
>
> > Xiangdong Huang created COMDEV-401:
> > --------------------------------------
> >
> >              Summary: Apache IoTDB: Metadata (Schema) Storage Engine
> >                  Key: COMDEV-401
> >                  URL: https://issues.apache.org/jira/browse/COMDEV-401
> >              Project: Community Development
> >           Issue Type: Task
> >           Components: GSoC/Mentoring ideas
> >             Reporter: Xiangdong Huang
> >
> >
> > Apache IoTDB [1] is an Open Source IoT database designed to meet the
> > rigorous data, storage, and analytics requirements of large-scale
> Internet
> > of Things (IoT) and Industrial Internet of Things (IIoT) applications.
> >
> > Different with traditional relational databases, IoTDB uses a tree-based
> > structure to store its schema (a.k.a, metadata) in memory, and only
> > persists the structure on disk in a write ahead log form.
> >
> > In memory, the memory cost of each time series wil take about 300 bytes.
> >
> > However, when the tree is too large (e.g., there are 100 million time
> > series), IoTDB will take too much memory.
> >
> > Therefore, we'd like to develop a metadata storage engine module. The
> > engine will do the following tasks:
> >
> > 1. data file: Persist the tree on disk, like a b-tree structure. (need to
> > support CRUD)
> > 2. WAL: to accelerate the IO operation, we can save schema modification
> > operations as Write Ahead Log, and then write back to the tree file in an
> > atomic way.
> > 3. Cache: If there is no much memory, just load a part of the tree into
> > memory.
> >
> > What you need:
> > 1. Java language
> > 2. Know some basic design idea of database data file, e.g., slot file.[2]
> >
> > [1] https://iotdb.apache.org
> > [2]
> >
> http://pages.cs.wisc.edu/~dbbook/openAccess/firstEdition/slides/pdfslides/mod2l1.pdf
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> > For additional commands, e-mail: dev-help@community.apache.org
> >
> >
>
> --
>
>
>
> INFORMACIÓ SOBRE PROTECCIÓ DE DADES DE LA UNIVERSITAT OBERTA DE
> CATALUNYA (UOC)
>
> Us informem que les vostres dades identificatives i les
> contingudes en els missatges electrònics i fitxers adjunts es poden
> incorporar a les nostres bases de dades amb la finalitat de gestionar les
> relacions i comunicacions vinculades a la UOC, i que es poden conservar
> mentre es mantingui la relació. Si ho voleu, podeu exercir el dret a
> accedir a les vostres dades, rectificar-les i suprimir-les i altres drets
> reconeguts normativament adreçant-vos a l'adreça de correu emissora o a
> fuoc_pd@uoc.edu <ma...@uoc.edu>.
>
> Aquest missatge i qualsevol
> fitxer que porti adjunt, si escau, tenen el caràcter de confidencials i
> s'adrecen únicament a la persona o entitat a qui s'han enviat.
>
> Així
> mateix, posem a la vostra disposició un delegat de protecció de dades que
> no només s'encarregarà de supervisar tots els tractaments de dades de la
> nostra entitat, sinó que us podrà atendre per a qualsevol qüestió
> relacionada amb el tractament de dades. La seva adreça de contacte és
> dpd@uoc.edu <ma...@uoc.edu>.
> INFORMACIÓN SOBRE PROTECCIÓN DE DATOS DE
> LA UNIVERSITAT OBERTA DE CATALUNYA (UOC)
> Os informamos de que vuestros
> datos identificativos y los contenidos en los mensajes electrónicos y
> ficheros adjuntos pueden incorporarse a nuestras bases de datos con el fin
> de gestionar las relaciones y comunicaciones vinculadas a la UOC, y de que
> pueden conservarse mientras se mantenga la relación. Si lo deseáis, podéis
> ejercer el derecho a acceder a vuestros datos, rectificarlos y suprimirlos
> y otros derechos reconocidos normativamente dirigiéndoos a la dirección de
> correo emisora o a fuoc_pd@uoc.edu <ma...@uoc.edu>.
> Este mensaje y
> cualquier fichero que lleve adjunto, si procede, tienen el carácter de
> confidenciales y se dirigen únicamente a la persona o entidad a quien se
> han enviado.
> Así mismo, ponemos a vuestra disposición a un delegado de
> protección de datos que no solo se encargará de supervisar todos los
> tratamientos de datos de nuestra entidad, sino que podrá atenderos para
> cualquier cuestión relacionada con el tratamiento de datos. Su dirección
> de
> contacto es dpd@uoc.edu <ma...@uoc.edu>.
>
>
> UNIVERSITAT OBERTA DE
> CATALUNYA (UOC) DATA PROTECTION INFORMATION
> Your personal data and the data
> contained in your email messages and attached files may be stored in our
> databases for the purpose of maintaining relations and communications
> linked to the UOC, and the data may be stored for as long as these
> relations and communications are maintained. If you so wish, you can
> exercise your rights to access, rectification and erasure of your data,
> and
> any other legally held rights, by writing to the sender’s email address or
> to fuoc_pd@uoc.edu <ht...@uoc.edu>.
> This message and, where
> applicable, any attachments are confidential and addressed solely to the
> individual or organization they were sent to.
> The UOC has a data protection
> officer who not only supervises the data processing carried out at the
> University, but who will also respond to any questions you may have about
> this data processing. You can contact our data protection officer by
> writing to dpd@uoc.edu <ht...@uoc.edu>.
>
>
>
>

Re: [jira] [Created] (COMDEV-401) Apache IoTDB: Metadata (Schema) Storage Engine

Posted by Isidoro Garcia Gutierrez <ig...@uoc.edu>.
Hello,

My name is Isidoro Garcia and next june I will finish my Software
Engineering degree. I've some experience in java programming and in my
final project to the university I have to desing a new DB.

I find interesting IoTDB and I would like to contribute to this project for
GSoC 2021. If there is something I can do, please, let me know.

Regards.


El jue, 11 mar 2021 a las 16:05, Xiangdong Huang (Jira) (<ji...@apache.org>)
escribió:

> Xiangdong Huang created COMDEV-401:
> --------------------------------------
>
>              Summary: Apache IoTDB: Metadata (Schema) Storage Engine
>                  Key: COMDEV-401
>                  URL: https://issues.apache.org/jira/browse/COMDEV-401
>              Project: Community Development
>           Issue Type: Task
>           Components: GSoC/Mentoring ideas
>             Reporter: Xiangdong Huang
>
>
> Apache IoTDB [1] is an Open Source IoT database designed to meet the
> rigorous data, storage, and analytics requirements of large-scale Internet
> of Things (IoT) and Industrial Internet of Things (IIoT) applications.
>
> Different with traditional relational databases, IoTDB uses a tree-based
> structure to store its schema (a.k.a, metadata) in memory, and only
> persists the structure on disk in a write ahead log form.
>
> In memory, the memory cost of each time series wil take about 300 bytes.
>
> However, when the tree is too large (e.g., there are 100 million time
> series), IoTDB will take too much memory.
>
> Therefore, we'd like to develop a metadata storage engine module. The
> engine will do the following tasks:
>
> 1. data file: Persist the tree on disk, like a b-tree structure. (need to
> support CRUD)
> 2. WAL: to accelerate the IO operation, we can save schema modification
> operations as Write Ahead Log, and then write back to the tree file in an
> atomic way.
> 3. Cache: If there is no much memory, just load a part of the tree into
> memory.
>
> What you need:
> 1. Java language
> 2. Know some basic design idea of database data file, e.g., slot file.[2]
>
> [1] https://iotdb.apache.org
> [2]
> http://pages.cs.wisc.edu/~dbbook/openAccess/firstEdition/slides/pdfslides/mod2l1.pdf
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@community.apache.org
> For additional commands, e-mail: dev-help@community.apache.org
>
>

-- 



INFORMACIÓ SOBRE PROTECCIÓ DE DADES DE LA UNIVERSITAT OBERTA DE 
CATALUNYA (UOC)

Us informem que les vostres dades identificatives i les 
contingudes en els missatges electrònics i fitxers adjunts es poden 
incorporar a les nostres bases de dades amb la finalitat de gestionar les 
relacions i comunicacions vinculades a la UOC, i que es poden conservar 
mentre es mantingui la relació. Si ho voleu, podeu exercir el dret a 
accedir a les vostres dades, rectificar-les i suprimir-les i altres drets 
reconeguts normativament adreçant-vos a l'adreça de correu emissora o a 
fuoc_pd@uoc.edu <ma...@uoc.edu>.

Aquest missatge i qualsevol 
fitxer que porti adjunt, si escau, tenen el caràcter de confidencials i 
s'adrecen únicament a la persona o entitat a qui s'han enviat.

Així 
mateix, posem a la vostra disposició un delegat de protecció de dades que 
no només s'encarregarà de supervisar tots els tractaments de dades de la 
nostra entitat, sinó que us podrà atendre per a qualsevol qüestió 
relacionada amb el tractament de dades. La seva adreça de contacte és 
dpd@uoc.edu <ma...@uoc.edu>.
INFORMACIÓN SOBRE PROTECCIÓN DE DATOS DE 
LA UNIVERSITAT OBERTA DE CATALUNYA (UOC)
Os informamos de que vuestros 
datos identificativos y los contenidos en los mensajes electrónicos y 
ficheros adjuntos pueden incorporarse a nuestras bases de datos con el fin 
de gestionar las relaciones y comunicaciones vinculadas a la UOC, y de que 
pueden conservarse mientras se mantenga la relación. Si lo deseáis, podéis 
ejercer el derecho a acceder a vuestros datos, rectificarlos y suprimirlos 
y otros derechos reconocidos normativamente dirigiéndoos a la dirección de 
correo emisora o a fuoc_pd@uoc.edu <ma...@uoc.edu>.
Este mensaje y 
cualquier fichero que lleve adjunto, si procede, tienen el carácter de 
confidenciales y se dirigen únicamente a la persona o entidad a quien se 
han enviado.
Así mismo, ponemos a vuestra disposición a un delegado de 
protección de datos que no solo se encargará de supervisar todos los 
tratamientos de datos de nuestra entidad, sino que podrá atenderos para 
cualquier cuestión relacionada con el tratamiento de datos. Su dirección de 
contacto es dpd@uoc.edu <ma...@uoc.edu>.


UNIVERSITAT OBERTA DE 
CATALUNYA (UOC) DATA PROTECTION INFORMATION
Your personal data and the data 
contained in your email messages and attached files may be stored in our 
databases for the purpose of maintaining relations and communications 
linked to the UOC, and the data may be stored for as long as these 
relations and communications are maintained. If you so wish, you can 
exercise your rights to access, rectification and erasure of your data, and 
any other legally held rights, by writing to the sender’s email address or 
to fuoc_pd@uoc.edu <ht...@uoc.edu>.
This message and, where 
applicable, any attachments are confidential and addressed solely to the 
individual or organization they were sent to.
The UOC has a data protection 
officer who not only supervises the data processing carried out at the 
University, but who will also respond to any questions you may have about 
this data processing. You can contact our data protection officer by 
writing to dpd@uoc.edu <ht...@uoc.edu>.