You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by QiangCai <qi...@qq.com> on 2016/08/05 06:31:42 UTC
Discussing to add carbondata-tools module
Hi all,
To improve the CarbonData system's usability and maintainability, I
suggest to add carbondata-tools module.
I think this module should provide some command tools as following.
1. import
import a data file/folder to any existing table
2. export
export the given columns to a file
3. schema
show the detail information of the specified table schema file
4. metadata
show tablestatus metadata
show the history track for dataloading and compaction
5. footer
show blocklet metadata list
show start/end key, min/max value, row number, total size for the
specified blocklet
6. blocklet
show blocket list
show blocket data, RLE map, Inverted index for the given columns
7. index
show BTree node list
show node information:start/end key, min/max value)
8. dictionary
show key-value list of specified gloabl/local dictionary file
show sort index
show dictionary metadata
Thank you and look forward to having your opinion on this carbon-tools
module.
David Cai
--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Re: Discussing to add carbondata-tools module
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1
Regards
JB
On 08/05/2016 08:31 AM, QiangCai wrote:
> Hi all,
>
> To improve the CarbonData system's usability and maintainability, I
> suggest to add carbondata-tools module.
> I think this module should provide some command tools as following.
>
> 1. import
> import a data file/folder to any existing table
>
> 2. export
> export the given columns to a file
>
> 3. schema
> show the detail information of the specified table schema file
>
> 4. metadata
> show tablestatus metadata
> show the history track for dataloading and compaction
>
> 5. footer
> show blocklet metadata list
> show start/end key, min/max value, row number, total size for the
> specified blocklet
>
> 6. blocklet
> show blocket list
> show blocket data, RLE map, Inverted index for the given columns
>
> 7. index
> show BTree node list
> show node information:start/end key, min/max value)
>
> 8. dictionary
> show key-value list of specified gloabl/local dictionary file
> show sort index
> show dictionary metadata
>
> Thank you and look forward to having your opinion on this carbon-tools
> module.
>
> David Cai
>
>
>
> --
> View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
>
--
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Discussing to add carbondata-tools module
Posted by chenliang613 <ch...@gmail.com>.
Thanks for you started the discussion, these tools look good.
I am trying to understand why need these tools : Can you provide some
scenarios info, these tools could help users to solve what issues?
Regards
Liang
--
View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4p9.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
Re: Discussing to add carbondata-tools module
Posted by Raghunandan S <ca...@gmail.com>.
+1
I suggest we separate the tools as usability and Dev/debugging tools
Take up the usability tools with highest priority
On Mon, 8 Aug 2016 at 10:32 AM, Venkata Gollamudi <g....@gmail.com>
wrote:
> +1
> These list looks like both user needed tools for maintenance and dev tools.
> Mostly they can be separated.
>
> On Fri, Aug 5, 2016, 6:03 PM 金铸 <ji...@neusoft.com> wrote:
>
> > +1
> >
> >
> > 在 2016/8/5 17:15, Jean-Baptiste Onofré 写道:
> > > I guess it's where we can put the checkstyle, ... (what we have in the
> > > dev folder right now), correct ?
> > >
> > > Regards
> > > JB
> > >
> > > On 08/05/2016 08:31 AM, QiangCai wrote:
> > >> Hi all,
> > >>
> > >> To improve the CarbonData system's usability and maintainability, I
> > >> suggest to add carbondata-tools module.
> > >> I think this module should provide some command tools as following.
> > >>
> > >> 1. import
> > >> import a data file/folder to any existing table
> > >>
> > >> 2. export
> > >> export the given columns to a file
> > >>
> > >> 3. schema
> > >> show the detail information of the specified table schema file
> > >>
> > >> 4. metadata
> > >> show tablestatus metadata
> > >> show the history track for dataloading and compaction
> > >>
> > >> 5. footer
> > >> show blocklet metadata list
> > >> show start/end key, min/max value, row number, total size for the
> > >> specified blocklet
> > >>
> > >> 6. blocklet
> > >> show blocket list
> > >> show blocket data, RLE map, Inverted index for the given columns
> > >>
> > >> 7. index
> > >> show BTree node list
> > >> show node information:start/end key, min/max value)
> > >>
> > >> 8. dictionary
> > >> show key-value list of specified gloabl/local dictionary file
> > >> show sort index
> > >> show dictionary metadata
> > >>
> > >> Thank you and look forward to having your opinion on this carbon-tools
> > >> module.
> > >>
> > >> David Cai
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
> > >> Sent from the Apache CarbonData Mailing List archive mailing list
> > >> archive at Nabble.com.
> > >>
> > >
> >
> > --
> > 金铸
> > 技术发展部（TDD）
> > 东软集团股份有限公司
> > 沈阳浑南新区新秀街2号东软软件园A2-105A
> > Postcode:110179
> > Tel: (86 24)8366 2049
> > Mobile：13897999526
> >
> >
> >
> >
> >
> >
> >
> ---------------------------------------------------------------------------------------------------
> > Confidentiality Notice: The information contained in this e-mail and any
> > accompanying attachment(s)
> > is intended only for the use of the intended recipient and may be
> > confidential and/or privileged of
> > Neusoft Corporation, its subsidiaries and/or its affiliates. If any
> reader
> > of this communication is
> > not the intended recipient, unauthorized use, forwarding, printing,
> > storing, disclosure or copying
> > is strictly prohibited, and may be unlawful.If you have received this
> > communication in error,please
> > immediately notify the sender by return e-mail, and delete the original
> > message and all copies from
> > your system. Thank you.
> >
> >
> ---------------------------------------------------------------------------------------------------
> >
>
Re: Discussing to add carbondata-tools module
Posted by Venkata Gollamudi <g....@gmail.com>.
+1
These list looks like both user needed tools for maintenance and dev tools.
Mostly they can be separated.
On Fri, Aug 5, 2016, 6:03 PM 金铸 <ji...@neusoft.com> wrote:
> +1
>
>
> 在 2016/8/5 17:15, Jean-Baptiste Onofré 写道:
> > I guess it's where we can put the checkstyle, ... (what we have in the
> > dev folder right now), correct ?
> >
> > Regards
> > JB
> >
> > On 08/05/2016 08:31 AM, QiangCai wrote:
> >> Hi all,
> >>
> >> To improve the CarbonData system's usability and maintainability, I
> >> suggest to add carbondata-tools module.
> >> I think this module should provide some command tools as following.
> >>
> >> 1. import
> >> import a data file/folder to any existing table
> >>
> >> 2. export
> >> export the given columns to a file
> >>
> >> 3. schema
> >> show the detail information of the specified table schema file
> >>
> >> 4. metadata
> >> show tablestatus metadata
> >> show the history track for dataloading and compaction
> >>
> >> 5. footer
> >> show blocklet metadata list
> >> show start/end key, min/max value, row number, total size for the
> >> specified blocklet
> >>
> >> 6. blocklet
> >> show blocket list
> >> show blocket data, RLE map, Inverted index for the given columns
> >>
> >> 7. index
> >> show BTree node list
> >> show node information:start/end key, min/max value)
> >>
> >> 8. dictionary
> >> show key-value list of specified gloabl/local dictionary file
> >> show sort index
> >> show dictionary metadata
> >>
> >> Thank you and look forward to having your opinion on this carbon-tools
> >> module.
> >>
> >> David Cai
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
> >> Sent from the Apache CarbonData Mailing List archive mailing list
> >> archive at Nabble.com.
> >>
> >
>
> --
> 金铸
> 技术发展部（TDD）
> 东软集团股份有限公司
> 沈阳浑南新区新秀街2号东软软件园A2-105A
> Postcode:110179
> Tel: (86 24)8366 2049
> Mobile：13897999526
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>
Re: Discussing to add carbondata-tools module
Posted by 金铸 <ji...@neusoft.com>.
+1
在 2016/8/5 17:15, Jean-Baptiste Onofré 写道:
> I guess it's where we can put the checkstyle, ... (what we have in the
> dev folder right now), correct ?
>
> Regards
> JB
>
> On 08/05/2016 08:31 AM, QiangCai wrote:
>> Hi all,
>>
>> To improve the CarbonData system's usability and maintainability, I
>> suggest to add carbondata-tools module.
>> I think this module should provide some command tools as following.
>>
>> 1. import
>> import a data file/folder to any existing table
>>
>> 2. export
>> export the given columns to a file
>>
>> 3. schema
>> show the detail information of the specified table schema file
>>
>> 4. metadata
>> show tablestatus metadata
>> show the history track for dataloading and compaction
>>
>> 5. footer
>> show blocklet metadata list
>> show start/end key, min/max value, row number, total size for the
>> specified blocklet
>>
>> 6. blocklet
>> show blocket list
>> show blocket data, RLE map, Inverted index for the given columns
>>
>> 7. index
>> show BTree node list
>> show node information:start/end key, min/max value)
>>
>> 8. dictionary
>> show key-value list of specified gloabl/local dictionary file
>> show sort index
>> show dictionary metadata
>>
>> Thank you and look forward to having your opinion on this carbon-tools
>> module.
>>
>> David Cai
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
>> Sent from the Apache CarbonData Mailing List archive mailing list
>> archive at Nabble.com.
>>
>
--
金铸
技术发展部（TDD）
东软集团股份有限公司
沈阳浑南新区新秀街2号东软软件园A2-105A
Postcode:110179
Tel: (86 24)8366 2049
Mobile：13897999526
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is
not the intended recipient, unauthorized use, forwarding, printing, storing, disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this communication in error,please
immediately notify the sender by return e-mail, and delete the original message and all copies from
your system. Thank you.
---------------------------------------------------------------------------------------------------
Re: Discussing to add carbondata-tools module
Posted by Vimal Das Kammath <vi...@gmail.com>.
+1
On Aug 5, 2016 2:45 PM, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
> I guess it's where we can put the checkstyle, ... (what we have in the dev
> folder right now), correct ?
>
> Regards
> JB
>
> On 08/05/2016 08:31 AM, QiangCai wrote:
>
>> Hi all,
>>
>> To improve the CarbonData system's usability and maintainability, I
>> suggest to add carbondata-tools module.
>> I think this module should provide some command tools as following.
>>
>> 1. import
>> import a data file/folder to any existing table
>>
>> 2. export
>> export the given columns to a file
>>
>> 3. schema
>> show the detail information of the specified table schema file
>>
>> 4. metadata
>> show tablestatus metadata
>> show the history track for dataloading and compaction
>>
>> 5. footer
>> show blocklet metadata list
>> show start/end key, min/max value, row number, total size for the
>> specified blocklet
>>
>> 6. blocklet
>> show blocket list
>> show blocket data, RLE map, Inverted index for the given columns
>>
>> 7. index
>> show BTree node list
>> show node information:start/end key, min/max value)
>>
>> 8. dictionary
>> show key-value list of specified gloabl/local dictionary file
>> show sort index
>> show dictionary metadata
>>
>> Thank you and look forward to having your opinion on this carbon-tools
>> module.
>>
>> David Cai
>>
>>
>>
>> --
>> View this message in context: http://apache-carbondata-maili
>> ng-list-archive.1130556.n5.nabble.com/Discussing-to-add-
>> carbondata-tools-module-tp4.html
>> Sent from the Apache CarbonData Mailing List archive mailing list archive
>> at Nabble.com.
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Discussing to add carbondata-tools module
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I guess it's where we can put the checkstyle, ... (what we have in the
dev folder right now), correct ?
Regards
JB
On 08/05/2016 08:31 AM, QiangCai wrote:
> Hi all,
>
> To improve the CarbonData system's usability and maintainability, I
> suggest to add carbondata-tools module.
> I think this module should provide some command tools as following.
>
> 1. import
> import a data file/folder to any existing table
>
> 2. export
> export the given columns to a file
>
> 3. schema
> show the detail information of the specified table schema file
>
> 4. metadata
> show tablestatus metadata
> show the history track for dataloading and compaction
>
> 5. footer
> show blocklet metadata list
> show start/end key, min/max value, row number, total size for the
> specified blocklet
>
> 6. blocklet
> show blocket list
> show blocket data, RLE map, Inverted index for the given columns
>
> 7. index
> show BTree node list
> show node information:start/end key, min/max value)
>
> 8. dictionary
> show key-value list of specified gloabl/local dictionary file
> show sort index
> show dictionary metadata
>
> Thank you and look forward to having your opinion on this carbon-tools
> module.
>
> David Cai
>
>
>
> --
> View this message in context: http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/Discussing-to-add-carbondata-tools-module-tp4.html
> Sent from the Apache CarbonData Mailing List archive mailing list archive at Nabble.com.
>
--
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com