You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "simonaubertbd (via GitHub)" <gi...@apache.org> on 2024/07/29 05:37:08 UTC

[I] Column/field description [parquet-format]

simonaubertbd opened a new issue, #447:
URL: https://github.com/apache/parquet-format/issues/447

   ### Describe the enhancement requested
   
   ### Describe the enhancement requested
   
   Hello,
   
   I may have misunderstood or misread the documentation but I didn't find it. I would love to have a field/column description just like we have in databases :
   
   ![image](https://github.com/user-attachments/assets/30b01b21-5b99-43e0-b125-c9eab7ffa033)
   
   Best regards,
   
   Simon
   
   ### Component(s)
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2380560534

   Since Parquet is now meant to be a standard file format, use for intermediate storage between tools, it really seems necessary to have this field/column description added. It exists in a lot of format such as Tableau Hyper, Alteryx yxdb, etc
   
   As for now, it means loosing informations when storing in Parquet.
   
   Best regards,
   
   Simon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3195077566

   > Any news about that feature?
   
   No, it would generally need a volunteer to shepherd this through and nobody has had a willingness to so yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3411826349

   > Could you add optional Key/Value Metadata to SchemaElement? This is more general than a description field. It might also to eliminate the need for some extension type handling.
   
   That seems like it could be a reasonable alternative.  I would need to refresh my self on the extension type proposal and think about it more deeply. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "liquidcarbon (via GitHub)" <gi...@apache.org>.
liquidcarbon commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2311019015

   Hi! I'm confused by different types of metadata.
   
   1) there's key-value metadata of type BLOB that you can write via schema metadata  (in python `pyarrow.table(data, metadata=...)`) and read similarly, or through https://duckdb.org/docs/data/parquet/metadata.html#parquet-key-value-metadata
   2) there's separate place, with these fields, that you can read with https://duckdb.org/docs/data/parquet/metadata.html#parquet-metadata
   ```
   ['file_name', 'row_group_id', 'row_group_num_rows',
          'row_group_num_columns', 'row_group_bytes', 'column_id', 'file_offset',
          'num_values', 'path_in_schema', 'type', 'stats_min', 'stats_max',
          'stats_null_count', 'stats_distinct_count', 'stats_min_value',
          'stats_max_value', 'compression', 'encodings', 'index_page_offset',
          'dictionary_page_offset', 'data_page_offset', 'total_compressed_size',
          'total_uncompressed_size', 'key_value_metadata']
   ```
   column-level information and data types are recorded here.  But how to you write something into THAT `key_level_metadata` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "abellgithub (via GitHub)" <gi...@apache.org>.
abellgithub commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3411800898

   > > You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.
   > 
   > Yes, one downside is this gets a little awkward because key value metadata for columns today is at a row-group level. which means for file level metadata one would need to define a serialization format for mapping column to description.
   
   Could you add optional Key/Value Metadata to `SchemaElement`? This is more general than a `description` field. It might also to eliminate the need for some extension type handling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3412167178

   > > > You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.
   > > 
   > > 
   > > Yes, one downside is this gets a little awkward because key value metadata for columns today is at a row-group level. which means for file level metadata one would need to define a serialization format for mapping column to description.
   > 
   > Could you add optional Key/Value Metadata to `SchemaElement`? This is more general than a `description` field. It might also to eliminate the need for some extension type handling.
   
   
   
   Hello. I'm not sure that would help. If you want the software industry to adopt it (and that's all the idea here), the specification really have to mention something like "field description" or "column description" that everyone will agree with.
   
   Best regards,
   
   Simon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "gaborcsardi (via GitHub)" <gi...@apache.org>.
gaborcsardi commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2380561558

   > Hello @gaborcsardi Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz.
   
   Why can't dataviz or etl software add the description as metadata. E.g. as a key called `description`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3411718908

   > You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.
   
   Yes, one downside is this gets a little awkward because key value metadata for columns today is at a row-group level.  which means for file level metadata one would need to define a serialization format for mapping column to description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "abellgithub (via GitHub)" <gi...@apache.org>.
abellgithub commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3411700116

   >> Why can't dataviz or etl software add the description as metadata? E.g. as a key called description?
   
   > Because it means asking the user of ETL a key name and the user of the dataviz the key name.
   
   You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2256030109

   Hello @gaborcsardi  Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "gaborcsardi (via GitHub)" <gi...@apache.org>.
gaborcsardi commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2255970779

   For every column, you can store arbitrary key-value metadata (string -> string), see at https://github.com/apache/parquet-format/blob/5a5c8948e60770f8a8356a8f5e616d5ae1079d4b/src/main/thrift/parquet.thrift#L809


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3815737921

   One of the big topic these last days is the Open Semantic Interchange... https://open-semantic-interchange.org/
   we could even think having the main fiields implemented by default in metadata.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2380620886

   @gaborcsardi Because it means asking the user of ETL a key name and the user of the dataviz the key name. This means another place (a file, a mail, an issue...) where the users have to share the information "hey guys, I put the field description in the key label". But since in most ETL or dataviz tool, you don't have the ability to retrieve metadata by key,  users of each tools have to ask the tool editor a way to retrieve a value by key description. The whole process isn't user friendly.
   
    Since it's a very common metadata and a lot of file format already have it, I think this is way better to have it in the file format specification. As of today, this is probably my main pain point with Parquet.
    
    Best regards,
    
    Simon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2516324150

   @simonaubertbd I don't think this exists in parquet.  I'm not against adding it but with the rise in popularity of table formats (which do include description) the utility might not be super high.  Is this something you would like to constribute?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-2516369956

   Hello @emkornfield  and thanks for the answer.
   1/I may disagree about the utility : Parquet has become in the last month a standard for data transfer just like csv (but way better), it's far to be just used on databases and it could even be used to replace proprietary file formats such as Qlik qvd, Alteryx yxdb (so less vendor locking and better interoperability). The only thing missing for that is the field description.
   
   2/I can contribute by testing and having some discussion on specs but I don't even know where to begin. I'm afraid to not have the sufficient skills to develop such a feature.
   
   Best regards,
   
   Simon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Column/field description [parquet-format]

Posted by "simonaubertbd (via GitHub)" <gi...@apache.org>.
simonaubertbd commented on issue #447:
URL: https://github.com/apache/parquet-format/issues/447#issuecomment-3161326273

   Hello guys,
   
   Any news about that feature?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org