You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/20 10:01:05 UTC

[GitHub] [arrow] mzhukovs opened a new issue #11186: Update column names or meta data "in-place" without reading in file contents

mzhukovs opened a new issue #11186:
URL: https://github.com/apache/arrow/issues/11186


   I would like to be able to do basic manipulation of parquet files stored in Azure Data Lake Storage (ADLS) Gen2 - is it possible to update ONLY the column names and/or metadata without having to recreate the file or read in all of the data?
   
   Using the handler provided by this library https://github.com/kaaveland/pyarrowfs-adlgen2 able to connect to storage just fine to read metadata/schema, but hoping there's a way to write back (by specifying the filesystem that the aforementioned lib enables easily). Is this possible?
   
   Looks like [rename_columns](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html?highlight=rename_columns#pyarrow.Table.rename_columns) is on the Table object, so data would have already been read in, and not sure about this one: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html?highlight=write_meta
   
   Would appreciate any insight.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on issue #11186: Update column names or meta data "in-place" without reading in file contents

Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #11186:
URL: https://github.com/apache/arrow/issues/11186#issuecomment-923208251


   No, not really, certainly not today at the pyarrow level.  You would need to have your data and metadata in separate files or otherwise introduce some kind of padding in between them.  Changing a column name or metadata in general is likely to change the size of the metadata block.  The current reader/writer packs them together and that would clobber existing data.
   
   In theory though, it might be possible.  I'm pretty sure (but not certain at all) the format allows for it.  Some parts of Arrow's C++ parquet library could potentially be reused but you would need to do quite a bit of novel development to get this I think.
   
   That being said, a potentially easier approach, which would handle column names and metadata, is to simply store an authoritative schema as a metadata-only standalone file (parquet or Arrow IPC).  Then, after reading in your data, you could create a table using your authoritative schema and the column data from the file(s) you read into memory.
   
   If you expand that concept much further you start to get into "schema evolution" and "metadata storage" concepts in something like Iceberg and so you may want to look at that project as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] mzhukovs commented on issue #11186: Update column names or meta data "in-place" without reading in file contents

Posted by GitBox <gi...@apache.org>.
mzhukovs commented on issue #11186:
URL: https://github.com/apache/arrow/issues/11186#issuecomment-923611036


   Alright thanks for the quick response.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] mzhukovs closed issue #11186: Update column names or meta data "in-place" without reading in file contents

Posted by GitBox <gi...@apache.org>.
mzhukovs closed issue #11186:
URL: https://github.com/apache/arrow/issues/11186


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org