You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "shanjixi (via GitHub)" <gi...@apache.org> on 2023/07/11 04:19:47 UTC
[GitHub] [arrow] shanjixi opened a new issue, #36608: Support reading Hadoop-snappy File Format Directly~
shanjixi opened a new issue, #36608:
URL: https://github.com/apache/arrow/issues/36608
### Describe the enhancement requested
Hadoop-snappy File Format is wide used for BigData and ML frameworks; We ues spark or hadoop mr to preprocess data for furture processing such like Data warehouse and DeepLearning traing;
Hadoop-snappy File is A compressed file consists of one or more blocks. A block consists of uncompressed length (big endian 4 byte integer) and one or more subblocks.
Arrow could read a file which is fully compressed with Snappy Codec but fails to read data;
If arrow can read the Hadoop-snappy File consisted of a serials of chunks, it will greatly wide the Arrow's using scenarios;
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1647573876
Ok, see https://github.com/apache/arrow/blob/main/cpp/src/arrow/io/compressed.h
You could pretty easily add a `HadoopCompressedInputStream` there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1641572666
(I'm horrified that there is even a Hadoop-ZSTD)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1641570876
@shanjixi I'm curious, what does Arrow bring to the table if the file format (TFR) is not supported by Arrow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly~
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1633682100
Seems it's just `SnappyCodec`? Can you just try the:
1. https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/compression.h
2. or https://arrow.apache.org/docs/python/generated/pyarrow.compress.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Support reading Hadoop-snappy File Format Directly [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-3921771486
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1634261236
Are you reading CSV files? If not, then I am not sure that a streaming view will be very helpful. We generally rely on random access to metadata. This allows us to implement things like column skipping (projection pushdown). This is why formats like the arrow format and the parquet format support buffer-compression (instead of whole-file compression). It allows the metadata to stay uncompressed.
When you talk about blocks are you talking about the snappy framing format? https://github.com/google/snappy/blob/main/framing_format.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shanjixi commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "shanjixi (via GitHub)" <gi...@apache.org>.
shanjixi commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1645368364
> supported
TFRecord is just like a file contains multi-rows；
In practice，we use a converter to transform TFRecord file (compressed) into arrow::Table Type. Then we apply Arrow filter/take/list_flatten functions before we send the Data to Tensorflow worker;
Arrow is used to read data from HDFS, make further processing and avoid memroy copy cost between these two steps;
What's more, within our company both Hadoop-ZSTD and Hadoop-SNAP are supported to be read with Arrow directly( we call them as zstd_decompress_inputstream and snappy_decompress_inputstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly~
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1632784456
Yes, I think we will need a link or more information. It is not clear to me if this is a new compression codec (some kind of block-based snappy) or file format or some change in the hdfs filesystem implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shanjixi commented on issue #36608: Support reading Hadoop-snappy File Format Directly~
Posted by "shanjixi (via GitHub)" <gi...@apache.org>.
shanjixi commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1632230759
> Yes, I mean enable Arrow reading **hadoop-snappy** file format directly, just the same as kubo/snip's behavior.
> Would you mind provide something about "hadoop-snappy"? Do you mean https://github.com/kubo/snzip#hadoop-snappy-format ?
> By the way, Arrow already has `snappy` codec support and `HDFS` datasource support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shanjixi commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly~
Posted by "shanjixi (via GitHub)" <gi...@apache.org>.
shanjixi commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1633675085
> some change in the hdfs filesystem implementation.
In Hadoop common it's named as BlockDecompressorStream with SnappyCodeC
Here is the link for hadoop-snappy: https://code.google.com/archive/p/hadoop-snappy/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shanjixi commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "shanjixi (via GitHub)" <gi...@apache.org>.
shanjixi commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1633795635
> Seems it's just `SnappyCodec`? Can you just try the:
>
> 1. https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/compression.h
> 2. or https://arrow.apache.org/docs/python/generated/pyarrow.compress.html
This is not block based compress/unpresss support. If we read a hadoop-snappy files, we need to uncompress blocks one by one, which is not convinenet enought for high level components;
If we read blocks from one Haddoop-Snappy file, definitely we do need to uncompress one block with the help of "arrow/util/compression.h"; However we still need to wright extra code to provide a StreamingView for a serial of uncompressed snappy blocks for hight level component.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] shanjixi commented on issue #36608: [C++] Support reading Hadoop-snappy File Format Directly
Posted by "shanjixi (via GitHub)" <gi...@apache.org>.
shanjixi commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1635232682
> When you talk about blocks are you talking about the snappy framing format? https://github.com/google/snappy/blob/main/framing_format.txt
NOT CSV files. We are reading some kind of TFRecod file which is commpressed into Hadoop-Snappy format such like training_instance_0001.tfrecord.snappy
There are 3 kind of general snappy-compressed file (not the parquet use snappy internally) , and this issue is about "hadoop-snappy file"
1.snappy whole file
2.snappy framing (google, rarely used.)
3.hadoop-snappy file (whichi is whide useed in big-data ecosystem)
**2.** is different from **3.** we can treat **3.** as a kind of general blocked based compressed format( the codec for the blocks could able replaced by ZSTD, gzip and so on;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #36608: Support reading Hadoop-snappy File Format Directly~
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #36608:
URL: https://github.com/apache/arrow/issues/36608#issuecomment-1631310642
Would you mind provide something about "hadoop-snappy"? Do you mean https://github.com/kubo/snzip#hadoop-snappy-format ?
By the way, Arrow already has `snappy` codec support and `HDFS` datasource support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org