You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Shannon Duncan <jo...@liveramp.com> on 2019/09/27 14:48:19 UTC
Avro SortedKeyValueFile
Does beam support the avro SortedKeyValueFile
<https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/hadoop/file/SortedKeyValueFile.html>?
Not able to find any documentation in beam area on this.
I need to be able to use either that or a MapFile for SequenceFiles to read
data from it's index and dig into those files at the exact byte offsets.
Also being able to write out files that are MapFile or SortedKeyValueFile
would be handy for creating the indexes and sorted files.
Thanks,
Shannon Duncan
Re: Avro SortedKeyValueFile
Posted by Shannon Duncan <jo...@liveramp.com>.
That was my fear. Will think on how to implement this correctly.
On Fri, Sep 27, 2019 at 12:00 PM Lukasz Cwik <lc...@google.com> wrote:
> Not to my knowledge.
>
> On Fri, Sep 27, 2019 at 7:48 AM Shannon Duncan <jo...@liveramp.com>
> wrote:
>
>> Does beam support the avro SortedKeyValueFile
>> <https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/hadoop/file/SortedKeyValueFile.html>?
>> Not able to find any documentation in beam area on this.
>>
>> I need to be able to use either that or a MapFile for SequenceFiles to
>> read data from it's index and dig into those files at the exact byte
>> offsets. Also being able to write out files that are MapFile or
>> SortedKeyValueFile would be handy for creating the indexes and sorted files.
>>
>> Thanks,
>> Shannon Duncan
>>
>
Re: Avro SortedKeyValueFile
Posted by Lukasz Cwik <lc...@google.com>.
Not to my knowledge.
On Fri, Sep 27, 2019 at 7:48 AM Shannon Duncan <jo...@liveramp.com>
wrote:
> Does beam support the avro SortedKeyValueFile
> <https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/hadoop/file/SortedKeyValueFile.html>?
> Not able to find any documentation in beam area on this.
>
> I need to be able to use either that or a MapFile for SequenceFiles to
> read data from it's index and dig into those files at the exact byte
> offsets. Also being able to write out files that are MapFile or
> SortedKeyValueFile would be handy for creating the indexes and sorted files.
>
> Thanks,
> Shannon Duncan
>