You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by L Ait <lh...@gmail.com> on 2022/07/02 18:12:43 UTC

cpp arrow manage stream with incomplete records

Hi,
I need help to integrate arrow cpp in an open source storage project. In
fact I built cpp library and can call api.

I have a c++ code that reads data by chunks then uses some erasure code to
rebuild original data.
The rebuild is done in chunks , At each iteration I can access a buffer of
rebuilt data.
My need is to pass this data as a stream to arrow process then send the
processed stream.

For example if my original file is a csv and I would like to filter and
save first column:

col1,col2, col3, col3
a1,b1,c1,d1
an,bn,cn,dn

split to 6 chunks of equal sizes chunk1:


chunk1:
a1,b1,c1,d1
ak,bk
chunk2:

ck,dk
...
am,bm,cm,dm

and so on.

My question is how to use the right StreamReader  in arrow and how this
deals with incomplete records( lines)  at the beginning and end of each
chunk ?

Here a snippet of code I use :
buffer_type_t res = fut.get0();
BOOST_LOG_TRIVIAL(trace) <<
"RawxBackendReader: Got result with buffer size: " << res.size();
std::shared_ptr<arrow::io::InputStream> input;

std::shared_ptr<arrow::io::BufferReader> buffer(new arrow::io::BufferReader(
reinterpret_cast<const uint8_t*>(res.get()), res.size()));
input = buffer;
BOOST_LOG_TRIVIAL(trace) << "laa type input" << input.get();

ArrowFilter arrow_filter = ArrowFilter(input);
arrow_filter.ToCsv();


result.push_back(std::move(res));

Thank you

Re: cpp arrow manage stream with incomplete records

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

How about using arrow::ipc::StreamDecoder instead of
arrow::ipc::StreamReader?

  class MyListener : public arrow::ipc::Listener {
  public:
    arrow::Status
    OnRecordBatchDecoded(std::shared_ptr<arrow::RecordBatch> record_batch)
    override {
      ArrowFilter arrow_filter = ArrowFilter(record_batch);
       arrow_filter.ToCsv();
    }
  }

  auto listener = std::make_shared<MyListener>();
  arrow::ipc::StreamDecoder decoder(listener);

  while (true) {
    buffer_type_t res = fut.get0();
    if (res.size() == 0) {
      break;
    }
    decoder.Consume(reinterpret_cast<const uint8_t*>(res.get()),
                    res.size());
  }


See also:

* arrow::ipc::StreamDecoder:
  https://arrow.apache.org/docs/cpp/api/ipc.html#classarrow_1_1ipc_1_1_stream_decoder

* Event-driven reading:
  https://arrow.apache.org/docs/cpp/ipc.html#event-driven-reading

* A real world usage:
  https://github.com/groonga/groonga/blob/master/lib/arrow.cpp#L1627


Thanks,
-- 
kou

In <CA...@mail.gmail.com>
  "cpp arrow manage stream with incomplete records" on Sat, 2 Jul 2022 20:12:43 +0200,
  L Ait <lh...@gmail.com> wrote:

> Hi,
> I need help to integrate arrow cpp in an open source storage project. In
> fact I built cpp library and can call api.
> 
> I have a c++ code that reads data by chunks then uses some erasure code to
> rebuild original data.
> The rebuild is done in chunks , At each iteration I can access a buffer of
> rebuilt data.
> My need is to pass this data as a stream to arrow process then send the
> processed stream.
> 
> For example if my original file is a csv and I would like to filter and
> save first column:
> 
> col1,col2, col3, col3
> a1,b1,c1,d1
> an,bn,cn,dn
> 
> split to 6 chunks of equal sizes chunk1:
> 
> 
> chunk1:
> a1,b1,c1,d1
> ak,bk
> chunk2:
> 
> ck,dk
> ...
> am,bm,cm,dm
> 
> and so on.
> 
> My question is how to use the right StreamReader  in arrow and how this
> deals with incomplete records( lines)  at the beginning and end of each
> chunk ?
> 
> Here a snippet of code I use :
> buffer_type_t res = fut.get0();
> BOOST_LOG_TRIVIAL(trace) <<
> "RawxBackendReader: Got result with buffer size: " << res.size();
> std::shared_ptr<arrow::io::InputStream> input;
> 
> std::shared_ptr<arrow::io::BufferReader> buffer(new arrow::io::BufferReader(
> reinterpret_cast<const uint8_t*>(res.get()), res.size()));
> input = buffer;
> BOOST_LOG_TRIVIAL(trace) << "laa type input" << input.get();
> 
> ArrowFilter arrow_filter = ArrowFilter(input);
> arrow_filter.ToCsv();
> 
> 
> result.push_back(std::move(res));
> 
> Thank you