You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by swizz one <so...@gmail.com> on 2020/05/01 16:12:34 UTC
Sending arrow via socket
Please, I am currently working on a project that require sending data from
an c program to perpespective(javascript) via socket from c. Since
perpespective works with arrow, it was a perfect choice. Is it possible to
send table at x interval from c to javascript via websocket without
creating a an arrow binary format file?
How to you create a memory pool primotive with c, without having to size to
have a fixed sized on the array like the memory pool in c++.
Your's Faithfully,
Thank you
Re: Sending arrow via socket
Posted by swizz one <so...@gmail.com>.
Is there any code in javascript to access the buffer?
Thank you for tour help
On Sun, May 3, 2020, 02:13 Sutou Kouhei <ko...@clear-code.com> wrote:
> Hi,
>
> >> Is it necessary to create a buffer to send data via socket, is, for
> >> example I create the schemas, do I have to create a buffer to store the
> >> data and then send he that data?,
>
> If your C program and JavaScript program run in the same
> process, you don't need to do it. You can just pass memory
> address.
>
> If your C program and JavaScript program run in the
> difference processes (I think so), you need to do it.
>
> >> Also is this method more effective compared to writing data to a arrow
> >> file and reading from that arrow file in the javascript program?
>
> If you use memory file system such as tmpfs in Linux for
> writing and reading the Apache Arrow file, it may be faster
> than sending data via socket.
>
> >> Finally, what are the option that are available for passing data from
> one
> >> program language to another using arrow?
>
> 1. Sending data via socket
> (It works with different processes and different hosts)
> 2. Writing data to a file and passing the file
> (It works with different processes)
> 3. Sending memory address (the same process)
> (It works with the same process)
>
> > And also, instead of sending the pointer, can you send the buffer
> directly?
>
> The example I introduced sends the buffer data not the
> pointer of the buffer data.
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
> "Re: Sending arrow via socket" on Sat, 2 May 2020 07:46:55 +0200,
> swizz one <so...@gmail.com> wrote:
>
> > And also, instead of sending the pointer, can you send the buffer
> directly?
> >
> > El sáb., 2 may. 2020 a las 7:07, swizz one (<so...@gmail.com>)
> escribió:
> >
> >> Thank you very much, Your answer has been really helpful. Can I ask a
> >> question?
> >> Is it necessary to create a buffer to send data via socket, is, for
> >> example I create the schemas, do I have to create a buffer to store the
> >> data and then send he that data?,
> >> Also is this method more effective compared to writing data to a arrow
> >> file and reading from that arrow file in the javascript program?
> >> Finally, what are the option that are available for passing data from
> one
> >> program language to another using arrow?
> >>
> >> Sorry, for the questions, I am a bit new to programming.
> >>
> >> El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<ko...@clear-code.com>)
> >> escribió:
> >>
> >>> Hi,
> >>>
> >>> Do you want to use Apache Arrow C GLib instead of Apache
> >>> Arrow C++, right?
> >>>
> >>> We don't provide a memory pool in Apache Arrow C GLib API.
> >>>
> >>> You can create a resizable buffer and output serialize data
> >>> to it:
> >>>
> >>> ---
> >>> GError *error = NULL;
> >>> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024,
> &error);
> >>> if (!buffer) {
> >>> g_print("error: %s\n", error->message);
> >>> g_error_free(error);
> >>> return;
> >>> }
> >>>
> >>> GArrowBufferOutputStream *output =
> >>> garrow_buffer_output_stream_new(buffer);
> >>>
> >>> GArrowRecordBatchStreamWriter *writer =
> >>> garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
> >>> schema, /* You need to create
> >>> this */
> >>> &error);
> >>> if (!writer) {
> >>> g_print("error: %s\n", error->message);
> >>> g_error_free(error);
> >>> g_object_unref(output);
> >>> g_object_unref(buffer);
> >>> return;
> >>> }
> >>>
> >>> if (!garrow_record_batch_writer_write_record_batch(
> >>> GARROW_RECORD_BATCH_WRITER(writer),
> >>> record_batch, /* You need to create this */
> >>> &error)) {
> >>> g_print("error: %s\n", error->message);
> >>> g_error_free(error);
> >>> g_object_unref(writer);
> >>> g_object_unref(output);
> >>> g_object_unref(buffer);
> >>> return;
> >>> }
> >>>
> >>> if (!garrow_record_batch_writer_close(
> >>> GARROW_RECORD_BATCH_WRITER(writer),
> >>> &error)) {
> >>> g_print("error: %s\n", error->message);
> >>> g_error_free(error);
> >>> g_object_unref(writer);
> >>> g_object_unref(output);
> >>> g_object_unref(buffer);
> >>> return;
> >>> }
> >>>
> >>> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
> >>> gsize data_size;
> >>> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
> >>> write(websocket_fd, data_raw, data_size);
> >>> g_bytes_unref(data);
> >>>
> >>> g_object_unref(writer);
> >>> g_object_unref(output);
> >>> g_object_unref(buffer);
> >>> ---
> >>>
> >>>
> >>> Thanks,
> >>> --
> >>> kou
> >>>
> >>> In <CACKQ_V3_yeuNn5TzPiF1RS=6WmwjL5jR4Vgx39GpQ3FvMgEqzg@mail.gmail.com
> >
> >>> "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
> >>> swizz one <so...@gmail.com> wrote:
> >>>
> >>> > Please, I am currently working on a project that require sending data
> >>> from
> >>> > an c program to perpespective(javascript) via socket from c. Since
> >>> > perpespective works with arrow, it was a perfect choice. Is it
> possible
> >>> to
> >>> > send table at x interval from c to javascript via websocket without
> >>> > creating a an arrow binary format file?
> >>> > How to you create a memory pool primotive with c, without having to
> >>> size to
> >>> > have a fixed sized on the array like the memory pool in c++.
> >>> >
> >>> > Your's Faithfully,
> >>> > Thank you
> >>>
> >>
>
Re: Sending arrow via socket
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
>> Is it necessary to create a buffer to send data via socket, is, for
>> example I create the schemas, do I have to create a buffer to store the
>> data and then send he that data?,
If your C program and JavaScript program run in the same
process, you don't need to do it. You can just pass memory
address.
If your C program and JavaScript program run in the
difference processes (I think so), you need to do it.
>> Also is this method more effective compared to writing data to a arrow
>> file and reading from that arrow file in the javascript program?
If you use memory file system such as tmpfs in Linux for
writing and reading the Apache Arrow file, it may be faster
than sending data via socket.
>> Finally, what are the option that are available for passing data from one
>> program language to another using arrow?
1. Sending data via socket
(It works with different processes and different hosts)
2. Writing data to a file and passing the file
(It works with different processes)
3. Sending memory address (the same process)
(It works with the same process)
> And also, instead of sending the pointer, can you send the buffer directly?
The example I introduced sends the buffer data not the
pointer of the buffer data.
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Re: Sending arrow via socket" on Sat, 2 May 2020 07:46:55 +0200,
swizz one <so...@gmail.com> wrote:
> And also, instead of sending the pointer, can you send the buffer directly?
>
> El sáb., 2 may. 2020 a las 7:07, swizz one (<so...@gmail.com>) escribió:
>
>> Thank you very much, Your answer has been really helpful. Can I ask a
>> question?
>> Is it necessary to create a buffer to send data via socket, is, for
>> example I create the schemas, do I have to create a buffer to store the
>> data and then send he that data?,
>> Also is this method more effective compared to writing data to a arrow
>> file and reading from that arrow file in the javascript program?
>> Finally, what are the option that are available for passing data from one
>> program language to another using arrow?
>>
>> Sorry, for the questions, I am a bit new to programming.
>>
>> El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<ko...@clear-code.com>)
>> escribió:
>>
>>> Hi,
>>>
>>> Do you want to use Apache Arrow C GLib instead of Apache
>>> Arrow C++, right?
>>>
>>> We don't provide a memory pool in Apache Arrow C GLib API.
>>>
>>> You can create a resizable buffer and output serialize data
>>> to it:
>>>
>>> ---
>>> GError *error = NULL;
>>> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
>>> if (!buffer) {
>>> g_print("error: %s\n", error->message);
>>> g_error_free(error);
>>> return;
>>> }
>>>
>>> GArrowBufferOutputStream *output =
>>> garrow_buffer_output_stream_new(buffer);
>>>
>>> GArrowRecordBatchStreamWriter *writer =
>>> garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
>>> schema, /* You need to create
>>> this */
>>> &error);
>>> if (!writer) {
>>> g_print("error: %s\n", error->message);
>>> g_error_free(error);
>>> g_object_unref(output);
>>> g_object_unref(buffer);
>>> return;
>>> }
>>>
>>> if (!garrow_record_batch_writer_write_record_batch(
>>> GARROW_RECORD_BATCH_WRITER(writer),
>>> record_batch, /* You need to create this */
>>> &error)) {
>>> g_print("error: %s\n", error->message);
>>> g_error_free(error);
>>> g_object_unref(writer);
>>> g_object_unref(output);
>>> g_object_unref(buffer);
>>> return;
>>> }
>>>
>>> if (!garrow_record_batch_writer_close(
>>> GARROW_RECORD_BATCH_WRITER(writer),
>>> &error)) {
>>> g_print("error: %s\n", error->message);
>>> g_error_free(error);
>>> g_object_unref(writer);
>>> g_object_unref(output);
>>> g_object_unref(buffer);
>>> return;
>>> }
>>>
>>> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
>>> gsize data_size;
>>> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
>>> write(websocket_fd, data_raw, data_size);
>>> g_bytes_unref(data);
>>>
>>> g_object_unref(writer);
>>> g_object_unref(output);
>>> g_object_unref(buffer);
>>> ---
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <CA...@mail.gmail.com>
>>> "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
>>> swizz one <so...@gmail.com> wrote:
>>>
>>> > Please, I am currently working on a project that require sending data
>>> from
>>> > an c program to perpespective(javascript) via socket from c. Since
>>> > perpespective works with arrow, it was a perfect choice. Is it possible
>>> to
>>> > send table at x interval from c to javascript via websocket without
>>> > creating a an arrow binary format file?
>>> > How to you create a memory pool primotive with c, without having to
>>> size to
>>> > have a fixed sized on the array like the memory pool in c++.
>>> >
>>> > Your's Faithfully,
>>> > Thank you
>>>
>>
Re: Sending arrow via socket
Posted by swizz one <so...@gmail.com>.
And also, instead of sending the pointer, can you send the buffer directly?
El sáb., 2 may. 2020 a las 7:07, swizz one (<so...@gmail.com>) escribió:
> Thank you very much, Your answer has been really helpful. Can I ask a
> question?
> Is it necessary to create a buffer to send data via socket, is, for
> example I create the schemas, do I have to create a buffer to store the
> data and then send he that data?,
> Also is this method more effective compared to writing data to a arrow
> file and reading from that arrow file in the javascript program?
> Finally, what are the option that are available for passing data from one
> program language to another using arrow?
>
> Sorry, for the questions, I am a bit new to programming.
>
> El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<ko...@clear-code.com>)
> escribió:
>
>> Hi,
>>
>> Do you want to use Apache Arrow C GLib instead of Apache
>> Arrow C++, right?
>>
>> We don't provide a memory pool in Apache Arrow C GLib API.
>>
>> You can create a resizable buffer and output serialize data
>> to it:
>>
>> ---
>> GError *error = NULL;
>> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
>> if (!buffer) {
>> g_print("error: %s\n", error->message);
>> g_error_free(error);
>> return;
>> }
>>
>> GArrowBufferOutputStream *output =
>> garrow_buffer_output_stream_new(buffer);
>>
>> GArrowRecordBatchStreamWriter *writer =
>> garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
>> schema, /* You need to create
>> this */
>> &error);
>> if (!writer) {
>> g_print("error: %s\n", error->message);
>> g_error_free(error);
>> g_object_unref(output);
>> g_object_unref(buffer);
>> return;
>> }
>>
>> if (!garrow_record_batch_writer_write_record_batch(
>> GARROW_RECORD_BATCH_WRITER(writer),
>> record_batch, /* You need to create this */
>> &error)) {
>> g_print("error: %s\n", error->message);
>> g_error_free(error);
>> g_object_unref(writer);
>> g_object_unref(output);
>> g_object_unref(buffer);
>> return;
>> }
>>
>> if (!garrow_record_batch_writer_close(
>> GARROW_RECORD_BATCH_WRITER(writer),
>> &error)) {
>> g_print("error: %s\n", error->message);
>> g_error_free(error);
>> g_object_unref(writer);
>> g_object_unref(output);
>> g_object_unref(buffer);
>> return;
>> }
>>
>> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
>> gsize data_size;
>> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
>> write(websocket_fd, data_raw, data_size);
>> g_bytes_unref(data);
>>
>> g_object_unref(writer);
>> g_object_unref(output);
>> g_object_unref(buffer);
>> ---
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In <CA...@mail.gmail.com>
>> "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
>> swizz one <so...@gmail.com> wrote:
>>
>> > Please, I am currently working on a project that require sending data
>> from
>> > an c program to perpespective(javascript) via socket from c. Since
>> > perpespective works with arrow, it was a perfect choice. Is it possible
>> to
>> > send table at x interval from c to javascript via websocket without
>> > creating a an arrow binary format file?
>> > How to you create a memory pool primotive with c, without having to
>> size to
>> > have a fixed sized on the array like the memory pool in c++.
>> >
>> > Your's Faithfully,
>> > Thank you
>>
>
Re: Sending arrow via socket
Posted by swizz one <so...@gmail.com>.
Thank you very much, Your answer has been really helpful. Can I ask a
question?
Is it necessary to create a buffer to send data via socket, is, for example
I create the schemas, do I have to create a buffer to store the data and
then send he that data?,
Also is this method more effective compared to writing data to a arrow file
and reading from that arrow file in the javascript program?
Finally, what are the option that are available for passing data from one
program language to another using arrow?
Sorry, for the questions, I am a bit new to programming.
El vie., 1 may. 2020 a las 23:30, Sutou Kouhei (<ko...@clear-code.com>)
escribió:
> Hi,
>
> Do you want to use Apache Arrow C GLib instead of Apache
> Arrow C++, right?
>
> We don't provide a memory pool in Apache Arrow C GLib API.
>
> You can create a resizable buffer and output serialize data
> to it:
>
> ---
> GError *error = NULL;
> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
> if (!buffer) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> return;
> }
>
> GArrowBufferOutputStream *output = garrow_buffer_output_stream_new(buffer);
>
> GArrowRecordBatchStreamWriter *writer =
> garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
> schema, /* You need to create this
> */
> &error);
> if (!writer) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> if (!garrow_record_batch_writer_write_record_batch(
> GARROW_RECORD_BATCH_WRITER(writer),
> record_batch, /* You need to create this */
> &error)) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> if (!garrow_record_batch_writer_close(
> GARROW_RECORD_BATCH_WRITER(writer),
> &error)) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
> gsize data_size;
> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
> write(websocket_fd, data_raw, data_size);
> g_bytes_unref(data);
>
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> ---
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
> "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
> swizz one <so...@gmail.com> wrote:
>
> > Please, I am currently working on a project that require sending data
> from
> > an c program to perpespective(javascript) via socket from c. Since
> > perpespective works with arrow, it was a perfect choice. Is it possible
> to
> > send table at x interval from c to javascript via websocket without
> > creating a an arrow binary format file?
> > How to you create a memory pool primotive with c, without having to size
> to
> > have a fixed sized on the array like the memory pool in c++.
> >
> > Your's Faithfully,
> > Thank you
>
Re: Sending arrow via socket
Posted by swizz one <so...@gmail.com>.
How do you read that buffer data from another process with a different
programming language? I am referring to the gconstpointer data type?
Thank you very much
On Fri, May 1, 2020, 23:30 Sutou Kouhei <ko...@clear-code.com> wrote:
> Hi,
>
> Do you want to use Apache Arrow C GLib instead of Apache
> Arrow C++, right?
>
> We don't provide a memory pool in Apache Arrow C GLib API.
>
> You can create a resizable buffer and output serialize data
> to it:
>
> ---
> GError *error = NULL;
> GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
> if (!buffer) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> return;
> }
>
> GArrowBufferOutputStream *output = garrow_buffer_output_stream_new(buffer);
>
> GArrowRecordBatchStreamWriter *writer =
> garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
> schema, /* You need to create this
> */
> &error);
> if (!writer) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> if (!garrow_record_batch_writer_write_record_batch(
> GARROW_RECORD_BATCH_WRITER(writer),
> record_batch, /* You need to create this */
> &error)) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> if (!garrow_record_batch_writer_close(
> GARROW_RECORD_BATCH_WRITER(writer),
> &error)) {
> g_print("error: %s\n", error->message);
> g_error_free(error);
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> return;
> }
>
> GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
> gsize data_size;
> gconstpointer data_raw = g_bytes_get_data(data, &data_size);
> write(websocket_fd, data_raw, data_size);
> g_bytes_unref(data);
>
> g_object_unref(writer);
> g_object_unref(output);
> g_object_unref(buffer);
> ---
>
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
> "Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
> swizz one <so...@gmail.com> wrote:
>
> > Please, I am currently working on a project that require sending data
> from
> > an c program to perpespective(javascript) via socket from c. Since
> > perpespective works with arrow, it was a perfect choice. Is it possible
> to
> > send table at x interval from c to javascript via websocket without
> > creating a an arrow binary format file?
> > How to you create a memory pool primotive with c, without having to size
> to
> > have a fixed sized on the array like the memory pool in c++.
> >
> > Your's Faithfully,
> > Thank you
>
Re: Sending arrow via socket
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
Do you want to use Apache Arrow C GLib instead of Apache
Arrow C++, right?
We don't provide a memory pool in Apache Arrow C GLib API.
You can create a resizable buffer and output serialize data
to it:
---
GError *error = NULL;
GArrowResizableBuffer *buffer = garrow_resizable_buffer_new(1024, &error);
if (!buffer) {
g_print("error: %s\n", error->message);
g_error_free(error);
return;
}
GArrowBufferOutputStream *output = garrow_buffer_output_stream_new(buffer);
GArrowRecordBatchStreamWriter *writer =
garrow_record_batch_stream_writer_new(GARROW_OUTPUT_STREAM(output),
schema, /* You need to create this */
&error);
if (!writer) {
g_print("error: %s\n", error->message);
g_error_free(error);
g_object_unref(output);
g_object_unref(buffer);
return;
}
if (!garrow_record_batch_writer_write_record_batch(
GARROW_RECORD_BATCH_WRITER(writer),
record_batch, /* You need to create this */
&error)) {
g_print("error: %s\n", error->message);
g_error_free(error);
g_object_unref(writer);
g_object_unref(output);
g_object_unref(buffer);
return;
}
if (!garrow_record_batch_writer_close(
GARROW_RECORD_BATCH_WRITER(writer),
&error)) {
g_print("error: %s\n", error->message);
g_error_free(error);
g_object_unref(writer);
g_object_unref(output);
g_object_unref(buffer);
return;
}
GBytes *data = garrow_buffer_get_data(GARROW_BUFFER(buffer));
gsize data_size;
gconstpointer data_raw = g_bytes_get_data(data, &data_size);
write(websocket_fd, data_raw, data_size);
g_bytes_unref(data);
g_object_unref(writer);
g_object_unref(output);
g_object_unref(buffer);
---
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Sending arrow via socket" on Fri, 1 May 2020 18:12:34 +0200,
swizz one <so...@gmail.com> wrote:
> Please, I am currently working on a project that require sending data from
> an c program to perpespective(javascript) via socket from c. Since
> perpespective works with arrow, it was a perfect choice. Is it possible to
> send table at x interval from c to javascript via websocket without
> creating a an arrow binary format file?
> How to you create a memory pool primotive with c, without having to size to
> have a fixed sized on the array like the memory pool in c++.
>
> Your's Faithfully,
> Thank you