You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ovidiu-Cristian MARCU <ov...@inria.fr> on 2018/05/02 09:53:30 UTC
plasma tutorial which version?
Hi,
Trying to follow the steps in https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with master branch does not work.
Could you please provide the arrow version that is compatible with this tutorial?
Thanks,
Ovidiu
Re: plasma tutorial which version?
Posted by Ovidiu-Cristian MARCU <ov...@inria.fr>.
I managed to make it work with the example listed below. I had to comment some includes unused to which I failed to link at compile time. I would appreciate some hints on how to create/compile/build clients better with arrow/plasma.
Thanks,
Ovidiu
+++++ create.cc ++++++
Compiled with: g++ create.cc `pkg-config --cflags --libs plasma arrow` --std=c++11 -o create
#include <assert.h>
#include <signal.h>
#include <stdlib.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <random>
//#include "arrow/test-util.h"
#include "plasma/client.h"
//#include "plasma/common.h"
//#include "plasma/plasma.h"
//#include "plasma/protocol.h"
using namespace plasma;
int main(int argc, char** argv) {
// Start up and connect a Plasma client.
PlasmaClient client;
ARROW_CHECK_OK(client.Connect("/tmp/plasma", "", PLASMA_DEFAULT_RELEASE_DELAY));
ObjectID object_id = ObjectID::from_random();
std::vector<uint8_t> data(100, 0);
uint8_t metadata[] = {5};
int64_t metadata_size = sizeof(metadata);
bool has_object;
std::shared_ptr<Buffer> data_buffer;
ARROW_CHECK_OK(client.Create(object_id, data.size(), &metadata[0], metadata_size,
&data_buffer));
for (size_t i = 0; i < data.size(); i++) {
data_buffer->mutable_data()[i] = data[i] + 1;
}
// Seal the object.
ARROW_CHECK_OK(client.Seal(object_id));
std::vector<ObjectID> object_ids = {object_id};
std::vector<ObjectBuffer> object_buffers;
ARROW_CHECK_OK(client.Get(object_ids, -1, &object_buffers));
fprintf(stdout, "%d\n", object_buffers[0].data->data()[0]);
ARROW_CHECK_OK(client.Contains(object_id, &has_object));
fprintf(stdout, "contains object:%d\n", has_object);
// Disconnect the client.
ARROW_CHECK_OK(client.Disconnect());
}
> On 02 May 2018, at 14:20, Ovidiu-Cristian MARCU <ov...@inria.fr> wrote:
>
> It seems the client Create signature changed (see below tutorial error). However, I’ve looked into client_tests and I see it passes ok all 9 tests. So I’m trying to modify the example to create an object. I still have some issues compiling, see below, maybe you can provide some help as I just started looking into apache arrow/plasma.
>
> I’m using create.cc <http://create.cc/>, however trying to compile fails with:
> g++ create.cc `pkg-config --cflags --libs plasma arrow` --std=c++11 -o create
> In file included from /usr/local/include/plasma/plasma.h:40:0,
> from create.cc:15:
> /usr/local/include/plasma/common_generated.h:7:37: fatal error: flatbuffers/flatbuffers.h: No such file or directory
> #include "flatbuffers/flatbuffers.h”
>
> Before, I managed to make && make install after
> cmake .. -DCMAKE_BUILD_TYPE=Release -DARROW_PLASMA=on
> and run
> /release/plasma_store -m 1000000000 -s /tmp/plasma
> Allowing the Plasma store to use up to 1GB of memory.
> Starting object store with directory /dev/shm and huge page support disabled
>
> ++++ create object example modified ++++
> create.cc
> #include <assert.h>
> #include <signal.h>
> #include <stdlib.h>
> #include <sys/time.h>
> #include <sys/types.h>
> #include <unistd.h>
>
> #include <random>
>
> #include "plasma/client.h"
> #include "plasma/common.h"
> #include "plasma/plasma.h"
> #include "plasma/protocol.h"
>
> using namespace plasma;
>
> int main(int argc, char** argv) {
> // Start up and connect a Plasma client.
> PlasmaClient client;
> ARROW_CHECK_OK(client.Connect("/tmp/plasma", "", PLASMA_DEFAULT_RELEASE_DELAY));
>
> ObjectID object_id = ObjectID::from_random();
> std::vector<uint8_t> data(100, 0);
> uint8_t metadata[] = {1};
> int64_t metadata_size = sizeof(metadata);
>
> std::shared_ptr<Buffer> data_buffer;
> ARROW_CHECK_OK(client.Create(object_id, data.size(), &metadata[0], metadata_size,
> &data_buffer));
> for (size_t i = 0; i < data.size(); i++) {
> data_buffer->mutable_data()[i] = data[i];
> }
>
> // Seal the object.
> ARROW_CHECK_OK(client.Seal(object_id));
> // Disconnect the client.
> ARROW_CHECK_OK(client.Disconnect());
> }
>
> +++++++++ tutorial error +++++++
> with https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> create.cc
> g++ create2.cc `pkg-config --cflags --libs plasma` --std=c++11 -o create
> In file included from /usr/local/include/plasma/common.h:32:0,
> from /usr/local/include/plasma/client.h:30,
> from create2.cc:1:
> create2.cc: In function ‘int main(int, char**)’:
> create2.cc:12:104: error: no matching function for call to ‘plasma::PlasmaClient::Create(plasma::ObjectID&, int64_t&, uint8_t*, std::basic_string<char>::size_type, uint8_t**)’
> ARROW_CHECK_OK(client.Create(object_id, data_size, (uint8_t*) metadata.data(), metadata.size(), &data));
> ^
> create2.cc:12:104: note: candidate is:
> In file included from create2.cc:1:0:
> /usr/local/include/plasma/client.h:92:10: note: arrow::Status plasma::PlasmaClient::Create(const ObjectID&, int64_t, const uint8_t*, int64_t, std::shared_ptr<arrow::Buffer>*, int)
> Status Create(const ObjectID& object_id, int64_t data_size, const uint8_t* metadata,
> ^
> /usr/local/include/plasma/client.h:92:10: note: no known conversion for argument 5 from ‘uint8_t** {aka unsigned char**}’ to ‘std::shared_ptr<arrow::Buffer>*’
>
> Thanks,
> Ovidiu
>
>> On 02 May 2018, at 14:08, Uwe L. Korn <uw...@xhochy.com> wrote:
>>
>> Hello Ovidiu,
>>
>> we actually would like to keep the tutorial in line with the API we have in master. Can you post the errors you're getting?
>>
>> This would help us in understanding what is outdated in the steps.
>>
>> Uwe
>>
>> On Wed, May 2, 2018, at 11:53 AM, Ovidiu-Cristian MARCU wrote:
>>> Hi,
>>>
>>> Trying to follow the steps in
>>> https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html
>>> <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with master
>>> branch does not work.
>>> Could you please provide the arrow version that is compatible with this
>>> tutorial?
>>>
>>> Thanks,
>>> Ovidiu
>
Re: plasma tutorial which version?
Posted by Ovidiu-Cristian MARCU <ov...@inria.fr>.
It seems the client Create signature changed (see below tutorial error). However, I’ve looked into client_tests and I see it passes ok all 9 tests. So I’m trying to modify the example to create an object. I still have some issues compiling, see below, maybe you can provide some help as I just started looking into apache arrow/plasma.
I’m using create.cc <http://create.cc/>, however trying to compile fails with:
g++ create.cc `pkg-config --cflags --libs plasma arrow` --std=c++11 -o create
In file included from /usr/local/include/plasma/plasma.h:40:0,
from create.cc:15:
/usr/local/include/plasma/common_generated.h:7:37: fatal error: flatbuffers/flatbuffers.h: No such file or directory
#include "flatbuffers/flatbuffers.h”
Before, I managed to make && make install after
cmake .. -DCMAKE_BUILD_TYPE=Release -DARROW_PLASMA=on
and run
/release/plasma_store -m 1000000000 -s /tmp/plasma
Allowing the Plasma store to use up to 1GB of memory.
Starting object store with directory /dev/shm and huge page support disabled
++++ create object example modified ++++
create.cc
#include <assert.h>
#include <signal.h>
#include <stdlib.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#include <random>
#include "plasma/client.h"
#include "plasma/common.h"
#include "plasma/plasma.h"
#include "plasma/protocol.h"
using namespace plasma;
int main(int argc, char** argv) {
// Start up and connect a Plasma client.
PlasmaClient client;
ARROW_CHECK_OK(client.Connect("/tmp/plasma", "", PLASMA_DEFAULT_RELEASE_DELAY));
ObjectID object_id = ObjectID::from_random();
std::vector<uint8_t> data(100, 0);
uint8_t metadata[] = {1};
int64_t metadata_size = sizeof(metadata);
std::shared_ptr<Buffer> data_buffer;
ARROW_CHECK_OK(client.Create(object_id, data.size(), &metadata[0], metadata_size,
&data_buffer));
for (size_t i = 0; i < data.size(); i++) {
data_buffer->mutable_data()[i] = data[i];
}
// Seal the object.
ARROW_CHECK_OK(client.Seal(object_id));
// Disconnect the client.
ARROW_CHECK_OK(client.Disconnect());
}
+++++++++ tutorial error +++++++
with https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> create.cc
g++ create2.cc `pkg-config --cflags --libs plasma` --std=c++11 -o create
In file included from /usr/local/include/plasma/common.h:32:0,
from /usr/local/include/plasma/client.h:30,
from create2.cc:1:
create2.cc: In function ‘int main(int, char**)’:
create2.cc:12:104: error: no matching function for call to ‘plasma::PlasmaClient::Create(plasma::ObjectID&, int64_t&, uint8_t*, std::basic_string<char>::size_type, uint8_t**)’
ARROW_CHECK_OK(client.Create(object_id, data_size, (uint8_t*) metadata.data(), metadata.size(), &data));
^
create2.cc:12:104: note: candidate is:
In file included from create2.cc:1:0:
/usr/local/include/plasma/client.h:92:10: note: arrow::Status plasma::PlasmaClient::Create(const ObjectID&, int64_t, const uint8_t*, int64_t, std::shared_ptr<arrow::Buffer>*, int)
Status Create(const ObjectID& object_id, int64_t data_size, const uint8_t* metadata,
^
/usr/local/include/plasma/client.h:92:10: note: no known conversion for argument 5 from ‘uint8_t** {aka unsigned char**}’ to ‘std::shared_ptr<arrow::Buffer>*’
Thanks,
Ovidiu
> On 02 May 2018, at 14:08, Uwe L. Korn <uw...@xhochy.com> wrote:
>
> Hello Ovidiu,
>
> we actually would like to keep the tutorial in line with the API we have in master. Can you post the errors you're getting?
>
> This would help us in understanding what is outdated in the steps.
>
> Uwe
>
> On Wed, May 2, 2018, at 11:53 AM, Ovidiu-Cristian MARCU wrote:
>> Hi,
>>
>> Trying to follow the steps in
>> https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html
>> <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with master
>> branch does not work.
>> Could you please provide the arrow version that is compatible with this
>> tutorial?
>>
>> Thanks,
>> Ovidiu
Re: plasma tutorial which version?
Posted by Antoine Pitrou <an...@python.org>.
Hi Kendall,
Le 09/05/2018 à 03:02, Kendall Willets a écrit :
> This reminds me -- I had similar trouble with the sample code on the C++
> tutorial (
> https://arrow.apache.org/docs/cpp/md_tutorials_row_wise_conversion.html),
> which appears to be out of date. Should I submit a PR for the doc, or open
> a jira, etc.? I also have a buildable sample.
You should open a JIRA and submit a PR, yes.
Regards
Antoine.
>
> On Wed, May 2, 2018 at 5:08 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
>
>> Hello Ovidiu,
>>
>> we actually would like to keep the tutorial in line with the API we have
>> in master. Can you post the errors you're getting?
>>
>> This would help us in understanding what is outdated in the steps.
>>
>> Uwe
>>
>> On Wed, May 2, 2018, at 11:53 AM, Ovidiu-Cristian MARCU wrote:
>>> Hi,
>>>
>>> Trying to follow the steps in
>>> https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html
>>> <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with
>> master
>>> branch does not work.
>>> Could you please provide the arrow version that is compatible with this
>>> tutorial?
>>>
>>> Thanks,
>>> Ovidiu
>>
>
Re: plasma tutorial which version?
Posted by Kendall Willets <ke...@willets.org>.
This reminds me -- I had similar trouble with the sample code on the C++
tutorial (
https://arrow.apache.org/docs/cpp/md_tutorials_row_wise_conversion.html),
which appears to be out of date. Should I submit a PR for the doc, or open
a jira, etc.? I also have a buildable sample.
On Wed, May 2, 2018 at 5:08 AM, Uwe L. Korn <uw...@xhochy.com> wrote:
> Hello Ovidiu,
>
> we actually would like to keep the tutorial in line with the API we have
> in master. Can you post the errors you're getting?
>
> This would help us in understanding what is outdated in the steps.
>
> Uwe
>
> On Wed, May 2, 2018, at 11:53 AM, Ovidiu-Cristian MARCU wrote:
> > Hi,
> >
> > Trying to follow the steps in
> > https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html
> > <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with
> master
> > branch does not work.
> > Could you please provide the arrow version that is compatible with this
> > tutorial?
> >
> > Thanks,
> > Ovidiu
>
Re: plasma tutorial which version?
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello Ovidiu,
we actually would like to keep the tutorial in line with the API we have in master. Can you post the errors you're getting?
This would help us in understanding what is outdated in the steps.
Uwe
On Wed, May 2, 2018, at 11:53 AM, Ovidiu-Cristian MARCU wrote:
> Hi,
>
> Trying to follow the steps in
> https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html
> <https://arrow.apache.org/docs/cpp/md_tutorials_plasma.html> with master
> branch does not work.
> Could you please provide the arrow version that is compatible with this
> tutorial?
>
> Thanks,
> Ovidiu