You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by John Mora <jh...@gmail.com> on 2019/06/08 20:26:40 UTC
Kudu datastore reports
Hi all.
I have just updated my weekly reports on Cwiki [1]. This next week I think
I should be focusing on the create schema operation and solving the issue
of the partitioning configurations in the mapping file.
Please let me know if you have suggestions, my last commits are available
here [2]
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Best,
John
Re: Kudu datastore reports
Posted by Kevin Ratnasekera <dj...@gmail.com>.
Hi John,
Thank you for the Clarification. If the current implementation has no
issues with build in those environments, I think there is *no* need to
invest time in other approaches Eg:- Docker. Simply just do build test
whether tests pass for those operating systems.
Regards
Kevin
On Wed, Aug 7, 2019 at 3:09 AM John Mora <jh...@gmail.com> wrote:
> Hi Kevin.
>
> KuduTestHarness, theoretically should detect environment through this
> plugin "os-maven-plugin" and download the corresponding kudu binaries [1],
> and it have worked fine for me.
>
> Nonetheless, docker is a good idea. I will give a look to testcontainers
> and docker.
>
> [1]
> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>
> Regards,
> John
>
> El lun., 5 ago. 2019 a las 23:58, Kevin Ratnasekera (<
> djkevincr1989@gmail.com>) escribió:
>
> > Hi John,
> >
> > Can't we spin up Kudu docker [1] instance for testing purposes? We have
> > used Test containers [2] some data stores like couch DB. Gora build
> should
> > work in both Linux and Non Linux environments. Eg:- Windows. Is
> classifier
> > [3] depend on the environment the build is running?
> >
> > Kudu is based on C/ C++, so to spin up a server instance, we need check a
> > approach like docker, using such approach allow us to avoid these OS,
> > dependency related stuff come in to play in builds.
> >
> > [1] https://hub.docker.com/r/usuresearch/apache-kudu/
> > [2] https://www.testcontainers.org/
> > [3] <classifier>linux-x86_64</classifier>
> >
> > Regards
> > Kevin
> >
> > On Tue, Aug 6, 2019 at 9:56 AM John Mora <jh...@gmail.com> wrote:
> >
> > > Hi Alfonso,
> > >
> > > Unfortunately, I have not been able to reproduce the issue. Maybe it is
> > > related with my Java version (Oracle), I will try with OpenJDK.
> > > Some details about my development environment:
> > >
> > > os.detected.name: linux
> > > os.detected.arch: x86_64
> > > os.detected.version: 4.10
> > > os.detected.version.major: 4
> > > os.detected.version.minor: 10
> > > os.detected.release: linuxmint
> > > os.detected.release.version: 18.3
> > > os.detected.release.like.linuxmint: true
> > > os.detected.release.like.ubuntu: true
> > > os.detected.classifier: linux-x86_64
> > >
> > > Java
> > > java version "1.8.0_171"
> > > Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
> > >
> > > Maven
> > > Apache Maven 3.3.9
> > > Maven home: /usr/share/maven
> > > Java version: 1.8.0_171, vendor: Oracle Corporation
> > > Java home: /usr/lib/jvm/java-8-oracle/jre
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
> > > "unix"
> > >
> > >
> > > Best,
> > > John.
> > >
> > > El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
> > > alfonso.nishikawa@gmail.com>) escribió:
> > >
> > >> Hi,
> > >>
> > >> I am using now the following pom configuration I got from executing
> `mvn
> > >> dependency:tree`:
> > >>
> > >> <dependency>
> > >> <groupId>org.apache.kudu</groupId>
> > >> <artifactId>kudu-binary</artifactId>
> > >> <classifier>linux-x86_64</classifier>
> > >> <version>1.9.0</version>
> > >> <scope>test</scope>
> > >> </dependency>
> > >>
> > >> When I execute `mvn clen package` on gora-kudu I find that it spawns
> the
> > >> following command:
> > >>
> > >> kudu-master
> > >> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
> > >> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
> > >> --block_manager=log --webserver_interface=localhost
> > --ipki_ca_key_size=1024
> > >> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
> > >> --webserver_interface=*127.26.116.190* --webserver_port=0
> --never_fsync
> > >> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none
> > >> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
> > >> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
> > >>
> >
> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
> > >> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
> > >> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
> > >> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
> > >> *127.26.116.188*:42253
> > >>
> > >>
> > >> I highlight the IP addresses because they clearly are not my computer,
> > >> and I guess that is why the tests can't connect to the the database.
> > >>
> > >> Any idea on how to solve this?
> > >>
> > >> Thank you!
> > >>
> > >>
> > >> Best Regards,
> > >>
> > >> Alfonso Nishikawa
> > >>
> > >>
> > >>
> > >> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
> > >> alfonso.nishikawa@gmail.com>) escribió:
> > >>
> > >>> Hi, John.
> > >>>
> > >>> I get a core dump from the binary kudu server when trying to run the
> > >>> tests. Didn't find a log file, but will search thoroughly later.
> > Happened
> > >>> anytime to you? Does it happens to anyone?
> > >>>
> > >>> I am using Ubuntu 18.04
> > >>>
> > >>> Thank you!
> > >>>
> > >>> Regards,
> > >>>
> > >>> Alfonso Nishikawa
> > >>>
> > >>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
> > >>> escribió:
> > >>>
> > >>>> Hi John,
> > >>>>
> > >>>> I've already made my comments at your PR. Please check them
> carefully
> > >>>> and ask me if you need help.
> > >>>>
> > >>>> For the documentation, I've checked what you've done. On the other
> > >>>> hand, I would want to encourage you to write a blog post about your
> > Kudu
> > >>>> implementation and demonstrate an example of Kudu integration with
> > Gora as
> > >>>> like a tutorial.
> > >>>>
> > >>>> Kind Regards,
> > >>>> Furkan KAMACI
> > >>>>
> > >>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Hi all.
> > >>>>>
> > >>>>> I have updated my report in the Wiki[1].
> > >>>>>
> > >>>>> Also, I have sent a PR with my last commits for review [2]. Please
> > >>>>> give it a look if you have time.
> > >>>>>
> > >>>>> This week, I will continue working on the documentation of the kudu
> > >>>>> datastore.
> > >>>>>
> > >>>>> Please let me know if you have suggestions.
> > >>>>>
> > >>>>> [1]
> > >>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>> [2] https://github.com/apache/gora/pull/178
> > >>>>>
> > >>>>> Best,
> > >>>>> John.
> > >>>>>
> > >>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<
> > carlosrmng@gmail.com>)
> > >>>>> escribió:
> > >>>>>
> > >>>>>> Hi John,
> > >>>>>>
> > >>>>>> Thanks for the update. I reviewed your code a little bit, it is
> > >>>>>> looking good. I think tha you should send a PR in order to receive
> > feedback
> > >>>>>> from other community members.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Carlos
> > >>>>>>
> > >>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<
> jhnmora000@gmail.com
> > >)
> > >>>>>> escribió:
> > >>>>>>
> > >>>>>>> Hi all.
> > >>>>>>>
> > >>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
> commits
> > >>>>>>> to my branch [2]. Please give it a look if you have time.
> > >>>>>>>
> > >>>>>>> This week, I will give a look to the documentation of datastores.
> > >>>>>>>
> > >>>>>>> Please let me know if you have suggestions.
> > >>>>>>>
> > >>>>>>> [1]
> > >>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> John
> > >>>>>>>
> > >>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<
> > jhnmora000@gmail.com>)
> > >>>>>>> escribió:
> > >>>>>>>
> > >>>>>>>> Hi Alfonso,
> > >>>>>>>>
> > >>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a
> problem
> > I
> > >>>>>>>> will find an alternative, it is only an utilitary class.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> John
> > >>>>>>>>
> > >>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
> > >>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> > >>>>>>>>
> > >>>>>>>>> Hi, John.
> > >>>>>>>>>
> > >>>>>>>>> I checked out your code and it looks good :)
> > >>>>>>>>> I found that you use javafx, but that is not present in OpenJDK
> > >>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I
> > would
> > >>>>>>>>> suggest to change it.
> > >>>>>>>>>
> > >>>>>>>>> Good job, keep it going :)
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>>
> > >>>>>>>>> Alfonso Nishikawa
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
> > >>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>
> > >>>>>>>>>> Hi.
> > >>>>>>>>>>
> > >>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
> > >>>>>>>>>> commits to my branch [2]. Please give it a look if you have
> > time.
> > >>>>>>>>>>
> > >>>>>>>>>> This week, I will give a look to the map reduce tests for
> > >>>>>>>>>> DataStores.
> > >>>>>>>>>>
> > >>>>>>>>>> Please let me know if you have suggestions.
> > >>>>>>>>>>
> > >>>>>>>>>> [1]
> > >>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> John
> > >>>>>>>>>>
> > >>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
> > >>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi all
> > >>>>>>>>>>>
> > >>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
> > >>>>>>>>>>> commits to my branch [2]. Please give it a look if you have
> > time.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This week, I will be working in the getPartitions and
> > >>>>>>>>>>> deleteByQuery methods and testing the other tests in the
> > DataStoreTestBase
> > >>>>>>>>>>> class.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Please let me know if you have suggestions.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [1]
> > >>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> John.
> > >>>>>>>>>>>
> > >>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
> > >>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Alfonso,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks so much for your time and support for this project. I
> > >>>>>>>>>>>> will work on your comments. Responses inline :)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
> > >>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, John.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been
> very
> > >>>>>>>>>>>>> busy :( I will try to answer your questions :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *> In the Employee example there is a field called
> > >>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the
> > UNIXTIME_MICROS datatype
> > >>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However,
> in
> > the java world
> > >>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is
> > a Timestamp.
> > >>>>>>>>>>>>> So, I was wondering whether I should force the usage of the
> > UNIXTIME_MICROS
> > >>>>>>>>>>>>> datatype for this field or just use a LONG datatype in
> Kudu.*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
> > >>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first
> > time I read about
> > >>>>>>>>>>>>> because until the last version upgrade of Avro this weren't
> > there. I would
> > >>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long,
> since
> > in any case
> > >>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first
> > approach, a design
> > >>>>>>>>>>>>> improvement would be great, though :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type
> so
> > >>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu
> > timestamp storage?
> > >>>>>>>>>>>>> - Is there any other approach?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
> > >>>>>>>>>>>> that the best alternative right now. Because, I would add
> > more compatible
> > >>>>>>>>>>>> datatypes to the mapping parameters which users can use. And
> > this
> > >>>>>>>>>>>> conversion should not be dificult to implement in my
> opinion.
> > Also, the new
> > >>>>>>>>>>>> Date datatype of avro could be implemented in newer versions
> > because it
> > >>>>>>>>>>>> would need further analysis in other datastores too. I will
> > work on that.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
> > >>>>>>>>>>>>> *> KuduClient has multiple flushing modes
> > >>>>>>>>>>>>> <
> >
> https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html
> > >and
> > >>>>>>>>>>>>> also can set time interval
> > >>>>>>>>>>>>> <
> >
> https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int-
> > >
> > >>>>>>>>>>>>> for automatic flush.*
> > >>>>>>>>>>>>> *> Should theses behaviors be configurable using
> > >>>>>>>>>>>>> gora.properties file? or just use the default
> > configurations.*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
> > >>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table,
> > but at the same
> > >>>>>>>>>>>>> time we implement the flush() method to force the flush
> [3].
> > I would
> > >>>>>>>>>>>>> suggest to follow that example, but adding the flushing
> > options of Kudu.
> > >>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you
> > suggest?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do
> > >>>>>>>>>>>> the job for most use cases. But I will add a configuration
> in
> > >>>>>>>>>>>> gora.properties for selecting the other modes and specifying
> > a autoflush
> > >>>>>>>>>>>> time if needed by the user.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
> > >>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the
> > expected
> > >>>>>>>>>>>>> behavior of this method?, should I use the partition
> > definition in the xml
> > >>>>>>>>>>>>> mapping file for this?.*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop.
> Apache
> > >>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and
> > Reduce that
> > >>>>>>>>>>>>> allows to get/write Entities directly.
> > >>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
> > >>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
> > >>>>>>>>>>>>> compute the splits (start key---end key) with the location
> > of the split to
> > >>>>>>>>>>>>> create a colection of partitions [6].
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local
> > kudu
> > >>>>>>>>>>>>> splits, then this method does the needed preparation to
> > allow to "send the
> > >>>>>>>>>>>>> computation to where the data is locally".
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In any case, you can see that:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - MongoDB store implementation does not implement
> > >>>>>>>>>>>>> splitting [7]
> > >>>>>>>>>>>>> - Cassandra store implementation does not implement
> > >>>>>>>>>>>>> splitting [8]
> > >>>>>>>>>>>>> - Aerospike store implementation does not implement
> > >>>>>>>>>>>>> splitting [9]
> > >>>>>>>>>>>>> - Accumulo store implementation* does* implement
> splitting
> > >>>>>>>>>>>>> [10]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> If Kudu has a method to get the different splits for a
> table
> > >>>>>>>>>>>>> and its locations, then you will be able to implement the
> > full feature.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
> > >>>>>>>>>>>>> elaborated much, so if you find you need more information
> > let me know :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> I will check whether Kudu has these features in order to
> > >>>>>>>>>>>> implement this method. If not I will use the default
> > implementation found
> > >>>>>>>>>>>> in other backends.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> About Queries, what I can tell is that Hbase only
> implements
> > >>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations:
> > "get" and "scan",
> > >>>>>>>>>>>>> and the querying is for "scan" operation, were you want an
> > interval (or
> > >>>>>>>>>>>>> all) of the rows. Does Kudu have more querying
> functionality?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
> > >>>>>>>>>>>> conditional predicates for filtering. I am using those
> > classes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> About other topic, I am trying to install Kudu in
> standalone
> > >>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do
> > you have a
> > >>>>>>>>>>>>> standalone installation? How do you do it? I found some
> > instructions, but
> > >>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for
> > something like
> > >>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> I am using an embedded mini-cluster which comes with
> compiled
> > >>>>>>>>>>>> binaries and can be used with maven[1] for testing my code.
> > Once I get it
> > >>>>>>>>>>>> mature enough I think I will be testing the datastore with a
> > docker
> > >>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle
> either
> > and I am
> > >>>>>>>>>>>> kinda noob for compiling it myself.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [1]
> > >>>>>>>>>>>>
> >
> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
> > >>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Good job and thank you!! :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Alfonso Nishikawa
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> [1] -
> > >>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
> > >>>>>>>>>>>>> [2] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
> > >>>>>>>>>>>>> [3] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
> > >>>>>>>>>>>>> [4] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
> > >>>>>>>>>>>>> [5] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
> > >>>>>>>>>>>>> [6] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
> > >>>>>>>>>>>>> [7] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
> > >>>>>>>>>>>>> [8] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
> > >>>>>>>>>>>>> [9] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
> > >>>>>>>>>>>>> [10] -
> > >>>>>>>>>>>>>
> >
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
> > >>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
> > >>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi all.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
> > >>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a
> > look if you have
> > >>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> This week, I will be continue working in the Queries
> > >>>>>>>>>>>>>> implementation, please reach me out if you have any
> > suggestions.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed
> this
> > >>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the
> > expected behavior of
> > >>>>>>>>>>>>>> this method?, should I use the partition definition in the
> > xml mapping file
> > >>>>>>>>>>>>>> for this?.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>> John.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
> > >>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi all.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
> > >>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for
> > your support and
> > >>>>>>>>>>>>>>> confidence to the project and me.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
> > >>>>>>>>>>>>>>> pushed my last commits to my branch [2].
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
> > >>>>>>>>>>>>>>> deserialization process in order to identify
> optimizations
> > specific for
> > >>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends
> > which probably
> > >>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start
> working
> > on the Queries
> > >>>>>>>>>>>>>>> implementation.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types.
> > Please
> > >>>>>>>>>>>>>>> give it a look if you have time.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
> > >>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Carlos.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday.
> > :D
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>> John.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
> > >>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi John
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on
> June
> > >>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors'
> > evaluation on time.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>> Carlos
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
> > >>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi all.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
> > >>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
> > >>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always
> > manually executed?.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
> > >>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get,
> > delete, exists. Do
> > >>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this
> > task?.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
> > >>>>>>>>>>>>>>>>>> problem.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
> > >>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi John
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
> > >>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for
> > datatypes, which is very
> > >>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should
> > replace [1] for [2]
> > >>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
> > >>>>>>>>>>>>>>>>>>> [2]
> > >>>>>>>>>>>>>>>>>>>
> > https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Carlos
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
> > >>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi all.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
> > >>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc
> > documentation I think I will
> > >>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to
> > enable and check schema
> > >>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>>>>> John.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
> > >>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on
> > your
> > >>>>>>>>>>>>>>>>>>>>> comments.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso
> Nishikawa
> > (<
> > >>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi, John.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations
> on
> > >>>>>>>>>>>>>>>>>>>>>> the mapping file.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
> > >>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is
> it
> > a problem for Apache Gora?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes
> with
> > >>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses
> > Maven 3 in a quite new
> > >>>>>>>>>>>>>>>>>>>>>> version :)
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> [1] -
> > >>>>>>>>>>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso
> Nishikawa
> > >>>>>>>>>>>>>>>>>>>>>> (<al...@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi, John.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thank you!
> > >>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go
> > on
> > >>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2].
> > Same for [3] and from there,
> > >>>>>>>>>>>>>>>>>>>>>>> should not set the version there.
> > >>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4]
> and
> > >>>>>>>>>>>>>>>>>>>>>>> from there.
> > >>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
> > >>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
> > >>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol"
> RFC
> > >>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it.
> > The default port for kudu
> > >>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it?
> > >>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality
> to
> > >>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's
> > store [7] in you
> > >>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on
> your
> > readMapping at [9], so
> > >>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
> > >>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
> > >>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have
> > the checked ones) like in
> > >>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An
> > example is [12]
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
> > >>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> [1] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
> > >>>>>>>>>>>>>>>>>>>>>>> [2] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> > https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
> > >>>>>>>>>>>>>>>>>>>>>>> [3] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
> > >>>>>>>>>>>>>>>>>>>>>>> [4] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
> > >>>>>>>>>>>>>>>>>>>>>>> [5] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
> > >>>>>>>>>>>>>>>>>>>>>>> [6] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
> > >>>>>>>>>>>>>>>>>>>>>>> [7] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
> > >>>>>>>>>>>>>>>>>>>>>>> [8] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
> > >>>>>>>>>>>>>>>>>>>>>>> [9] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
> > >>>>>>>>>>>>>>>>>>>>>>> [10] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
> > >>>>>>>>>>>>>>>>>>>>>>> [11] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
> > >>>>>>>>>>>>>>>>>>>>>>> [12] -
> > >>>>>>>>>>>>>>>>>>>>>>>
> >
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
> > >>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi all.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki
> > [1].
> > >>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on
> > the create schema operation
> > >>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning
> > configurations in the mapping
> > >>>>>>>>>>>>>>>>>>>>>>>> file.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my
> > last
> > >>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>>>>>>>>>>>>
> >
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> > >>>>>>>>>>>>>>>>>>>>>>>> [2]
> > >>>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/tree/GORA-485
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>> John
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> >
>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Kevin.
KuduTestHarness, theoretically should detect environment through this
plugin "os-maven-plugin" and download the corresponding kudu binaries [1],
and it have worked fine for me.
Nonetheless, docker is a good idea. I will give a look to testcontainers
and docker.
[1]
https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
Regards,
John
El lun., 5 ago. 2019 a las 23:58, Kevin Ratnasekera (<
djkevincr1989@gmail.com>) escribió:
> Hi John,
>
> Can't we spin up Kudu docker [1] instance for testing purposes? We have
> used Test containers [2] some data stores like couch DB. Gora build should
> work in both Linux and Non Linux environments. Eg:- Windows. Is classifier
> [3] depend on the environment the build is running?
>
> Kudu is based on C/ C++, so to spin up a server instance, we need check a
> approach like docker, using such approach allow us to avoid these OS,
> dependency related stuff come in to play in builds.
>
> [1] https://hub.docker.com/r/usuresearch/apache-kudu/
> [2] https://www.testcontainers.org/
> [3] <classifier>linux-x86_64</classifier>
>
> Regards
> Kevin
>
> On Tue, Aug 6, 2019 at 9:56 AM John Mora <jh...@gmail.com> wrote:
>
> > Hi Alfonso,
> >
> > Unfortunately, I have not been able to reproduce the issue. Maybe it is
> > related with my Java version (Oracle), I will try with OpenJDK.
> > Some details about my development environment:
> >
> > os.detected.name: linux
> > os.detected.arch: x86_64
> > os.detected.version: 4.10
> > os.detected.version.major: 4
> > os.detected.version.minor: 10
> > os.detected.release: linuxmint
> > os.detected.release.version: 18.3
> > os.detected.release.like.linuxmint: true
> > os.detected.release.like.ubuntu: true
> > os.detected.classifier: linux-x86_64
> >
> > Java
> > java version "1.8.0_171"
> > Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
> >
> > Maven
> > Apache Maven 3.3.9
> > Maven home: /usr/share/maven
> > Java version: 1.8.0_171, vendor: Oracle Corporation
> > Java home: /usr/lib/jvm/java-8-oracle/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
> > "unix"
> >
> >
> > Best,
> > John.
> >
> > El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
> > alfonso.nishikawa@gmail.com>) escribió:
> >
> >> Hi,
> >>
> >> I am using now the following pom configuration I got from executing `mvn
> >> dependency:tree`:
> >>
> >> <dependency>
> >> <groupId>org.apache.kudu</groupId>
> >> <artifactId>kudu-binary</artifactId>
> >> <classifier>linux-x86_64</classifier>
> >> <version>1.9.0</version>
> >> <scope>test</scope>
> >> </dependency>
> >>
> >> When I execute `mvn clen package` on gora-kudu I find that it spawns the
> >> following command:
> >>
> >> kudu-master
> >> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
> >> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
> >> --block_manager=log --webserver_interface=localhost
> --ipki_ca_key_size=1024
> >> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
> >> --webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync
> >> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none
> >> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
> >> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
> >>
> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
> >> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
> >> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
> >> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
> >> *127.26.116.188*:42253
> >>
> >>
> >> I highlight the IP addresses because they clearly are not my computer,
> >> and I guess that is why the tests can't connect to the the database.
> >>
> >> Any idea on how to solve this?
> >>
> >> Thank you!
> >>
> >>
> >> Best Regards,
> >>
> >> Alfonso Nishikawa
> >>
> >>
> >>
> >> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
> >> alfonso.nishikawa@gmail.com>) escribió:
> >>
> >>> Hi, John.
> >>>
> >>> I get a core dump from the binary kudu server when trying to run the
> >>> tests. Didn't find a log file, but will search thoroughly later.
> Happened
> >>> anytime to you? Does it happens to anyone?
> >>>
> >>> I am using Ubuntu 18.04
> >>>
> >>> Thank you!
> >>>
> >>> Regards,
> >>>
> >>> Alfonso Nishikawa
> >>>
> >>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
> >>> escribió:
> >>>
> >>>> Hi John,
> >>>>
> >>>> I've already made my comments at your PR. Please check them carefully
> >>>> and ask me if you need help.
> >>>>
> >>>> For the documentation, I've checked what you've done. On the other
> >>>> hand, I would want to encourage you to write a blog post about your
> Kudu
> >>>> implementation and demonstrate an example of Kudu integration with
> Gora as
> >>>> like a tutorial.
> >>>>
> >>>> Kind Regards,
> >>>> Furkan KAMACI
> >>>>
> >>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com>
> wrote:
> >>>>
> >>>>> Hi all.
> >>>>>
> >>>>> I have updated my report in the Wiki[1].
> >>>>>
> >>>>> Also, I have sent a PR with my last commits for review [2]. Please
> >>>>> give it a look if you have time.
> >>>>>
> >>>>> This week, I will continue working on the documentation of the kudu
> >>>>> datastore.
> >>>>>
> >>>>> Please let me know if you have suggestions.
> >>>>>
> >>>>> [1]
> >>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>> [2] https://github.com/apache/gora/pull/178
> >>>>>
> >>>>> Best,
> >>>>> John.
> >>>>>
> >>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<
> carlosrmng@gmail.com>)
> >>>>> escribió:
> >>>>>
> >>>>>> Hi John,
> >>>>>>
> >>>>>> Thanks for the update. I reviewed your code a little bit, it is
> >>>>>> looking good. I think tha you should send a PR in order to receive
> feedback
> >>>>>> from other community members.
> >>>>>>
> >>>>>> Best,
> >>>>>> Carlos
> >>>>>>
> >>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jhnmora000@gmail.com
> >)
> >>>>>> escribió:
> >>>>>>
> >>>>>>> Hi all.
> >>>>>>>
> >>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
> >>>>>>> to my branch [2]. Please give it a look if you have time.
> >>>>>>>
> >>>>>>> This week, I will give a look to the documentation of datastores.
> >>>>>>>
> >>>>>>> Please let me know if you have suggestions.
> >>>>>>>
> >>>>>>> [1]
> >>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> John
> >>>>>>>
> >>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<
> jhnmora000@gmail.com>)
> >>>>>>> escribió:
> >>>>>>>
> >>>>>>>> Hi Alfonso,
> >>>>>>>>
> >>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem
> I
> >>>>>>>> will find an alternative, it is only an utilitary class.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> John
> >>>>>>>>
> >>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
> >>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> >>>>>>>>
> >>>>>>>>> Hi, John.
> >>>>>>>>>
> >>>>>>>>> I checked out your code and it looks good :)
> >>>>>>>>> I found that you use javafx, but that is not present in OpenJDK
> >>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I
> would
> >>>>>>>>> suggest to change it.
> >>>>>>>>>
> >>>>>>>>> Good job, keep it going :)
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>>
> >>>>>>>>> Alfonso Nishikawa
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
> >>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>
> >>>>>>>>>> Hi.
> >>>>>>>>>>
> >>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
> >>>>>>>>>> commits to my branch [2]. Please give it a look if you have
> time.
> >>>>>>>>>>
> >>>>>>>>>> This week, I will give a look to the map reduce tests for
> >>>>>>>>>> DataStores.
> >>>>>>>>>>
> >>>>>>>>>> Please let me know if you have suggestions.
> >>>>>>>>>>
> >>>>>>>>>> [1]
> >>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> John
> >>>>>>>>>>
> >>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
> >>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>
> >>>>>>>>>>> Hi all
> >>>>>>>>>>>
> >>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
> >>>>>>>>>>> commits to my branch [2]. Please give it a look if you have
> time.
> >>>>>>>>>>>
> >>>>>>>>>>> This week, I will be working in the getPartitions and
> >>>>>>>>>>> deleteByQuery methods and testing the other tests in the
> DataStoreTestBase
> >>>>>>>>>>> class.
> >>>>>>>>>>>
> >>>>>>>>>>> Please let me know if you have suggestions.
> >>>>>>>>>>>
> >>>>>>>>>>> [1]
> >>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> John.
> >>>>>>>>>>>
> >>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
> >>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Alfonso,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks so much for your time and support for this project. I
> >>>>>>>>>>>> will work on your comments. Responses inline :)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
> >>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, John.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very
> >>>>>>>>>>>>> busy :( I will try to answer your questions :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *> In the Employee example there is a field called
> >>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the
> UNIXTIME_MICROS datatype
> >>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in
> the java world
> >>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is
> a Timestamp.
> >>>>>>>>>>>>> So, I was wondering whether I should force the usage of the
> UNIXTIME_MICROS
> >>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
> >>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first
> time I read about
> >>>>>>>>>>>>> because until the last version upgrade of Avro this weren't
> there. I would
> >>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since
> in any case
> >>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first
> approach, a design
> >>>>>>>>>>>>> improvement would be great, though :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
> >>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu
> timestamp storage?
> >>>>>>>>>>>>> - Is there any other approach?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
> >>>>>>>>>>>> that the best alternative right now. Because, I would add
> more compatible
> >>>>>>>>>>>> datatypes to the mapping parameters which users can use. And
> this
> >>>>>>>>>>>> conversion should not be dificult to implement in my opinion.
> Also, the new
> >>>>>>>>>>>> Date datatype of avro could be implemented in newer versions
> because it
> >>>>>>>>>>>> would need further analysis in other datastores too. I will
> work on that.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
> >>>>>>>>>>>>> *> KuduClient has multiple flushing modes
> >>>>>>>>>>>>> <
> https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html
> >and
> >>>>>>>>>>>>> also can set time interval
> >>>>>>>>>>>>> <
> https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int-
> >
> >>>>>>>>>>>>> for automatic flush.*
> >>>>>>>>>>>>> *> Should theses behaviors be configurable using
> >>>>>>>>>>>>> gora.properties file? or just use the default
> configurations.*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
> >>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table,
> but at the same
> >>>>>>>>>>>>> time we implement the flush() method to force the flush [3].
> I would
> >>>>>>>>>>>>> suggest to follow that example, but adding the flushing
> options of Kudu.
> >>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you
> suggest?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do
> >>>>>>>>>>>> the job for most use cases. But I will add a configuration in
> >>>>>>>>>>>> gora.properties for selecting the other modes and specifying
> a autoflush
> >>>>>>>>>>>> time if needed by the user.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
> >>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the
> expected
> >>>>>>>>>>>>> behavior of this method?, should I use the partition
> definition in the xml
> >>>>>>>>>>>>> mapping file for this?.*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache
> >>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and
> Reduce that
> >>>>>>>>>>>>> allows to get/write Entities directly.
> >>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
> >>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
> >>>>>>>>>>>>> compute the splits (start key---end key) with the location
> of the split to
> >>>>>>>>>>>>> create a colection of partitions [6].
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local
> kudu
> >>>>>>>>>>>>> splits, then this method does the needed preparation to
> allow to "send the
> >>>>>>>>>>>>> computation to where the data is locally".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In any case, you can see that:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - MongoDB store implementation does not implement
> >>>>>>>>>>>>> splitting [7]
> >>>>>>>>>>>>> - Cassandra store implementation does not implement
> >>>>>>>>>>>>> splitting [8]
> >>>>>>>>>>>>> - Aerospike store implementation does not implement
> >>>>>>>>>>>>> splitting [9]
> >>>>>>>>>>>>> - Accumulo store implementation* does* implement splitting
> >>>>>>>>>>>>> [10]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If Kudu has a method to get the different splits for a table
> >>>>>>>>>>>>> and its locations, then you will be able to implement the
> full feature.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
> >>>>>>>>>>>>> elaborated much, so if you find you need more information
> let me know :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> I will check whether Kudu has these features in order to
> >>>>>>>>>>>> implement this method. If not I will use the default
> implementation found
> >>>>>>>>>>>> in other backends.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
> >>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations:
> "get" and "scan",
> >>>>>>>>>>>>> and the querying is for "scan" operation, were you want an
> interval (or
> >>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
> >>>>>>>>>>>> conditional predicates for filtering. I am using those
> classes.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone
> >>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do
> you have a
> >>>>>>>>>>>>> standalone installation? How do you do it? I found some
> instructions, but
> >>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for
> something like
> >>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
> >>>>>>>>>>>> binaries and can be used with maven[1] for testing my code.
> Once I get it
> >>>>>>>>>>>> mature enough I think I will be testing the datastore with a
> docker
> >>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either
> and I am
> >>>>>>>>>>>> kinda noob for compiling it myself.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1]
> >>>>>>>>>>>>
> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
> >>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Good job and thank you!! :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alfonso Nishikawa
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] -
> >>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
> >>>>>>>>>>>>> [2] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
> >>>>>>>>>>>>> [3] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
> >>>>>>>>>>>>> [4] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
> >>>>>>>>>>>>> [5] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
> >>>>>>>>>>>>> [6] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
> >>>>>>>>>>>>> [7] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
> >>>>>>>>>>>>> [8] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
> >>>>>>>>>>>>> [9] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
> >>>>>>>>>>>>> [10] -
> >>>>>>>>>>>>>
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
> >>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
> >>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi all.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
> >>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a
> look if you have
> >>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This week, I will be continue working in the Queries
> >>>>>>>>>>>>>> implementation, please reach me out if you have any
> suggestions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
> >>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the
> expected behavior of
> >>>>>>>>>>>>>> this method?, should I use the partition definition in the
> xml mapping file
> >>>>>>>>>>>>>> for this?.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> John.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
> >>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
> >>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for
> your support and
> >>>>>>>>>>>>>>> confidence to the project and me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
> >>>>>>>>>>>>>>> pushed my last commits to my branch [2].
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
> >>>>>>>>>>>>>>> deserialization process in order to identify optimizations
> specific for
> >>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends
> which probably
> >>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working
> on the Queries
> >>>>>>>>>>>>>>> implementation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types.
> Please
> >>>>>>>>>>>>>>> give it a look if you have time.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
> >>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Carlos.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday.
> :D
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> John.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
> >>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
> >>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors'
> evaluation on time.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>> Carlos
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
> >>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi all.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
> >>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
> >>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always
> manually executed?.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
> >>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get,
> delete, exists. Do
> >>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this
> task?.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
> >>>>>>>>>>>>>>>>>> problem.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
> >>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
> >>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for
> datatypes, which is very
> >>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should
> replace [1] for [2]
> >>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
> >>>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>>>>
> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Carlos
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
> >>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi all.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
> >>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc
> documentation I think I will
> >>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to
> enable and check schema
> >>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> John.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
> >>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on
> your
> >>>>>>>>>>>>>>>>>>>>> comments.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa
> (<
> >>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi, John.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on
> >>>>>>>>>>>>>>>>>>>>>> the mapping file.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
> >>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it
> a problem for Apache Gora?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
> >>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses
> Maven 3 in a quite new
> >>>>>>>>>>>>>>>>>>>>>> version :)
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [1] -
> >>>>>>>>>>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa
> >>>>>>>>>>>>>>>>>>>>>> (<al...@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, John.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thank you!
> >>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go
> on
> >>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2].
> Same for [3] and from there,
> >>>>>>>>>>>>>>>>>>>>>>> should not set the version there.
> >>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and
> >>>>>>>>>>>>>>>>>>>>>>> from there.
> >>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
> >>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
> >>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC
> >>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it.
> The default port for kudu
> >>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it?
> >>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to
> >>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's
> store [7] in you
> >>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your
> readMapping at [9], so
> >>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
> >>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
> >>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have
> the checked ones) like in
> >>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An
> example is [12]
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
> >>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> [1] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
> >>>>>>>>>>>>>>>>>>>>>>> [2] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
> >>>>>>>>>>>>>>>>>>>>>>> [3] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
> >>>>>>>>>>>>>>>>>>>>>>> [4] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
> >>>>>>>>>>>>>>>>>>>>>>> [5] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
> >>>>>>>>>>>>>>>>>>>>>>> [6] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
> >>>>>>>>>>>>>>>>>>>>>>> [7] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
> >>>>>>>>>>>>>>>>>>>>>>> [8] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
> >>>>>>>>>>>>>>>>>>>>>>> [9] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
> >>>>>>>>>>>>>>>>>>>>>>> [10] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
> >>>>>>>>>>>>>>>>>>>>>>> [11] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
> >>>>>>>>>>>>>>>>>>>>>>> [12] -
> >>>>>>>>>>>>>>>>>>>>>>>
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
> >>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi all.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki
> [1].
> >>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on
> the create schema operation
> >>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning
> configurations in the mapping
> >>>>>>>>>>>>>>>>>>>>>>>> file.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my
> last
> >>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> >>>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>> John
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
>
Re: Kudu datastore reports
Posted by Kevin Ratnasekera <dj...@gmail.com>.
Hi John,
Can't we spin up Kudu docker [1] instance for testing purposes? We have
used Test containers [2] some data stores like couch DB. Gora build should
work in both Linux and Non Linux environments. Eg:- Windows. Is classifier
[3] depend on the environment the build is running?
Kudu is based on C/ C++, so to spin up a server instance, we need check a
approach like docker, using such approach allow us to avoid these OS,
dependency related stuff come in to play in builds.
[1] https://hub.docker.com/r/usuresearch/apache-kudu/
[2] https://www.testcontainers.org/
[3] <classifier>linux-x86_64</classifier>
Regards
Kevin
On Tue, Aug 6, 2019 at 9:56 AM John Mora <jh...@gmail.com> wrote:
> Hi Alfonso,
>
> Unfortunately, I have not been able to reproduce the issue. Maybe it is
> related with my Java version (Oracle), I will try with OpenJDK.
> Some details about my development environment:
>
> os.detected.name: linux
> os.detected.arch: x86_64
> os.detected.version: 4.10
> os.detected.version.major: 4
> os.detected.version.minor: 10
> os.detected.release: linuxmint
> os.detected.release.version: 18.3
> os.detected.release.like.linuxmint: true
> os.detected.release.like.ubuntu: true
> os.detected.classifier: linux-x86_64
>
> Java
> java version "1.8.0_171"
> Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
>
> Maven
> Apache Maven 3.3.9
> Maven home: /usr/share/maven
> Java version: 1.8.0_171, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
> "unix"
>
>
> Best,
> John.
>
> El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi,
>>
>> I am using now the following pom configuration I got from executing `mvn
>> dependency:tree`:
>>
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-binary</artifactId>
>> <classifier>linux-x86_64</classifier>
>> <version>1.9.0</version>
>> <scope>test</scope>
>> </dependency>
>>
>> When I execute `mvn clen package` on gora-kudu I find that it spawns the
>> following command:
>>
>> kudu-master
>> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
>> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
>> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
>> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
>> --webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync
>> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none
>> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
>> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
>> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
>> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
>> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
>> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
>> *127.26.116.188*:42253
>>
>>
>> I highlight the IP addresses because they clearly are not my computer,
>> and I guess that is why the tests can't connect to the the database.
>>
>> Any idea on how to solve this?
>>
>> Thank you!
>>
>>
>> Best Regards,
>>
>> Alfonso Nishikawa
>>
>>
>>
>> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> I get a core dump from the binary kudu server when trying to run the
>>> tests. Didn't find a log file, but will search thoroughly later. Happened
>>> anytime to you? Does it happens to anyone?
>>>
>>> I am using Ubuntu 18.04
>>>
>>> Thank you!
>>>
>>> Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
>>> escribió:
>>>
>>>> Hi John,
>>>>
>>>> I've already made my comments at your PR. Please check them carefully
>>>> and ask me if you need help.
>>>>
>>>> For the documentation, I've checked what you've done. On the other
>>>> hand, I would want to encourage you to write a blog post about your Kudu
>>>> implementation and demonstrate an example of Kudu integration with Gora as
>>>> like a tutorial.
>>>>
>>>> Kind Regards,
>>>> Furkan KAMACI
>>>>
>>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com> wrote:
>>>>
>>>>> Hi all.
>>>>>
>>>>> I have updated my report in the Wiki[1].
>>>>>
>>>>> Also, I have sent a PR with my last commits for review [2]. Please
>>>>> give it a look if you have time.
>>>>>
>>>>> This week, I will continue working on the documentation of the kudu
>>>>> datastore.
>>>>>
>>>>> Please let me know if you have suggestions.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/apache/gora/pull/178
>>>>>
>>>>> Best,
>>>>> John.
>>>>>
>>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> Thanks for the update. I reviewed your code a little bit, it is
>>>>>> looking good. I think tha you should send a PR in order to receive feedback
>>>>>> from other community members.
>>>>>>
>>>>>> Best,
>>>>>> Carlos
>>>>>>
>>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>
>>>>>>> This week, I will give a look to the documentation of datastores.
>>>>>>>
>>>>>>> Please let me know if you have suggestions.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Cheers,
>>>>>>> John
>>>>>>>
>>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Alfonso,
>>>>>>>>
>>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I
>>>>>>>> will find an alternative, it is only an utilitary class.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi, John.
>>>>>>>>>
>>>>>>>>> I checked out your code and it looks good :)
>>>>>>>>> I found that you use javafx, but that is not present in OpenJDK
>>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I would
>>>>>>>>> suggest to change it.
>>>>>>>>>
>>>>>>>>> Good job, keep it going :)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>
>>>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>>>> DataStores.
>>>>>>>>>>
>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all
>>>>>>>>>>>
>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>
>>>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>>>> class.
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> John.
>>>>>>>>>>>
>>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks so much for your time and support for this project. I
>>>>>>>>>>>> will work on your comments. Responses inline :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very
>>>>>>>>>>>>> busy :( I will try to answer your questions :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>>>
>>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
>>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I read about
>>>>>>>>>>>>> because until the last version upgrade of Avro this weren't there. I would
>>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in any case
>>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, a design
>>>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
>>>>>>>>>>>> that the best alternative right now. Because, I would add more compatible
>>>>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>>>> also can set time interval
>>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>>>> *> Should theses behaviors be configurable using
>>>>>>>>>>>>> gora.properties file? or just use the default configurations.*
>>>>>>>>>>>>>
>>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do
>>>>>>>>>>>> the job for most use cases. But I will add a configuration in
>>>>>>>>>>>> gora.properties for selecting the other modes and specifying a autoflush
>>>>>>>>>>>> time if needed by the user.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>> mapping file for this?.*
>>>>>>>>>>>>>
>>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache
>>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and Reduce that
>>>>>>>>>>>>> allows to get/write Entities directly.
>>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
>>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
>>>>>>>>>>>>> compute the splits (start key---end key) with the location of the split to
>>>>>>>>>>>>> create a colection of partitions [6].
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>>>>>>> computation to where the data is locally".
>>>>>>>>>>>>>
>>>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - MongoDB store implementation does not implement
>>>>>>>>>>>>> splitting [7]
>>>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>>>> splitting [8]
>>>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>>>> splitting [9]
>>>>>>>>>>>>> - Accumulo store implementation* does* implement splitting
>>>>>>>>>>>>> [10]
>>>>>>>>>>>>>
>>>>>>>>>>>>> If Kudu has a method to get the different splits for a table
>>>>>>>>>>>>> and its locations, then you will be able to implement the full feature.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>>>> in other backends.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone
>>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do you have a
>>>>>>>>>>>>> standalone installation? How do you do it? I found some instructions, but
>>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for something like
>>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>>>>> for this?.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
>>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please
>>>>>>>>>>>>>>> give it a look if you have time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
>>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
>>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on
>>>>>>>>>>>>>>>>>>>>>> the mapping file.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>> (<al...@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on
>>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and
>>>>>>>>>>>>>>>>>>>>>>> from there.
>>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC
>>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it. The default port for kudu
>>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to
>>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1].
>>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the create schema operation
>>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning configurations in the mapping
>>>>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
I finally managed to pass the tests!!!
It seems it needs the packages ntp-wait and/or chronyc, so in order to fix
the error:
java.io.IOException: failed to start masters: Unable to start Master at
index 0:
/tmp/kudu-binary-jar2880948780300354108/kudu-binary-1.9.0-linux-x86_64/bin/kudu-master:
process exited on signal 6 (core dumped)
I just installed:
sudo apt-get install chrony
And the minicluster started working :)
Will take further looks to the PR soon.
Thank you!!
Regards,
Alfonso Nishikawa
El vie., 9 ago. 2019 a las 20:35, John Mora (<jh...@gmail.com>)
escribió:
> Hi.
>
> I updated my PR[2] with some improvements.
>
> Also, I tested my code with this docker image [3] and it worked fine.
>
> This what I did:
> Run docker container:
> docker run --hostname localhost -d --rm --name apache-kudu -p 7051:7051 -p
> 7050:7050 -p 8051:8051 -p 8050:8050 usuresearch/apache-kudu:1.8.0
>
> Change master address within tests [1]:
>
> @Override
> public void setUpClass() throws Exception {
> //comment KuduTestHarness start
> //harness.before();
> conf.set(KuduBackendConstants.PROP_MASTERADDRESSES, "localhost:7051");
> }
>
> @Override
> public void tearDownClass() throws Exception {
> // comment KuduTestHarness end
> ///harness.after();
> }
>
>
> If you agree I could a create second version of the tests using
> testcontainers and this docker image.
>
>
> [1]
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/java/org/apache/gora/kudu/GoraKuduTestDriver.java#L41
> [2] https://github.com/apache/gora/pull/178
> [3] https://hub.docker.com/r/usuresearch/apache-kudu/
>
> Cheers,
> John
>
> El vie., 9 ago. 2019 a las 13:33, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi Alfonso,
>>
>> Please take into a consideration that the property
>> 'gora.datastore.kudu.masterAddress' is overridden in the class
>> GoraKuduTestDriver [1]. Because KuduTestHarness generates random ports
>> which need to be configured at runtime. Probably, you should change the
>> property there too.
>>
>> I will test my code with a docker container, in order to figure out the
>> origin of the issue. Please let me know if someone faces this issue when
>> building the project.
>>
>>
>> [1]
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/java/org/apache/gora/kudu/GoraKuduTestDriver.java#L41
>>
>>
>> Cheers,
>> John.
>>
>>
>> El vie., 9 ago. 2019 a las 12:22, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi,
>>>
>>> I installed a standalone Kudu server (compiled from sources) in a
>>> virtual machine and configured test/resources/gora.properties to use
>>> "nosql:8051" master.
>>>
>>> The tests freezes waiting for kudu response and a tons of connections to
>>> the master gets created.
>>>
>>> *To the community:* Can someone please clone
>>> https://github.com/jhnmora000/gora/tree/GORA-485 and build it to see if
>>> the problem can be reproduced?
>>>
>>> I attach snapshots showing how connections stays waiting.
>>>
>>> Thank you!
>>>
>>> Best Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>>
>>>
>>> El jue., 8 ago. 2019 a las 20:18, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi, John.
>>>>
>>>> Tried using Oracle jdk 1.8 and found the same core dump:
>>>>
>>>> [INFO] Running org.apache.gora.kudu.store.TestKuduStore
>>>> [ERROR] Tests run: 44, Failures: 0, Errors: 40, Skipped: 4, Time
>>>> elapsed: 52.466 s <<< FAILURE! - in org.apache.gora.kudu.store.TestKuduStore
>>>> [ERROR] testNewInstance(org.apache.gora.kudu.store.TestKuduStore) Time
>>>> elapsed: 1.834 s <<< ERROR!
>>>> java.io.IOException: failed to start masters: Unable to start Master at
>>>> index 0:
>>>> /tmp/kudu-binary-jar4319751617646651391/kudu-binary-1.9.0-linux-x86_64/bin/kudu-master:
>>>> process exited on signal 6 (core dumped)
>>>>
>>>> (I expected to fail too, since the problem doesn't look like being
>>>> related to the jvm).
>>>>
>>>> Thanks for giving it a look. Don't know what must be the problem :\
>>>>
>>>> Best Regards,
>>>>
>>>> Alfosno Nishikawa
>>>>
>>>>
>>>> El mar., 6 ago. 2019 a las 4:26, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Alfonso,
>>>>>
>>>>> Unfortunately, I have not been able to reproduce the issue. Maybe it
>>>>> is related with my Java version (Oracle), I will try with OpenJDK.
>>>>> Some details about my development environment:
>>>>>
>>>>> os.detected.name: linux
>>>>> os.detected.arch: x86_64
>>>>> os.detected.version: 4.10
>>>>> os.detected.version.major: 4
>>>>> os.detected.version.minor: 10
>>>>> os.detected.release: linuxmint
>>>>> os.detected.release.version: 18.3
>>>>> os.detected.release.like.linuxmint: true
>>>>> os.detected.release.like.ubuntu: true
>>>>> os.detected.classifier: linux-x86_64
>>>>>
>>>>> Java
>>>>> java version "1.8.0_171"
>>>>> Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
>>>>>
>>>>> Maven
>>>>> Apache Maven 3.3.9
>>>>> Maven home: /usr/share/maven
>>>>> Java version: 1.8.0_171, vendor: Oracle Corporation
>>>>> Java home: /usr/lib/jvm/java-8-oracle/jre
>>>>> Default locale: en_US, platform encoding: UTF-8
>>>>> OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
>>>>> "unix"
>>>>>
>>>>>
>>>>> Best,
>>>>> John.
>>>>>
>>>>> El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using now the following pom configuration I got from executing
>>>>>> `mvn dependency:tree`:
>>>>>>
>>>>>> <dependency>
>>>>>> <groupId>org.apache.kudu</groupId>
>>>>>> <artifactId>kudu-binary</artifactId>
>>>>>> <classifier>linux-x86_64</classifier>
>>>>>> <version>1.9.0</version>
>>>>>> <scope>test</scope>
>>>>>> </dependency>
>>>>>>
>>>>>> When I execute `mvn clen package` on gora-kudu I find that it spawns
>>>>>> the following command:
>>>>>>
>>>>>> kudu-master
>>>>>> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
>>>>>> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
>>>>>> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
>>>>>> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
>>>>>> --webserver_interface=*127.26.116.190* --webserver_port=0
>>>>>> --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false
>>>>>> --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
>>>>>> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
>>>>>> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
>>>>>> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
>>>>>> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
>>>>>> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
>>>>>> *127.26.116.188*:42253
>>>>>>
>>>>>>
>>>>>> I highlight the IP addresses because they clearly are not my
>>>>>> computer, and I guess that is why the tests can't connect to the the
>>>>>> database.
>>>>>>
>>>>>> Any idea on how to solve this?
>>>>>>
>>>>>> Thank you!
>>>>>>
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>>
>>>>>>
>>>>>> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi, John.
>>>>>>>
>>>>>>> I get a core dump from the binary kudu server when trying to run the
>>>>>>> tests. Didn't find a log file, but will search thoroughly later. Happened
>>>>>>> anytime to you? Does it happens to anyone?
>>>>>>>
>>>>>>> I am using Ubuntu 18.04
>>>>>>>
>>>>>>> Thank you!
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Alfonso Nishikawa
>>>>>>>
>>>>>>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> I've already made my comments at your PR. Please check them
>>>>>>>> carefully and ask me if you need help.
>>>>>>>>
>>>>>>>> For the documentation, I've checked what you've done. On the other
>>>>>>>> hand, I would want to encourage you to write a blog post about your Kudu
>>>>>>>> implementation and demonstrate an example of Kudu integration with Gora as
>>>>>>>> like a tutorial.
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>> Furkan KAMACI
>>>>>>>>
>>>>>>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> I have updated my report in the Wiki[1].
>>>>>>>>>
>>>>>>>>> Also, I have sent a PR with my last commits for review [2]. Please
>>>>>>>>> give it a look if you have time.
>>>>>>>>>
>>>>>>>>> This week, I will continue working on the documentation of the
>>>>>>>>> kudu datastore.
>>>>>>>>>
>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/apache/gora/pull/178
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> John.
>>>>>>>>>
>>>>>>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<
>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>>
>>>>>>>>>> Thanks for the update. I reviewed your code a little bit, it is
>>>>>>>>>> looking good. I think tha you should send a PR in order to receive feedback
>>>>>>>>>> from other community members.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Carlos
>>>>>>>>>>
>>>>>>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>
>>>>>>>>>>> This week, I will give a look to the documentation of datastores.
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a
>>>>>>>>>>>> problem I will find an alternative, it is only an utilitary class.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked out your code and it looks good :)
>>>>>>>>>>>>> I found that you use javafx, but that is not present in
>>>>>>>>>>>>> OpenJDK and fails to compile, and since we don't stick to Oracle JVM I
>>>>>>>>>>>>> would suggest to change it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Good job, keep it going :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>>>>>>>> DataStores.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks so much for your time and support for this project.
>>>>>>>>>>>>>>>> I will work on your comments. Responses inline :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been
>>>>>>>>>>>>>>>>> very busy :( I will try to answer your questions :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
>>>>>>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I read about
>>>>>>>>>>>>>>>>> because until the last version upgrade of Avro this weren't there. I would
>>>>>>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in any case
>>>>>>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, a design
>>>>>>>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type
>>>>>>>>>>>>>>>>> so KuduStore converts between the Entity long field <-> Kudu timestamp
>>>>>>>>>>>>>>>>> storage?
>>>>>>>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp
>>>>>>>>>>>>>>>> conversion that the best alternative right now. Because, I would add more
>>>>>>>>>>>>>>>> compatible datatypes to the mapping parameters which users can use. And
>>>>>>>>>>>>>>>> this conversion should not be dificult to implement in my opinion. Also,
>>>>>>>>>>>>>>>> the new Date datatype of avro could be implemented in newer versions
>>>>>>>>>>>>>>>> because it would need further analysis in other datastores too. I will work
>>>>>>>>>>>>>>>> on that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>>>>>>>> also can set time interval
>>>>>>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>>>>>>>> *> Should theses behaviors be configurable using
>>>>>>>>>>>>>>>>> gora.properties file? or just use the default configurations.*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will
>>>>>>>>>>>>>>>> do the job for most use cases. But I will add a configuration in
>>>>>>>>>>>>>>>> gora.properties for selecting the other modes and specifying a autoflush
>>>>>>>>>>>>>>>> time if needed by the user.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>>>> mapping file for this?.*
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop.
>>>>>>>>>>>>>>>>> Apache Gora integrates with Hadoop implementing a custom Map and Reduce
>>>>>>>>>>>>>>>>> that allows to get/write Entities directly.
>>>>>>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
>>>>>>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
>>>>>>>>>>>>>>>>> compute the splits (start key---end key) with the location of the split to
>>>>>>>>>>>>>>>>> create a colection of partitions [6].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local
>>>>>>>>>>>>>>>>> kudu splits, then this method does the needed preparation to allow to "send
>>>>>>>>>>>>>>>>> the computation to where the data is locally".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - MongoDB store implementation does not implement
>>>>>>>>>>>>>>>>> splitting [7]
>>>>>>>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>>>>>>>> splitting [8]
>>>>>>>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>>>>>>>> splitting [9]
>>>>>>>>>>>>>>>>> - Accumulo store implementation* does* implement
>>>>>>>>>>>>>>>>> splitting [10]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If Kudu has a method to get the different splits for a
>>>>>>>>>>>>>>>>> table and its locations, then you will be able to implement the full
>>>>>>>>>>>>>>>>> feature.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>>>>>>>> in other backends.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> About Queries, what I can tell is that Hbase only
>>>>>>>>>>>>>>>>> implements "Start key" + "End key" because it has only 2 operations: "get"
>>>>>>>>>>>>>>>>> and "scan", and the querying is for "scan" operation, were you want an
>>>>>>>>>>>>>>>>> interval (or all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> About other topic, I am trying to install Kudu in
>>>>>>>>>>>>>>>>> standalone (all in 1 node). Do you use a Cloudera installation or do you
>>>>>>>>>>>>>>>>> have a standalone installation? How do you do it? I found some
>>>>>>>>>>>>>>>>> instructions, but they talk about compiling Kudu [11]. I was looking for
>>>>>>>>>>>>>>>>> something like HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am using an embedded mini-cluster which comes with
>>>>>>>>>>>>>>>> compiled binaries and can be used with maven[1] for testing my code. Once I
>>>>>>>>>>>>>>>> get it mature enough I think I will be testing the datastore with a docker
>>>>>>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>>>>> mapping file for this?.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
>>>>>>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types.
>>>>>>>>>>>>>>>>>>> Please give it a look if you have time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form
>>>>>>>>>>>>>>>>>>>> yesterday. :D
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on
>>>>>>>>>>>>>>>>>>>>> June 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
>>>>>>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know
>>>>>>>>>>>>>>>>>>>>>> how datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But,
>>>>>>>>>>>>>>>>>>>>>>> I noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on
>>>>>>>>>>>>>>>>>>>>>>>>> your comments.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso
>>>>>>>>>>>>>>>>>>>>>>>>> Nishikawa (<al...@gmail.com>)
>>>>>>>>>>>>>>>>>>>>>>>>> escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning
>>>>>>>>>>>>>>>>>>>>>>>>>> configurations on the mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes
>>>>>>>>>>>>>>>>>>>>>>>>>> with 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite
>>>>>>>>>>>>>>>>>>>>>>>>>> new version :)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso
>>>>>>>>>>>>>>>>>>>>>>>>>> Nishikawa (<al...@gmail.com>)
>>>>>>>>>>>>>>>>>>>>>>>>>> escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should
>>>>>>>>>>>>>>>>>>>>>>>>>>> go on the Dependency Management of the root pom [2]. Same for [3] and from
>>>>>>>>>>>>>>>>>>>>>>>>>>> there, should not set the version there.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4]
>>>>>>>>>>>>>>>>>>>>>>>>>>> and from there.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol"
>>>>>>>>>>>>>>>>>>>>>>>>>>> RFC 867 and you will need root permission to run it. The default port for
>>>>>>>>>>>>>>>>>>>>>>>>>>> kudu is 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality
>>>>>>>>>>>>>>>>>>>>>>>>>>> to load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]. This next week I think I should be focusing on the create schema
>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation and solving the issue of the partitioning configurations in the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my
>>>>>>>>>>>>>>>>>>>>>>>>>>>> last commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi.
I updated my PR[2] with some improvements.
Also, I tested my code with this docker image [3] and it worked fine.
This what I did:
Run docker container:
docker run --hostname localhost -d --rm --name apache-kudu -p 7051:7051 -p
7050:7050 -p 8051:8051 -p 8050:8050 usuresearch/apache-kudu:1.8.0
Change master address within tests [1]:
@Override
public void setUpClass() throws Exception {
//comment KuduTestHarness start
//harness.before();
conf.set(KuduBackendConstants.PROP_MASTERADDRESSES, "localhost:7051");
}
@Override
public void tearDownClass() throws Exception {
// comment KuduTestHarness end
///harness.after();
}
If you agree I could a create second version of the tests using
testcontainers and this docker image.
[1]
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/java/org/apache/gora/kudu/GoraKuduTestDriver.java#L41
[2] https://github.com/apache/gora/pull/178
[3] https://hub.docker.com/r/usuresearch/apache-kudu/
Cheers,
John
El vie., 9 ago. 2019 a las 13:33, John Mora (<jh...@gmail.com>)
escribió:
> Hi Alfonso,
>
> Please take into a consideration that the property
> 'gora.datastore.kudu.masterAddress' is overridden in the class
> GoraKuduTestDriver [1]. Because KuduTestHarness generates random ports
> which need to be configured at runtime. Probably, you should change the
> property there too.
>
> I will test my code with a docker container, in order to figure out the
> origin of the issue. Please let me know if someone faces this issue when
> building the project.
>
>
> [1]
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/java/org/apache/gora/kudu/GoraKuduTestDriver.java#L41
>
>
> Cheers,
> John.
>
>
> El vie., 9 ago. 2019 a las 12:22, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi,
>>
>> I installed a standalone Kudu server (compiled from sources) in a virtual
>> machine and configured test/resources/gora.properties to use "nosql:8051"
>> master.
>>
>> The tests freezes waiting for kudu response and a tons of connections to
>> the master gets created.
>>
>> *To the community:* Can someone please clone
>> https://github.com/jhnmora000/gora/tree/GORA-485 and build it to see if
>> the problem can be reproduced?
>>
>> I attach snapshots showing how connections stays waiting.
>>
>> Thank you!
>>
>> Best Regards,
>>
>> Alfonso Nishikawa
>>
>>
>>
>> El jue., 8 ago. 2019 a las 20:18, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> Tried using Oracle jdk 1.8 and found the same core dump:
>>>
>>> [INFO] Running org.apache.gora.kudu.store.TestKuduStore
>>> [ERROR] Tests run: 44, Failures: 0, Errors: 40, Skipped: 4, Time
>>> elapsed: 52.466 s <<< FAILURE! - in org.apache.gora.kudu.store.TestKuduStore
>>> [ERROR] testNewInstance(org.apache.gora.kudu.store.TestKuduStore) Time
>>> elapsed: 1.834 s <<< ERROR!
>>> java.io.IOException: failed to start masters: Unable to start Master at
>>> index 0:
>>> /tmp/kudu-binary-jar4319751617646651391/kudu-binary-1.9.0-linux-x86_64/bin/kudu-master:
>>> process exited on signal 6 (core dumped)
>>>
>>> (I expected to fail too, since the problem doesn't look like being
>>> related to the jvm).
>>>
>>> Thanks for giving it a look. Don't know what must be the problem :\
>>>
>>> Best Regards,
>>>
>>> Alfosno Nishikawa
>>>
>>>
>>> El mar., 6 ago. 2019 a las 4:26, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Alfonso,
>>>>
>>>> Unfortunately, I have not been able to reproduce the issue. Maybe it is
>>>> related with my Java version (Oracle), I will try with OpenJDK.
>>>> Some details about my development environment:
>>>>
>>>> os.detected.name: linux
>>>> os.detected.arch: x86_64
>>>> os.detected.version: 4.10
>>>> os.detected.version.major: 4
>>>> os.detected.version.minor: 10
>>>> os.detected.release: linuxmint
>>>> os.detected.release.version: 18.3
>>>> os.detected.release.like.linuxmint: true
>>>> os.detected.release.like.ubuntu: true
>>>> os.detected.classifier: linux-x86_64
>>>>
>>>> Java
>>>> java version "1.8.0_171"
>>>> Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
>>>>
>>>> Maven
>>>> Apache Maven 3.3.9
>>>> Maven home: /usr/share/maven
>>>> Java version: 1.8.0_171, vendor: Oracle Corporation
>>>> Java home: /usr/lib/jvm/java-8-oracle/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
>>>> "unix"
>>>>
>>>>
>>>> Best,
>>>> John.
>>>>
>>>> El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am using now the following pom configuration I got from executing
>>>>> `mvn dependency:tree`:
>>>>>
>>>>> <dependency>
>>>>> <groupId>org.apache.kudu</groupId>
>>>>> <artifactId>kudu-binary</artifactId>
>>>>> <classifier>linux-x86_64</classifier>
>>>>> <version>1.9.0</version>
>>>>> <scope>test</scope>
>>>>> </dependency>
>>>>>
>>>>> When I execute `mvn clen package` on gora-kudu I find that it spawns
>>>>> the following command:
>>>>>
>>>>> kudu-master
>>>>> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
>>>>> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
>>>>> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
>>>>> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
>>>>> --webserver_interface=*127.26.116.190* --webserver_port=0
>>>>> --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false
>>>>> --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
>>>>> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
>>>>> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
>>>>> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
>>>>> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
>>>>> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
>>>>> *127.26.116.188*:42253
>>>>>
>>>>>
>>>>> I highlight the IP addresses because they clearly are not my computer,
>>>>> and I guess that is why the tests can't connect to the the database.
>>>>>
>>>>> Any idea on how to solve this?
>>>>>
>>>>> Thank you!
>>>>>
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>>
>>>>>
>>>>> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi, John.
>>>>>>
>>>>>> I get a core dump from the binary kudu server when trying to run the
>>>>>> tests. Didn't find a log file, but will search thoroughly later. Happened
>>>>>> anytime to you? Does it happens to anyone?
>>>>>>
>>>>>> I am using Ubuntu 18.04
>>>>>>
>>>>>> Thank you!
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
>>>>>> escribió:
>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> I've already made my comments at your PR. Please check them
>>>>>>> carefully and ask me if you need help.
>>>>>>>
>>>>>>> For the documentation, I've checked what you've done. On the other
>>>>>>> hand, I would want to encourage you to write a blog post about your Kudu
>>>>>>> implementation and demonstrate an example of Kudu integration with Gora as
>>>>>>> like a tutorial.
>>>>>>>
>>>>>>> Kind Regards,
>>>>>>> Furkan KAMACI
>>>>>>>
>>>>>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> I have updated my report in the Wiki[1].
>>>>>>>>
>>>>>>>> Also, I have sent a PR with my last commits for review [2]. Please
>>>>>>>> give it a look if you have time.
>>>>>>>>
>>>>>>>> This week, I will continue working on the documentation of the kudu
>>>>>>>> datastore.
>>>>>>>>
>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/apache/gora/pull/178
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> John.
>>>>>>>>
>>>>>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<
>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi John,
>>>>>>>>>
>>>>>>>>> Thanks for the update. I reviewed your code a little bit, it is
>>>>>>>>> looking good. I think tha you should send a PR in order to receive feedback
>>>>>>>>> from other community members.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Carlos
>>>>>>>>>
>>>>>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>
>>>>>>>>>> This week, I will give a look to the documentation of datastores.
>>>>>>>>>>
>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>
>>>>>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a
>>>>>>>>>>> problem I will find an alternative, it is only an utilitary class.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>
>>>>>>>>>>>> I checked out your code and it looks good :)
>>>>>>>>>>>> I found that you use javafx, but that is not present in OpenJDK
>>>>>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I would
>>>>>>>>>>>> suggest to change it.
>>>>>>>>>>>>
>>>>>>>>>>>> Good job, keep it going :)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>>>>>>> DataStores.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks so much for your time and support for this project. I
>>>>>>>>>>>>>>> will work on your comments. Responses inline :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been
>>>>>>>>>>>>>>>> very busy :( I will try to answer your questions :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
>>>>>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I read about
>>>>>>>>>>>>>>>> because until the last version upgrade of Avro this weren't there. I would
>>>>>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in any case
>>>>>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, a design
>>>>>>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type
>>>>>>>>>>>>>>>> so KuduStore converts between the Entity long field <-> Kudu timestamp
>>>>>>>>>>>>>>>> storage?
>>>>>>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
>>>>>>>>>>>>>>> that the best alternative right now. Because, I would add more compatible
>>>>>>>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>>>>>>> also can set time interval
>>>>>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>>>>>>> *> Should theses behaviors be configurable using
>>>>>>>>>>>>>>>> gora.properties file? or just use the default configurations.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do
>>>>>>>>>>>>>>> the job for most use cases. But I will add a configuration in
>>>>>>>>>>>>>>> gora.properties for selecting the other modes and specifying a autoflush
>>>>>>>>>>>>>>> time if needed by the user.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>>> mapping file for this?.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop.
>>>>>>>>>>>>>>>> Apache Gora integrates with Hadoop implementing a custom Map and Reduce
>>>>>>>>>>>>>>>> that allows to get/write Entities directly.
>>>>>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
>>>>>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
>>>>>>>>>>>>>>>> compute the splits (start key---end key) with the location of the split to
>>>>>>>>>>>>>>>> create a colection of partitions [6].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local
>>>>>>>>>>>>>>>> kudu splits, then this method does the needed preparation to allow to "send
>>>>>>>>>>>>>>>> the computation to where the data is locally".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - MongoDB store implementation does not implement
>>>>>>>>>>>>>>>> splitting [7]
>>>>>>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>>>>>>> splitting [8]
>>>>>>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>>>>>>> splitting [9]
>>>>>>>>>>>>>>>> - Accumulo store implementation* does* implement
>>>>>>>>>>>>>>>> splitting [10]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If Kudu has a method to get the different splits for a
>>>>>>>>>>>>>>>> table and its locations, then you will be able to implement the full
>>>>>>>>>>>>>>>> feature.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>>>>>>> in other backends.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About Queries, what I can tell is that Hbase only
>>>>>>>>>>>>>>>> implements "Start key" + "End key" because it has only 2 operations: "get"
>>>>>>>>>>>>>>>> and "scan", and the querying is for "scan" operation, were you want an
>>>>>>>>>>>>>>>> interval (or all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About other topic, I am trying to install Kudu in
>>>>>>>>>>>>>>>> standalone (all in 1 node). Do you use a Cloudera installation or do you
>>>>>>>>>>>>>>>> have a standalone installation? How do you do it? I found some
>>>>>>>>>>>>>>>> instructions, but they talk about compiling Kudu [11]. I was looking for
>>>>>>>>>>>>>>>> something like HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am using an embedded mini-cluster which comes with
>>>>>>>>>>>>>>> compiled binaries and can be used with maven[1] for testing my code. Once I
>>>>>>>>>>>>>>> get it mature enough I think I will be testing the datastore with a docker
>>>>>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>>>> mapping file for this?.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
>>>>>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types.
>>>>>>>>>>>>>>>>>> Please give it a look if you have time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday.
>>>>>>>>>>>>>>>>>>> :D
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on
>>>>>>>>>>>>>>>>>>>> June 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
>>>>>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on
>>>>>>>>>>>>>>>>>>>>>>>> your comments.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso
>>>>>>>>>>>>>>>>>>>>>>>> Nishikawa (<al...@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations
>>>>>>>>>>>>>>>>>>>>>>>>> on the mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes
>>>>>>>>>>>>>>>>>>>>>>>>> with 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite
>>>>>>>>>>>>>>>>>>>>>>>>> new version :)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso
>>>>>>>>>>>>>>>>>>>>>>>>> Nishikawa (<al...@gmail.com>)
>>>>>>>>>>>>>>>>>>>>>>>>> escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go
>>>>>>>>>>>>>>>>>>>>>>>>>> on the Dependency Management of the root pom [2]. Same for [3] and from
>>>>>>>>>>>>>>>>>>>>>>>>>> there, should not set the version there.
>>>>>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4]
>>>>>>>>>>>>>>>>>>>>>>>>>> and from there.
>>>>>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol"
>>>>>>>>>>>>>>>>>>>>>>>>>> RFC 867 and you will need root permission to run it. The default port for
>>>>>>>>>>>>>>>>>>>>>>>>>> kudu is 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality
>>>>>>>>>>>>>>>>>>>>>>>>>> to load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]. This next week I think I should be focusing on the create schema
>>>>>>>>>>>>>>>>>>>>>>>>>>> operation and solving the issue of the partitioning configurations in the
>>>>>>>>>>>>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my
>>>>>>>>>>>>>>>>>>>>>>>>>>> last commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Alfonso,
Please take into a consideration that the property
'gora.datastore.kudu.masterAddress' is overridden in the class
GoraKuduTestDriver [1]. Because KuduTestHarness generates random ports
which need to be configured at runtime. Probably, you should change the
property there too.
I will test my code with a docker container, in order to figure out the
origin of the issue. Please let me know if someone faces this issue when
building the project.
[1]
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/java/org/apache/gora/kudu/GoraKuduTestDriver.java#L41
Cheers,
John.
El vie., 9 ago. 2019 a las 12:22, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi,
>
> I installed a standalone Kudu server (compiled from sources) in a virtual
> machine and configured test/resources/gora.properties to use "nosql:8051"
> master.
>
> The tests freezes waiting for kudu response and a tons of connections to
> the master gets created.
>
> *To the community:* Can someone please clone
> https://github.com/jhnmora000/gora/tree/GORA-485 and build it to see if
> the problem can be reproduced?
>
> I attach snapshots showing how connections stays waiting.
>
> Thank you!
>
> Best Regards,
>
> Alfonso Nishikawa
>
>
>
> El jue., 8 ago. 2019 a las 20:18, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> Tried using Oracle jdk 1.8 and found the same core dump:
>>
>> [INFO] Running org.apache.gora.kudu.store.TestKuduStore
>> [ERROR] Tests run: 44, Failures: 0, Errors: 40, Skipped: 4, Time elapsed:
>> 52.466 s <<< FAILURE! - in org.apache.gora.kudu.store.TestKuduStore
>> [ERROR] testNewInstance(org.apache.gora.kudu.store.TestKuduStore) Time
>> elapsed: 1.834 s <<< ERROR!
>> java.io.IOException: failed to start masters: Unable to start Master at
>> index 0:
>> /tmp/kudu-binary-jar4319751617646651391/kudu-binary-1.9.0-linux-x86_64/bin/kudu-master:
>> process exited on signal 6 (core dumped)
>>
>> (I expected to fail too, since the problem doesn't look like being
>> related to the jvm).
>>
>> Thanks for giving it a look. Don't know what must be the problem :\
>>
>> Best Regards,
>>
>> Alfosno Nishikawa
>>
>>
>> El mar., 6 ago. 2019 a las 4:26, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi Alfonso,
>>>
>>> Unfortunately, I have not been able to reproduce the issue. Maybe it is
>>> related with my Java version (Oracle), I will try with OpenJDK.
>>> Some details about my development environment:
>>>
>>> os.detected.name: linux
>>> os.detected.arch: x86_64
>>> os.detected.version: 4.10
>>> os.detected.version.major: 4
>>> os.detected.version.minor: 10
>>> os.detected.release: linuxmint
>>> os.detected.release.version: 18.3
>>> os.detected.release.like.linuxmint: true
>>> os.detected.release.like.ubuntu: true
>>> os.detected.classifier: linux-x86_64
>>>
>>> Java
>>> java version "1.8.0_171"
>>> Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
>>>
>>> Maven
>>> Apache Maven 3.3.9
>>> Maven home: /usr/share/maven
>>> Java version: 1.8.0_171, vendor: Oracle Corporation
>>> Java home: /usr/lib/jvm/java-8-oracle/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
>>> "unix"
>>>
>>>
>>> Best,
>>> John.
>>>
>>> El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi,
>>>>
>>>> I am using now the following pom configuration I got from executing
>>>> `mvn dependency:tree`:
>>>>
>>>> <dependency>
>>>> <groupId>org.apache.kudu</groupId>
>>>> <artifactId>kudu-binary</artifactId>
>>>> <classifier>linux-x86_64</classifier>
>>>> <version>1.9.0</version>
>>>> <scope>test</scope>
>>>> </dependency>
>>>>
>>>> When I execute `mvn clen package` on gora-kudu I find that it spawns
>>>> the following command:
>>>>
>>>> kudu-master
>>>> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
>>>> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
>>>> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
>>>> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
>>>> --webserver_interface=*127.26.116.190* --webserver_port=0
>>>> --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false
>>>> --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
>>>> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
>>>> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
>>>> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
>>>> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
>>>> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
>>>> *127.26.116.188*:42253
>>>>
>>>>
>>>> I highlight the IP addresses because they clearly are not my computer,
>>>> and I guess that is why the tests can't connect to the the database.
>>>>
>>>> Any idea on how to solve this?
>>>>
>>>> Thank you!
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Alfonso Nishikawa
>>>>
>>>>
>>>>
>>>> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi, John.
>>>>>
>>>>> I get a core dump from the binary kudu server when trying to run the
>>>>> tests. Didn't find a log file, but will search thoroughly later. Happened
>>>>> anytime to you? Does it happens to anyone?
>>>>>
>>>>> I am using Ubuntu 18.04
>>>>>
>>>>> Thank you!
>>>>>
>>>>> Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
>>>>> escribió:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> I've already made my comments at your PR. Please check them carefully
>>>>>> and ask me if you need help.
>>>>>>
>>>>>> For the documentation, I've checked what you've done. On the other
>>>>>> hand, I would want to encourage you to write a blog post about your Kudu
>>>>>> implementation and demonstrate an example of Kudu integration with Gora as
>>>>>> like a tutorial.
>>>>>>
>>>>>> Kind Regards,
>>>>>> Furkan KAMACI
>>>>>>
>>>>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I have updated my report in the Wiki[1].
>>>>>>>
>>>>>>> Also, I have sent a PR with my last commits for review [2]. Please
>>>>>>> give it a look if you have time.
>>>>>>>
>>>>>>> This week, I will continue working on the documentation of the kudu
>>>>>>> datastore.
>>>>>>>
>>>>>>> Please let me know if you have suggestions.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/apache/gora/pull/178
>>>>>>>
>>>>>>> Best,
>>>>>>> John.
>>>>>>>
>>>>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<
>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> Thanks for the update. I reviewed your code a little bit, it is
>>>>>>>> looking good. I think tha you should send a PR in order to receive feedback
>>>>>>>> from other community members.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Carlos
>>>>>>>>
>>>>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>>>
>>>>>>>>> This week, I will give a look to the documentation of datastores.
>>>>>>>>>
>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>
>>>>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem
>>>>>>>>>> I will find an alternative, it is only an utilitary class.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi, John.
>>>>>>>>>>>
>>>>>>>>>>> I checked out your code and it looks good :)
>>>>>>>>>>> I found that you use javafx, but that is not present in OpenJDK
>>>>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I would
>>>>>>>>>>> suggest to change it.
>>>>>>>>>>>
>>>>>>>>>>> Good job, keep it going :)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi.
>>>>>>>>>>>>
>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>
>>>>>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>>>>>> DataStores.
>>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all
>>>>>>>>>>>>>
>>>>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>>>>>> class.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks so much for your time and support for this project. I
>>>>>>>>>>>>>> will work on your comments. Responses inline :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very
>>>>>>>>>>>>>>> busy :( I will try to answer your questions :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
>>>>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I read about
>>>>>>>>>>>>>>> because until the last version upgrade of Avro this weren't there. I would
>>>>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in any case
>>>>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, a design
>>>>>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
>>>>>>>>>>>>>> that the best alternative right now. Because, I would add more compatible
>>>>>>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>>>>>> also can set time interval
>>>>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>>>>>> *> Should theses behaviors be configurable using
>>>>>>>>>>>>>>> gora.properties file? or just use the default configurations.*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do
>>>>>>>>>>>>>> the job for most use cases. But I will add a configuration in
>>>>>>>>>>>>>> gora.properties for selecting the other modes and specifying a autoflush
>>>>>>>>>>>>>> time if needed by the user.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>> mapping file for this?.*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache
>>>>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and Reduce that
>>>>>>>>>>>>>>> allows to get/write Entities directly.
>>>>>>>>>>>>>>> You can take a look at HBase's implementation [4], which
>>>>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to
>>>>>>>>>>>>>>> compute the splits (start key---end key) with the location of the split to
>>>>>>>>>>>>>>> create a colection of partitions [6].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local
>>>>>>>>>>>>>>> kudu splits, then this method does the needed preparation to allow to "send
>>>>>>>>>>>>>>> the computation to where the data is locally".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - MongoDB store implementation does not implement
>>>>>>>>>>>>>>> splitting [7]
>>>>>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>>>>>> splitting [8]
>>>>>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>>>>>> splitting [9]
>>>>>>>>>>>>>>> - Accumulo store implementation* does* implement
>>>>>>>>>>>>>>> splitting [10]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If Kudu has a method to get the different splits for a table
>>>>>>>>>>>>>>> and its locations, then you will be able to implement the full feature.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>>>>>> in other backends.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone
>>>>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do you have a
>>>>>>>>>>>>>>> standalone installation? How do you do it? I found some instructions, but
>>>>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for something like
>>>>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed
>>>>>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the expected
>>>>>>>>>>>>>>>> behavior of this method?, should I use the partition definition in the xml
>>>>>>>>>>>>>>>> mapping file for this?.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of
>>>>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types.
>>>>>>>>>>>>>>>>> Please give it a look if you have time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday.
>>>>>>>>>>>>>>>>>> :D
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on
>>>>>>>>>>>>>>>>>>> June 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
>>>>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on
>>>>>>>>>>>>>>>>>>>>>>> your comments.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>> (<al...@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations
>>>>>>>>>>>>>>>>>>>>>>>> on the mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso
>>>>>>>>>>>>>>>>>>>>>>>> Nishikawa (<al...@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go
>>>>>>>>>>>>>>>>>>>>>>>>> on the Dependency Management of the root pom [2]. Same for [3] and from
>>>>>>>>>>>>>>>>>>>>>>>>> there, should not set the version there.
>>>>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and
>>>>>>>>>>>>>>>>>>>>>>>>> from there.
>>>>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC
>>>>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it. The default port for kudu
>>>>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to
>>>>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]. This next week I think I should be focusing on the create schema
>>>>>>>>>>>>>>>>>>>>>>>>>> operation and solving the issue of the partitioning configurations in the
>>>>>>>>>>>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my
>>>>>>>>>>>>>>>>>>>>>>>>>> last commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Alfonso,
Unfortunately, I have not been able to reproduce the issue. Maybe it is
related with my Java version (Oracle), I will try with OpenJDK.
Some details about my development environment:
os.detected.name: linux
os.detected.arch: x86_64
os.detected.version: 4.10
os.detected.version.major: 4
os.detected.version.minor: 10
os.detected.release: linuxmint
os.detected.release.version: 18.3
os.detected.release.like.linuxmint: true
os.detected.release.like.ubuntu: true
os.detected.classifier: linux-x86_64
Java
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
Maven
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_171, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family:
"unix"
Best,
John.
El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi,
>
> I am using now the following pom configuration I got from executing `mvn
> dependency:tree`:
>
> <dependency>
> <groupId>org.apache.kudu</groupId>
> <artifactId>kudu-binary</artifactId>
> <classifier>linux-x86_64</classifier>
> <version>1.9.0</version>
> <scope>test</scope>
> </dependency>
>
> When I execute `mvn clen package` on gora-kudu I find that it spawns the
> following command:
>
> kudu-master
> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
> --webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync
> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none
> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
> *127.26.116.188*:42253
>
>
> I highlight the IP addresses because they clearly are not my computer, and
> I guess that is why the tests can't connect to the the database.
>
> Any idea on how to solve this?
>
> Thank you!
>
>
> Best Regards,
>
> Alfonso Nishikawa
>
>
>
> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> I get a core dump from the binary kudu server when trying to run the
>> tests. Didn't find a log file, but will search thoroughly later. Happened
>> anytime to you? Does it happens to anyone?
>>
>> I am using Ubuntu 18.04
>>
>> Thank you!
>>
>> Regards,
>>
>> Alfonso Nishikawa
>>
>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
>> escribió:
>>
>>> Hi John,
>>>
>>> I've already made my comments at your PR. Please check them carefully
>>> and ask me if you need help.
>>>
>>> For the documentation, I've checked what you've done. On the other hand,
>>> I would want to encourage you to write a blog post about your Kudu
>>> implementation and demonstrate an example of Kudu integration with Gora as
>>> like a tutorial.
>>>
>>> Kind Regards,
>>> Furkan KAMACI
>>>
>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>> I have updated my report in the Wiki[1].
>>>>
>>>> Also, I have sent a PR with my last commits for review [2]. Please give
>>>> it a look if you have time.
>>>>
>>>> This week, I will continue working on the documentation of the kudu
>>>> datastore.
>>>>
>>>> Please let me know if you have suggestions.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/apache/gora/pull/178
>>>>
>>>> Best,
>>>> John.
>>>>
>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi John,
>>>>>
>>>>> Thanks for the update. I reviewed your code a little bit, it is
>>>>> looking good. I think tha you should send a PR in order to receive feedback
>>>>> from other community members.
>>>>>
>>>>> Best,
>>>>> Carlos
>>>>>
>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>>> my branch [2]. Please give it a look if you have time.
>>>>>>
>>>>>> This week, I will give a look to the documentation of datastores.
>>>>>>
>>>>>> Please let me know if you have suggestions.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Cheers,
>>>>>> John
>>>>>>
>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi Alfonso,
>>>>>>>
>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I
>>>>>>> will find an alternative, it is only an utilitary class.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi, John.
>>>>>>>>
>>>>>>>> I checked out your code and it looks good :)
>>>>>>>> I found that you use javafx, but that is not present in OpenJDK and
>>>>>>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>>>>>>> change it.
>>>>>>>>
>>>>>>>> Good job, keep it going :)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Alfonso Nishikawa
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>>>
>>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>>> DataStores.
>>>>>>>>>
>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi all
>>>>>>>>>>
>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last
>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>
>>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>>> class.
>>>>>>>>>>
>>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> John.
>>>>>>>>>>
>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>>
>>>>>>>>>>> Thanks so much for your time and support for this project. I
>>>>>>>>>>> will work on your comments. Responses inline :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very
>>>>>>>>>>>> busy :( I will try to answer your questions :)
>>>>>>>>>>>>
>>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>>
>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a
>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I read about
>>>>>>>>>>>> because until the last version upgrade of Avro this weren't there. I would
>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in any case
>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, a design
>>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>>
>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion
>>>>>>>>>>> that the best alternative right now. Because, I would add more compatible
>>>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>>> also can set time interval
>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>>> *> Should theses behaviors be configurable using
>>>>>>>>>>>> gora.properties file? or just use the default configurations.*
>>>>>>>>>>>>
>>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the
>>>>>>>>>>> job for most use cases. But I will add a configuration in gora.properties
>>>>>>>>>>> for selecting the other modes and specifying a autoflush time if needed
>>>>>>>>>>> by the user.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>>> for this?.*
>>>>>>>>>>>>
>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache
>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and Reduce that
>>>>>>>>>>>> allows to get/write Entities directly.
>>>>>>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>>>>>>> split to create a colection of partitions [6].
>>>>>>>>>>>>
>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>>>>>> computation to where the data is locally".
>>>>>>>>>>>>
>>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>>
>>>>>>>>>>>> - MongoDB store implementation does not implement splitting
>>>>>>>>>>>> [7]
>>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>>> splitting [8]
>>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>>> splitting [9]
>>>>>>>>>>>> - Accumulo store implementation* does* implement splitting
>>>>>>>>>>>> [10]
>>>>>>>>>>>>
>>>>>>>>>>>> If Kudu has a method to get the different splits for a table
>>>>>>>>>>>> and its locations, then you will be able to implement the full feature.
>>>>>>>>>>>>
>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>>> in other backends.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone
>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do you have a
>>>>>>>>>>>> standalone installation? How do you do it? I found some instructions, but
>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for something like
>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1] -
>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>>> [2] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>>> [3] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>>> [4] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>>> [5] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>>> [6] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>>> [7] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>>> [8] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>>> [9] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>>> [10] -
>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>>> time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>>>> for this?.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please
>>>>>>>>>>>>>> give it a look if you have time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1].
>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on
>>>>>>>>>>>>>>>>>>>>> the mapping file.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on
>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and
>>>>>>>>>>>>>>>>>>>>>> from there.
>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC
>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it. The default port for kudu
>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to
>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1].
>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the create schema operation
>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning configurations in the mapping
>>>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi,
I am using now the following pom configuration I got from executing `mvn
dependency:tree`:
<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-binary</artifactId>
<classifier>linux-x86_64</classifier>
<version>1.9.0</version>
<scope>test</scope>
</dependency>
When I execute `mvn clen package` on gora-kudu I find that it spawns the
following command:
kudu-master
--fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal
--fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data
--block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024
--tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535
--webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync
--ipki_server_key_size=1024 --enable_minidumps=false --redact=none
--metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1
--log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs
--server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb
--server_dump_info_format=pb --rpc_server_allow_ephemeral_ports
--unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true
--master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913,
*127.26.116.188*:42253
I highlight the IP addresses because they clearly are not my computer, and
I guess that is why the tests can't connect to the the database.
Any idea on how to solve this?
Thank you!
Best Regards,
Alfonso Nishikawa
El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi, John.
>
> I get a core dump from the binary kudu server when trying to run the
> tests. Didn't find a log file, but will search thoroughly later. Happened
> anytime to you? Does it happens to anyone?
>
> I am using Ubuntu 18.04
>
> Thank you!
>
> Regards,
>
> Alfonso Nishikawa
>
> El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com>
> escribió:
>
>> Hi John,
>>
>> I've already made my comments at your PR. Please check them carefully and
>> ask me if you need help.
>>
>> For the documentation, I've checked what you've done. On the other hand,
>> I would want to encourage you to write a blog post about your Kudu
>> implementation and demonstrate an example of Kudu integration with Gora as
>> like a tutorial.
>>
>> Kind Regards,
>> Furkan KAMACI
>>
>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com> wrote:
>>
>>> Hi all.
>>>
>>> I have updated my report in the Wiki[1].
>>>
>>> Also, I have sent a PR with my last commits for review [2]. Please give
>>> it a look if you have time.
>>>
>>> This week, I will continue working on the documentation of the kudu
>>> datastore.
>>>
>>> Please let me know if you have suggestions.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/apache/gora/pull/178
>>>
>>> Best,
>>> John.
>>>
>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
>>> escribió:
>>>
>>>> Hi John,
>>>>
>>>> Thanks for the update. I reviewed your code a little bit, it is looking
>>>> good. I think tha you should send a PR in order to receive feedback from
>>>> other community members.
>>>>
>>>> Best,
>>>> Carlos
>>>>
>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>> my branch [2]. Please give it a look if you have time.
>>>>>
>>>>> This week, I will give a look to the documentation of datastores.
>>>>>
>>>>> Please let me know if you have suggestions.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Cheers,
>>>>> John
>>>>>
>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi Alfonso,
>>>>>>
>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I
>>>>>> will find an alternative, it is only an utilitary class.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi, John.
>>>>>>>
>>>>>>> I checked out your code and it looks good :)
>>>>>>> I found that you use javafx, but that is not present in OpenJDK and
>>>>>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>>>>>> change it.
>>>>>>>
>>>>>>> Good job, keep it going :)
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Alfonso Nishikawa
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>>
>>>>>>>> This week, I will give a look to the map reduce tests for
>>>>>>>> DataStores.
>>>>>>>>
>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all
>>>>>>>>>
>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>>>
>>>>>>>>> This week, I will be working in the getPartitions and
>>>>>>>>> deleteByQuery methods and testing the other tests in the DataStoreTestBase
>>>>>>>>> class.
>>>>>>>>>
>>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> John.
>>>>>>>>>
>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi Alfonso,
>>>>>>>>>>
>>>>>>>>>> Thanks so much for your time and support for this project. I will
>>>>>>>>>> work on your comments. Responses inline :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi, John.
>>>>>>>>>>>
>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very
>>>>>>>>>>> busy :( I will try to answer your questions :)
>>>>>>>>>>>
>>>>>>>>>>> *> In the Employee example there is a field called
>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS datatype
>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the java world
>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a Timestamp.
>>>>>>>>>>> So, I was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>>
>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date"
>>>>>>>>>>> type with an underlying "int" [1]. It's the first time I read about because
>>>>>>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>>
>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion that
>>>>>>>>>> the best alternative right now. Because, I would add more compatible
>>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>>> also can set time interval
>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>>> for automatic flush.*
>>>>>>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>>>>>>> file? or just use the default configurations.*
>>>>>>>>>>>
>>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the
>>>>>>>>>> job for most use cases. But I will add a configuration in gora.properties
>>>>>>>>>> for selecting the other modes and specifying a autoflush time if needed
>>>>>>>>>> by the user.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>> for this?.*
>>>>>>>>>>>
>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache
>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and Reduce that
>>>>>>>>>>> allows to get/write Entities directly.
>>>>>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>>>>>> split to create a colection of partitions [6].
>>>>>>>>>>>
>>>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>>>>> computation to where the data is locally".
>>>>>>>>>>>
>>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>>
>>>>>>>>>>> - MongoDB store implementation does not implement splitting
>>>>>>>>>>> [7]
>>>>>>>>>>> - Cassandra store implementation does not implement
>>>>>>>>>>> splitting [8]
>>>>>>>>>>> - Aerospike store implementation does not implement
>>>>>>>>>>> splitting [9]
>>>>>>>>>>> - Accumulo store implementation* does* implement splitting
>>>>>>>>>>> [10]
>>>>>>>>>>>
>>>>>>>>>>> If Kudu has a method to get the different splits for a table and
>>>>>>>>>>> its locations, then you will be able to implement the full feature.
>>>>>>>>>>>
>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I will check whether Kudu has these features in order to
>>>>>>>>>> implement this method. If not I will use the default implementation found
>>>>>>>>>> in other backends.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone
>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do you have a
>>>>>>>>>>> standalone installation? How do you do it? I found some instructions, but
>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for something like
>>>>>>>>>>> HBase, that it is unzip + execute "hbase start".
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>>> [2] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>>> [3] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>>> [4] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>>> [5] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>>> [6] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>>> [7] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>>> [8] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>>> [9] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>>> [10] -
>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>
>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I
>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look if you have
>>>>>>>>>>>> time.
>>>>>>>>>>>>
>>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>>> for this?.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> John.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I
>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>
>>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please
>>>>>>>>>>>>> give it a look if you have time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also,
>>>>>>>>>>>>>>>> I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I
>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation I think I will
>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on
>>>>>>>>>>>>>>>>>>>> the mapping file.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on
>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and
>>>>>>>>>>>>>>>>>>>>> from there.
>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867
>>>>>>>>>>>>>>>>>>>>> and you will need root permission to run it. The default port for kudu is
>>>>>>>>>>>>>>>>>>>>> 7051, isn't it?
>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to
>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's store [7] in you
>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your readMapping at [9], so
>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1].
>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the create schema operation
>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning configurations in the mapping
>>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
I get a core dump from the binary kudu server when trying to run the tests.
Didn't find a log file, but will search thoroughly later. Happened anytime
to you? Does it happens to anyone?
I am using Ubuntu 18.04
Thank you!
Regards,
Alfonso Nishikawa
El dom., 4 ago. 2019 20:10, Furkan KAMACI <fu...@gmail.com> escribió:
> Hi John,
>
> I've already made my comments at your PR. Please check them carefully and
> ask me if you need help.
>
> For the documentation, I've checked what you've done. On the other hand, I
> would want to encourage you to write a blog post about your Kudu
> implementation and demonstrate an example of Kudu integration with Gora as
> like a tutorial.
>
> Kind Regards,
> Furkan KAMACI
>
> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com> wrote:
>
>> Hi all.
>>
>> I have updated my report in the Wiki[1].
>>
>> Also, I have sent a PR with my last commits for review [2]. Please give
>> it a look if you have time.
>>
>> This week, I will continue working on the documentation of the kudu
>> datastore.
>>
>> Please let me know if you have suggestions.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/apache/gora/pull/178
>>
>> Best,
>> John.
>>
>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
>> escribió:
>>
>>> Hi John,
>>>
>>> Thanks for the update. I reviewed your code a little bit, it is looking
>>> good. I think tha you should send a PR in order to receive feedback from
>>> other community members.
>>>
>>> Best,
>>> Carlos
>>>
>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>> my branch [2]. Please give it a look if you have time.
>>>>
>>>> This week, I will give a look to the documentation of datastores.
>>>>
>>>> Please let me know if you have suggestions.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Cheers,
>>>> John
>>>>
>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Alfonso,
>>>>>
>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I
>>>>> will find an alternative, it is only an utilitary class.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi, John.
>>>>>>
>>>>>> I checked out your code and it looks good :)
>>>>>> I found that you use javafx, but that is not present in OpenJDK and
>>>>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>>>>> change it.
>>>>>>
>>>>>> Good job, keep it going :)
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>
>>>>>>> This week, I will give a look to the map reduce tests for DataStores.
>>>>>>>
>>>>>>> Please let me know if you have suggestions.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>>
>>>>>>>> This week, I will be working in the getPartitions and deleteByQuery
>>>>>>>> methods and testing the other tests in the DataStoreTestBase class.
>>>>>>>>
>>>>>>>> Please let me know if you have suggestions.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> John.
>>>>>>>>
>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi Alfonso,
>>>>>>>>>
>>>>>>>>> Thanks so much for your time and support for this project. I will
>>>>>>>>> work on your comments. Responses inline :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi, John.
>>>>>>>>>>
>>>>>>>>>> Sorry for the delay, I am changing work and I have been very busy
>>>>>>>>>> :( I will try to answer your questions :)
>>>>>>>>>>
>>>>>>>>>> *> In the Employee example there is a field called 'dateOfBirth'.
>>>>>>>>>> I tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>>>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>>>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>>>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>>
>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date"
>>>>>>>>>> type with an underlying "int" [1]. It's the first time I read about because
>>>>>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>>>>>> improvement would be great, though :)
>>>>>>>>>>
>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>>> - Is there any other approach?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion that
>>>>>>>>> the best alternative right now. Because, I would add more compatible
>>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>>> also can set time interval
>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>>> for automatic flush.*
>>>>>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>>>>>> file? or just use the default configurations.*
>>>>>>>>>>
>>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the
>>>>>>>>> job for most use cases. But I will add a configuration in gora.properties
>>>>>>>>> for selecting the other modes and specifying a autoflush time if needed
>>>>>>>>> by the user.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>> for this?.*
>>>>>>>>>>
>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>>>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>>>>>>> get/write Entities directly.
>>>>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>>>>> split to create a colection of partitions [6].
>>>>>>>>>>
>>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>>>> computation to where the data is locally".
>>>>>>>>>>
>>>>>>>>>> In any case, you can see that:
>>>>>>>>>>
>>>>>>>>>> - MongoDB store implementation does not implement splitting
>>>>>>>>>> [7]
>>>>>>>>>> - Cassandra store implementation does not implement splitting
>>>>>>>>>> [8]
>>>>>>>>>> - Aerospike store implementation does not implement splitting
>>>>>>>>>> [9]
>>>>>>>>>> - Accumulo store implementation* does* implement splitting
>>>>>>>>>> [10]
>>>>>>>>>>
>>>>>>>>>> If Kudu has a method to get the different splits for a table and
>>>>>>>>>> its locations, then you will be able to implement the full feature.
>>>>>>>>>>
>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't
>>>>>>>>>> elaborated much, so if you find you need more information let me know :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I will check whether Kudu has these features in order to implement
>>>>>>>>> this method. If not I will use the default implementation found in other
>>>>>>>>> backends.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> About other topic, I am trying to install Kudu in standalone (all
>>>>>>>>>> in 1 node). Do you use a Cloudera installation or do you have a standalone
>>>>>>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>>>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>>>>>>> is unzip + execute "hbase start".
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>>> [2] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>>> [3] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>>> [4] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>>> [5] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>>> [6] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>>> [7] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>>> [8] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>>> [9] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>>> [10] -
>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>>>>>>> escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed
>>>>>>>>>>> my last commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>>
>>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>>
>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>>> for this?.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> John.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>
>>>>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>>
>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed
>>>>>>>>>>>> my last commits to my branch [2].
>>>>>>>>>>>>
>>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>>> implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please
>>>>>>>>>>>> give it a look if you have time.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also,
>>>>>>>>>>>>>>> I pushed my last commits to my branch [2].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed
>>>>>>>>>>>>>>>>> that my code is lacking some javadoc documentation I think I will be
>>>>>>>>>>>>>>>>> working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on
>>>>>>>>>>>>>>>>>>> the mapping file.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on
>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867
>>>>>>>>>>>>>>>>>>>> and you will need root permission to run it. The default port for kudu is
>>>>>>>>>>>>>>>>>>>> 7051, isn't it?
>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load
>>>>>>>>>>>>>>>>>>>> the mapping from configuration as in HBase's store [7] in you KuduStore
>>>>>>>>>>>>>>>>>>>> [8]. This will have implications on your readMapping at [9], so take a look
>>>>>>>>>>>>>>>>>>>> at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1].
>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the create schema operation
>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning configurations in the mapping
>>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi John,
I've already made my comments at your PR. Please check them carefully and
ask me if you need help.
For the documentation, I've checked what you've done. On the other hand, I
would want to encourage you to write a blog post about your Kudu
implementation and demonstrate an example of Kudu integration with Gora as
like a tutorial.
Kind Regards,
Furkan KAMACI
On Sun, Aug 4, 2019 at 1:59 AM John Mora <jh...@gmail.com> wrote:
> Hi all.
>
> I have updated my report in the Wiki[1].
>
> Also, I have sent a PR with my last commits for review [2]. Please give it
> a look if you have time.
>
> This week, I will continue working on the documentation of the kudu
> datastore.
>
> Please let me know if you have suggestions.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/apache/gora/pull/178
>
> Best,
> John.
>
> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
> escribió:
>
>> Hi John,
>>
>> Thanks for the update. I reviewed your code a little bit, it is looking
>> good. I think tha you should send a PR in order to receive feedback from
>> other community members.
>>
>> Best,
>> Carlos
>>
>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>>> branch [2]. Please give it a look if you have time.
>>>
>>> This week, I will give a look to the documentation of datastores.
>>>
>>> Please let me know if you have suggestions.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Cheers,
>>> John
>>>
>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Alfonso,
>>>>
>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I
>>>> will find an alternative, it is only an utilitary class.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi, John.
>>>>>
>>>>> I checked out your code and it looks good :)
>>>>> I found that you use javafx, but that is not present in OpenJDK and
>>>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>>>> change it.
>>>>>
>>>>> Good job, keep it going :)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>>> my branch [2]. Please give it a look if you have time.
>>>>>>
>>>>>> This week, I will give a look to the map reduce tests for DataStores.
>>>>>>
>>>>>> Please let me know if you have suggestions.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all
>>>>>>>
>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits
>>>>>>> to my branch [2]. Please give it a look if you have time.
>>>>>>>
>>>>>>> This week, I will be working in the getPartitions and deleteByQuery
>>>>>>> methods and testing the other tests in the DataStoreTestBase class.
>>>>>>>
>>>>>>> Please let me know if you have suggestions.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Best,
>>>>>>> John.
>>>>>>>
>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Alfonso,
>>>>>>>>
>>>>>>>> Thanks so much for your time and support for this project. I will
>>>>>>>> work on your comments. Responses inline :)
>>>>>>>>
>>>>>>>>
>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi, John.
>>>>>>>>>
>>>>>>>>> Sorry for the delay, I am changing work and I have been very busy
>>>>>>>>> :( I will try to answer your questions :)
>>>>>>>>>
>>>>>>>>> *> In the Employee example there is a field called 'dateOfBirth'.
>>>>>>>>> I tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>>
>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date"
>>>>>>>>> type with an underlying "int" [1]. It's the first time I read about because
>>>>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>>>>> improvement would be great, though :)
>>>>>>>>>
>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>>> - Is there any other approach?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion that
>>>>>>>> the best alternative right now. Because, I would add more compatible
>>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>>> also can set time interval
>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>>> for automatic flush.*
>>>>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>>>>> file? or just use the default configurations.*
>>>>>>>>>
>>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the
>>>>>>>> job for most use cases. But I will add a configuration in gora.properties
>>>>>>>> for selecting the other modes and specifying a autoflush time if needed
>>>>>>>> by the user.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>> for this?.*
>>>>>>>>>
>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>>>>>> get/write Entities directly.
>>>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>>>> split to create a colection of partitions [6].
>>>>>>>>>
>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>>> computation to where the data is locally".
>>>>>>>>>
>>>>>>>>> In any case, you can see that:
>>>>>>>>>
>>>>>>>>> - MongoDB store implementation does not implement splitting [7]
>>>>>>>>> - Cassandra store implementation does not implement splitting
>>>>>>>>> [8]
>>>>>>>>> - Aerospike store implementation does not implement splitting
>>>>>>>>> [9]
>>>>>>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>>>>>>
>>>>>>>>> If Kudu has a method to get the different splits for a table and
>>>>>>>>> its locations, then you will be able to implement the full feature.
>>>>>>>>>
>>>>>>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>>>>>>> much, so if you find you need more information let me know :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I will check whether Kudu has these features in order to implement
>>>>>>>> this method. If not I will use the default implementation found in other
>>>>>>>> backends.
>>>>>>>>
>>>>>>>>
>>>>>>>>> About Queries, what I can tell is that Hbase only implements
>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" and "scan",
>>>>>>>>> and the querying is for "scan" operation, were you want an interval (or
>>>>>>>>> all) of the rows. Does Kudu have more querying functionality?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>>
>>>>>>>>
>>>>>>>>> About other topic, I am trying to install Kudu in standalone (all
>>>>>>>>> in 1 node). Do you use a Cloudera installation or do you have a standalone
>>>>>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>>>>>> is unzip + execute "hbase start".
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>>> kinda noob for compiling it myself.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>>
>>>>>>>>
>>>>>>>>> Good job and thank you!! :)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>>> [2] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>>> [3] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>>> [4] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>>> [5] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>>> [6] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>>> [7] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>>> [8] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>>> [9] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>>> [10] -
>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>>>>>> escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed
>>>>>>>>>> my last commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>>
>>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>>
>>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>>> for this?.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> John.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>>
>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed
>>>>>>>>>>> my last commits to my branch [2].
>>>>>>>>>>>
>>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>>> implementation.
>>>>>>>>>>>
>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please
>>>>>>>>>>> give it a look if you have time.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> John.
>>>>>>>>>>>>
>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>
>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June
>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>
>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the
>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I
>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, which is very
>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should replace [1] for [2]
>>>>>>>>>>>>>>> in order to avoid code duplication.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>>> [2]
>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed
>>>>>>>>>>>>>>>> that my code is lacking some javadoc documentation I think I will be
>>>>>>>>>>>>>>>> working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with
>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven 3 in a quite new
>>>>>>>>>>>>>>>>>> version :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867
>>>>>>>>>>>>>>>>>>> and you will need root permission to run it. The default port for kudu is
>>>>>>>>>>>>>>>>>>> 7051, isn't it?
>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load
>>>>>>>>>>>>>>>>>>> the mapping from configuration as in HBase's store [7] in you KuduStore
>>>>>>>>>>>>>>>>>>> [8]. This will have implications on your readMapping at [9], so take a look
>>>>>>>>>>>>>>>>>>> at the one for HBase at [10]
>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1].
>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the create schema operation
>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning configurations in the mapping
>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
I have updated my report in the Wiki[1].
Also, I have sent a PR with my last commits for review [2]. Please give it
a look if you have time.
This week, I will continue working on the documentation of the kudu
datastore.
Please let me know if you have suggestions.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/apache/gora/pull/178
Best,
John.
El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<ca...@gmail.com>)
escribió:
> Hi John,
>
> Thanks for the update. I reviewed your code a little bit, it is looking
> good. I think tha you should send a PR in order to receive feedback from
> other community members.
>
> Best,
> Carlos
>
> El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>> branch [2]. Please give it a look if you have time.
>>
>> This week, I will give a look to the documentation of datastores.
>>
>> Please let me know if you have suggestions.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Cheers,
>> John
>>
>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi Alfonso,
>>>
>>> Yes, I was using this class javafx.util.Pair. It is not a problem I will
>>> find an alternative, it is only an utilitary class.
>>>
>>> Thanks,
>>> John
>>>
>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi, John.
>>>>
>>>> I checked out your code and it looks good :)
>>>> I found that you use javafx, but that is not present in OpenJDK and
>>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>>> change it.
>>>>
>>>> Good job, keep it going :)
>>>>
>>>> Regards,
>>>>
>>>> Alfonso Nishikawa
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi.
>>>>>
>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>> my branch [2]. Please give it a look if you have time.
>>>>>
>>>>> This week, I will give a look to the map reduce tests for DataStores.
>>>>>
>>>>> Please let me know if you have suggestions.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>>> my branch [2]. Please give it a look if you have time.
>>>>>>
>>>>>> This week, I will be working in the getPartitions and deleteByQuery
>>>>>> methods and testing the other tests in the DataStoreTestBase class.
>>>>>>
>>>>>> Please let me know if you have suggestions.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Best,
>>>>>> John.
>>>>>>
>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi Alfonso,
>>>>>>>
>>>>>>> Thanks so much for your time and support for this project. I will
>>>>>>> work on your comments. Responses inline :)
>>>>>>>
>>>>>>>
>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi, John.
>>>>>>>>
>>>>>>>> Sorry for the delay, I am changing work and I have been very busy
>>>>>>>> :( I will try to answer your questions :)
>>>>>>>>
>>>>>>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>>>>>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>>
>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date"
>>>>>>>> type with an underlying "int" [1]. It's the first time I read about because
>>>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>>>> improvement would be great, though :)
>>>>>>>>
>>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>>> - Is there any other approach?
>>>>>>>>
>>>>>>>
>>>>>>> I think that Entity long field <-> Kudu timestamp conversion that
>>>>>>> the best alternative right now. Because, I would add more compatible
>>>>>>> datatypes to the mapping parameters which users can use. And this
>>>>>>> conversion should not be dificult to implement in my opinion. Also, the new
>>>>>>> Date datatype of avro could be implemented in newer versions because it
>>>>>>> would need further analysis in other datastores too. I will work on that.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>>> also can set time interval
>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>>> for automatic flush.*
>>>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>>>> file? or just use the default configurations.*
>>>>>>>>
>>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>>
>>>>>>>
>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the job
>>>>>>> for most use cases. But I will add a configuration in gora.properties for
>>>>>>> selecting the other modes and specifying a autoflush time if needed by
>>>>>>> the user.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>> for this?.*
>>>>>>>>
>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>>>>> get/write Entities directly.
>>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>>> split to create a colection of partitions [6].
>>>>>>>>
>>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>>> computation to where the data is locally".
>>>>>>>>
>>>>>>>> In any case, you can see that:
>>>>>>>>
>>>>>>>> - MongoDB store implementation does not implement splitting [7]
>>>>>>>> - Cassandra store implementation does not implement splitting
>>>>>>>> [8]
>>>>>>>> - Aerospike store implementation does not implement splitting
>>>>>>>> [9]
>>>>>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>>>>>
>>>>>>>> If Kudu has a method to get the different splits for a table and
>>>>>>>> its locations, then you will be able to implement the full feature.
>>>>>>>>
>>>>>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>>>>>> much, so if you find you need more information let me know :)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> I will check whether Kudu has these features in order to implement
>>>>>>> this method. If not I will use the default implementation found in other
>>>>>>> backends.
>>>>>>>
>>>>>>>
>>>>>>>> About Queries, what I can tell is that Hbase only implements "Start
>>>>>>>> key" + "End key" because it has only 2 operations: "get" and "scan", and
>>>>>>>> the querying is for "scan" operation, were you want an interval (or all) of
>>>>>>>> the rows. Does Kudu have more querying functionality?
>>>>>>>>
>>>>>>>>
>>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>>
>>>>>>>
>>>>>>>> About other topic, I am trying to install Kudu in standalone (all
>>>>>>>> in 1 node). Do you use a Cloudera installation or do you have a standalone
>>>>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>>>>> is unzip + execute "hbase start".
>>>>>>>>
>>>>>>>>
>>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>>> kinda noob for compiling it myself.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>>
>>>>>>>
>>>>>>>> Good job and thank you!! :)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Alfonso Nishikawa
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>>> [2] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>>> [3] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>>> [4] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>>> [5] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>>> [6] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>>> [7] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>>> [8] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>>> [9] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>>> [10] -
>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed
>>>>>>>>> my last commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>>
>>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>>
>>>>>>>>> Also, while reviewing the datastore interface I noticed this
>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>>>> for this?.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> John.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>>> confidence to the project and me.
>>>>>>>>>>
>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed
>>>>>>>>>> my last commits to my branch [2].
>>>>>>>>>>
>>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>>> implementation.
>>>>>>>>>>
>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please give
>>>>>>>>>> it a look if you have time.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi Carlos.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> John.
>>>>>>>>>>>
>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi John
>>>>>>>>>>>>
>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Carlos
>>>>>>>>>>>>
>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>>
>>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed
>>>>>>>>>>>>>> that you have created an Enumeration for datatypes, which is very similar
>>>>>>>>>>>>>> to the kudu-client's [2]. Probably you should replace [1] for [2] in order
>>>>>>>>>>>>>> to avoid code duplication.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed
>>>>>>>>>>>>>>> that my code is lacking some javadoc documentation I think I will be
>>>>>>>>>>>>>>> working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0,
>>>>>>>>>>>>>>>>> far from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867
>>>>>>>>>>>>>>>>>> and you will need root permission to run it. The default port for kudu is
>>>>>>>>>>>>>>>>>> 7051, isn't it?
>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load
>>>>>>>>>>>>>>>>>> the mapping from configuration as in HBase's store [7] in you KuduStore
>>>>>>>>>>>>>>>>>> [8]. This will have implications on your readMapping at [9], so take a look
>>>>>>>>>>>>>>>>>> at the one for HBase at [10]
>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This
>>>>>>>>>>>>>>>>>>> next week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by carlos muñoz <ca...@gmail.com>.
Hi John,
Thanks for the update. I reviewed your code a little bit, it is looking
good. I think tha you should send a PR in order to receive feedback from
other community members.
Best,
Carlos
El dom., 28 jul. 2019 a las 23:20, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
> branch [2]. Please give it a look if you have time.
>
> This week, I will give a look to the documentation of datastores.
>
> Please let me know if you have suggestions.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Cheers,
> John
>
> El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi Alfonso,
>>
>> Yes, I was using this class javafx.util.Pair. It is not a problem I will
>> find an alternative, it is only an utilitary class.
>>
>> Thanks,
>> John
>>
>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> I checked out your code and it looks good :)
>>> I found that you use javafx, but that is not present in OpenJDK and
>>> fails to compile, and since we don't stick to Oracle JVM I would suggest to
>>> change it.
>>>
>>> Good job, keep it going :)
>>>
>>> Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>>
>>>
>>>
>>>
>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi.
>>>>
>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>> my branch [2]. Please give it a look if you have time.
>>>>
>>>> This week, I will give a look to the map reduce tests for DataStores.
>>>>
>>>> Please let me know if you have suggestions.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all
>>>>>
>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>>> my branch [2]. Please give it a look if you have time.
>>>>>
>>>>> This week, I will be working in the getPartitions and deleteByQuery
>>>>> methods and testing the other tests in the DataStoreTestBase class.
>>>>>
>>>>> Please let me know if you have suggestions.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Best,
>>>>> John.
>>>>>
>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi Alfonso,
>>>>>>
>>>>>> Thanks so much for your time and support for this project. I will
>>>>>> work on your comments. Responses inline :)
>>>>>>
>>>>>>
>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi, John.
>>>>>>>
>>>>>>> Sorry for the delay, I am changing work and I have been very busy :(
>>>>>>> I will try to answer your questions :)
>>>>>>>
>>>>>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>>>>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>>
>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date"
>>>>>>> type with an underlying "int" [1]. It's the first time I read about because
>>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>>> improvement would be great, though :)
>>>>>>>
>>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>>> - Is there any other approach?
>>>>>>>
>>>>>>
>>>>>> I think that Entity long field <-> Kudu timestamp conversion that the
>>>>>> best alternative right now. Because, I would add more compatible datatypes
>>>>>> to the mapping parameters which users can use. And this conversion should
>>>>>> not be dificult to implement in my opinion. Also, the new Date datatype of
>>>>>> avro could be implemented in newer versions because it would need further
>>>>>> analysis in other datastores too. I will work on that.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>>> *> KuduClient has multiple flushing modes
>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>>> also can set time interval
>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>>> for automatic flush.*
>>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>>> file? or just use the default configurations.*
>>>>>>>
>>>>>>> What we do in HBase is configure an autoflush option in
>>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>>
>>>>>>
>>>>>> Well, IMHO the default flush mode (auto flush sync) will do the job
>>>>>> for most use cases. But I will add a configuration in gora.properties for
>>>>>> selecting the other modes and specifying a autoflush time if needed by
>>>>>> the user.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>>> for this?.*
>>>>>>>
>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>>>> get/write Entities directly.
>>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>>> split to create a colection of partitions [6].
>>>>>>>
>>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>>> computation to where the data is locally".
>>>>>>>
>>>>>>> In any case, you can see that:
>>>>>>>
>>>>>>> - MongoDB store implementation does not implement splitting [7]
>>>>>>> - Cassandra store implementation does not implement splitting [8]
>>>>>>> - Aerospike store implementation does not implement splitting [9]
>>>>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>>>>
>>>>>>> If Kudu has a method to get the different splits for a table and its
>>>>>>> locations, then you will be able to implement the full feature.
>>>>>>>
>>>>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>>>>> much, so if you find you need more information let me know :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I will check whether Kudu has these features in order to implement
>>>>>> this method. If not I will use the default implementation found in other
>>>>>> backends.
>>>>>>
>>>>>>
>>>>>>> About Queries, what I can tell is that Hbase only implements "Start
>>>>>>> key" + "End key" because it has only 2 operations: "get" and "scan", and
>>>>>>> the querying is for "scan" operation, were you want an interval (or all) of
>>>>>>> the rows. Does Kudu have more querying functionality?
>>>>>>>
>>>>>>>
>>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>>> conditional predicates for filtering. I am using those classes.
>>>>>>
>>>>>>
>>>>>>> About other topic, I am trying to install Kudu in standalone (all in
>>>>>>> 1 node). Do you use a Cloudera installation or do you have a standalone
>>>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>>>> is unzip + execute "hbase start".
>>>>>>>
>>>>>>>
>>>>>> I am using an embedded mini-cluster which comes with compiled
>>>>>> binaries and can be used with maven[1] for testing my code. Once I get it
>>>>>> mature enough I think I will be testing the datastore with a docker
>>>>>> container [2]. I could not find a unzip+execute bundle either and I am
>>>>>> kinda noob for compiling it myself.
>>>>>>
>>>>>> [1]
>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>>
>>>>>>
>>>>>>> Good job and thank you!! :)
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Alfonso Nishikawa
>>>>>>>
>>>>>>>
>>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>>> [2] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>>> [3] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>>> [4] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>>> [5] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>>> [6] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>>> [7] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>>> [8] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>>> [9] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>>> [10] -
>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>>
>>>>>>>
>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed my
>>>>>>>> last commits to my branch [2]. Please give it a look if you have time.
>>>>>>>>
>>>>>>>> This week, I will be continue working in the Queries
>>>>>>>> implementation, please reach me out if you have any suggestions.
>>>>>>>>
>>>>>>>> Also, while reviewing the datastore interface I noticed this method
>>>>>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>>>>>> method?, should I use the partition definition in the xml mapping file for
>>>>>>>> this?.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> John.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>>
>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>>> confidence to the project and me.
>>>>>>>>>
>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my
>>>>>>>>> last commits to my branch [2].
>>>>>>>>>
>>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>>> implementation.
>>>>>>>>>
>>>>>>>>> BTW, I added a question to the wiki about Date types. Please give
>>>>>>>>> it a look if you have time.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi Carlos.
>>>>>>>>>>
>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> John.
>>>>>>>>>>
>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi John
>>>>>>>>>>>
>>>>>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Carlos
>>>>>>>>>>>
>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>
>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>>
>>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>>
>>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>>
>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi John
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed
>>>>>>>>>>>>> that you have created an Enumeration for datatypes, which is very similar
>>>>>>>>>>>>> to the kudu-client's [2]. Probably you should replace [1] for [2] in order
>>>>>>>>>>>>> to avoid code duplication.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Carlos
>>>>>>>>>>>>>
>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed
>>>>>>>>>>>>>> that my code is lacking some javadoc documentation I think I will be
>>>>>>>>>>>>>> working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0,
>>>>>>>>>>>>>>>> far from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and
>>>>>>>>>>>>>>>>> you will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>>>>>>> isn't it?
>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load
>>>>>>>>>>>>>>>>> the mapping from configuration as in HBase's store [7] in you KuduStore
>>>>>>>>>>>>>>>>> [8]. This will have implications on your readMapping at [9], so take a look
>>>>>>>>>>>>>>>>> at the one for HBase at [10]
>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This
>>>>>>>>>>>>>>>>>> next week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
I updated my report in the Wiki[1]. Also, I pushed my last commits to my
branch [2]. Please give it a look if you have time.
This week, I will give a look to the documentation of datastores.
Please let me know if you have suggestions.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Cheers,
John
El mié., 24 jul. 2019 a las 11:34, John Mora (<jh...@gmail.com>)
escribió:
> Hi Alfonso,
>
> Yes, I was using this class javafx.util.Pair. It is not a problem I will
> find an alternative, it is only an utilitary class.
>
> Thanks,
> John
>
> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> I checked out your code and it looks good :)
>> I found that you use javafx, but that is not present in OpenJDK and fails
>> to compile, and since we don't stick to Oracle JVM I would suggest to
>> change it.
>>
>> Good job, keep it going :)
>>
>> Regards,
>>
>> Alfonso Nishikawa
>>
>>
>>
>>
>>
>> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi.
>>>
>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>>> branch [2]. Please give it a look if you have time.
>>>
>>> This week, I will give a look to the map reduce tests for DataStores.
>>>
>>> Please let me know if you have suggestions.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Thanks,
>>> John
>>>
>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all
>>>>
>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to
>>>> my branch [2]. Please give it a look if you have time.
>>>>
>>>> This week, I will be working in the getPartitions and deleteByQuery
>>>> methods and testing the other tests in the DataStoreTestBase class.
>>>>
>>>> Please let me know if you have suggestions.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Best,
>>>> John.
>>>>
>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Alfonso,
>>>>>
>>>>> Thanks so much for your time and support for this project. I will work
>>>>> on your comments. Responses inline :)
>>>>>
>>>>>
>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi, John.
>>>>>>
>>>>>> Sorry for the delay, I am changing work and I have been very busy :(
>>>>>> I will try to answer your questions :)
>>>>>>
>>>>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>>>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>>
>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date" type
>>>>>> with an underlying "int" [1]. It's the first time I read about because
>>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>>> the value is the unix epoch. After this first approach, a design
>>>>>> improvement would be great, though :)
>>>>>>
>>>>>> - Would be good to have in the mapping a "timestamp" type so
>>>>>> KuduStore converts between the Entity long field <-> Kudu timestamp storage?
>>>>>> - Is there any other approach?
>>>>>>
>>>>>
>>>>> I think that Entity long field <-> Kudu timestamp conversion that the
>>>>> best alternative right now. Because, I would add more compatible datatypes
>>>>> to the mapping parameters which users can use. And this conversion should
>>>>> not be dificult to implement in my opinion. Also, the new Date datatype of
>>>>> avro could be implemented in newer versions because it would need further
>>>>> analysis in other datastores too. I will work on that.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> *> What is the Gora's policy regarding flush()? *
>>>>>> *> KuduClient has multiple flushing modes
>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>>> also can set time interval
>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>>> for automatic flush.*
>>>>>> *> Should theses behaviors be configurable using gora.properties
>>>>>> file? or just use the default configurations.*
>>>>>>
>>>>>> What we do in HBase is configure an autoflush option in
>>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>>
>>>>>
>>>>> Well, IMHO the default flush mode (auto flush sync) will do the job
>>>>> for most use cases. But I will add a configuration in gora.properties for
>>>>> selecting the other modes and specifying a autoflush time if needed by
>>>>> the user.
>>>>>
>>>>>
>>>>>>
>>>>>> *> Also, while reviewing the datastore interface I noticed this
>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected behavior of
>>>>>> this method?, should I use the partition definition in the xml mapping file
>>>>>> for this?.*
>>>>>>
>>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>>> get/write Entities directly.
>>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>>> split to create a colection of partitions [6].
>>>>>>
>>>>>> So, if Kudu is allowed to perform computation using local kudu
>>>>>> splits, then this method does the needed preparation to allow to "send the
>>>>>> computation to where the data is locally".
>>>>>>
>>>>>> In any case, you can see that:
>>>>>>
>>>>>> - MongoDB store implementation does not implement splitting [7]
>>>>>> - Cassandra store implementation does not implement splitting [8]
>>>>>> - Aerospike store implementation does not implement splitting [9]
>>>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>>>
>>>>>> If Kudu has a method to get the different splits for a table and its
>>>>>> locations, then you will be able to implement the full feature.
>>>>>>
>>>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>>>> much, so if you find you need more information let me know :)
>>>>>>
>>>>>>
>>>>>>
>>>>> I will check whether Kudu has these features in order to implement
>>>>> this method. If not I will use the default implementation found in other
>>>>> backends.
>>>>>
>>>>>
>>>>>> About Queries, what I can tell is that Hbase only implements "Start
>>>>>> key" + "End key" because it has only 2 operations: "get" and "scan", and
>>>>>> the querying is for "scan" operation, were you want an interval (or all) of
>>>>>> the rows. Does Kudu have more querying functionality?
>>>>>>
>>>>>>
>>>>> Yes, Kudu implements a Scanner for querying data among with
>>>>> conditional predicates for filtering. I am using those classes.
>>>>>
>>>>>
>>>>>> About other topic, I am trying to install Kudu in standalone (all in
>>>>>> 1 node). Do you use a Cloudera installation or do you have a standalone
>>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>>> is unzip + execute "hbase start".
>>>>>>
>>>>>>
>>>>> I am using an embedded mini-cluster which comes with compiled binaries
>>>>> and can be used with maven[1] for testing my code. Once I get it mature
>>>>> enough I think I will be testing the datastore with a docker container [2].
>>>>> I could not find a unzip+execute bundle either and I am kinda noob for
>>>>> compiling it myself.
>>>>>
>>>>> [1]
>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>>
>>>>>
>>>>>> Good job and thank you!! :)
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>>
>>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>>> [2] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>>> [3] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>>> [4] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>>> [5] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>>> [6] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>>> [7] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>>> [8] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>>> [9] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>>> [10] -
>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>>
>>>>>>
>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed my
>>>>>>> last commits to my branch [2]. Please give it a look if you have time.
>>>>>>>
>>>>>>> This week, I will be continue working in the Queries implementation,
>>>>>>> please reach me out if you have any suggestions.
>>>>>>>
>>>>>>> Also, while reviewing the datastore interface I noticed this method
>>>>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>>>>> method?, should I use the partition definition in the xml mapping file for
>>>>>>> this?.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> John.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>>
>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>>> confidence to the project and me.
>>>>>>>>
>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my
>>>>>>>> last commits to my branch [2].
>>>>>>>>
>>>>>>>> This week, I will be reviewing my the serialization/
>>>>>>>> deserialization process in order to identify optimizations specific for
>>>>>>>> Kudu. Because I used a generic methods of other backends which probably
>>>>>>>> could be better tuned for kudu. Also, I will start working on the Queries
>>>>>>>> implementation.
>>>>>>>>
>>>>>>>> BTW, I added a question to the wiki about Date types. Please give
>>>>>>>> it a look if you have time.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> John
>>>>>>>>
>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi Carlos.
>>>>>>>>>
>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> John.
>>>>>>>>>
>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi John
>>>>>>>>>>
>>>>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Carlos
>>>>>>>>>>
>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>>
>>>>>>>>>>> As I mentioned in the reports I would like to know how
>>>>>>>>>>> datastores deal with flush(), should it work always manually executed?.
>>>>>>>>>>>
>>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>>
>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi John
>>>>>>>>>>>>
>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed
>>>>>>>>>>>> that you have created an Enumeration for datatypes, which is very similar
>>>>>>>>>>>> to the kudu-client's [2]. Probably you should replace [1] for [2] in order
>>>>>>>>>>>> to avoid code duplication.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Carlos
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed
>>>>>>>>>>>>> that my code is lacking some javadoc documentation I think I will be
>>>>>>>>>>>>> working on that this week, also I would like to enable and check schema
>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> John.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0,
>>>>>>>>>>>>>>> far from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and
>>>>>>>>>>>>>>>> you will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>>>>>> isn't it?
>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This
>>>>>>>>>>>>>>>>> next week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last
>>>>>>>>>>>>>>>>> commits are available here [2]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Alfonso,
Yes, I was using this class javafx.util.Pair. It is not a problem I will
find an alternative, it is only an utilitary class.
Thanks,
John
El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi, John.
>
> I checked out your code and it looks good :)
> I found that you use javafx, but that is not present in OpenJDK and fails
> to compile, and since we don't stick to Oracle JVM I would suggest to
> change it.
>
> Good job, keep it going :)
>
> Regards,
>
> Alfonso Nishikawa
>
>
>
>
>
> El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi.
>>
>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>> branch [2]. Please give it a look if you have time.
>>
>> This week, I will give a look to the map reduce tests for DataStores.
>>
>> Please let me know if you have suggestions.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Thanks,
>> John
>>
>> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all
>>>
>>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>>> branch [2]. Please give it a look if you have time.
>>>
>>> This week, I will be working in the getPartitions and deleteByQuery
>>> methods and testing the other tests in the DataStoreTestBase class.
>>>
>>> Please let me know if you have suggestions.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Best,
>>> John.
>>>
>>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Alfonso,
>>>>
>>>> Thanks so much for your time and support for this project. I will work
>>>> on your comments. Responses inline :)
>>>>
>>>>
>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi, John.
>>>>>
>>>>> Sorry for the delay, I am changing work and I have been very busy :( I
>>>>> will try to answer your questions :)
>>>>>
>>>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>>> intuitively assumed this is a date.). However, in the java world the
>>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>>
>>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date" type
>>>>> with an underlying "int" [1]. It's the first time I read about because
>>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>>> the value is the unix epoch. After this first approach, a design
>>>>> improvement would be great, though :)
>>>>>
>>>>> - Would be good to have in the mapping a "timestamp" type so KuduStore
>>>>> converts between the Entity long field <-> Kudu timestamp storage?
>>>>> - Is there any other approach?
>>>>>
>>>>
>>>> I think that Entity long field <-> Kudu timestamp conversion that the
>>>> best alternative right now. Because, I would add more compatible datatypes
>>>> to the mapping parameters which users can use. And this conversion should
>>>> not be dificult to implement in my opinion. Also, the new Date datatype of
>>>> avro could be implemented in newer versions because it would need further
>>>> analysis in other datastores too. I will work on that.
>>>>
>>>>
>>>>>
>>>>>
>>>>> *> What is the Gora's policy regarding flush()? *
>>>>> *> KuduClient has multiple flushing modes
>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>>> also can set time interval
>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>>> for automatic flush.*
>>>>> *> Should theses behaviors be configurable using gora.properties file?
>>>>> or just use the default configurations.*
>>>>>
>>>>> What we do in HBase is configure an autoflush option in
>>>>> gora.properties [2] which is used when instanced the Table, but at the same
>>>>> time we implement the flush() method to force the flush [3]. I would
>>>>> suggest to follow that example, but adding the flushing options of Kudu.
>>>>> What flushing mode (and time interval if it applies) do you suggest?
>>>>>
>>>>
>>>> Well, IMHO the default flush mode (auto flush sync) will do the job
>>>> for most use cases. But I will add a configuration in gora.properties for
>>>> selecting the other modes and specifying a autoflush time if needed by
>>>> the user.
>>>>
>>>>
>>>>>
>>>>> *> Also, while reviewing the datastore interface I noticed this method
>>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>>> method?, should I use the partition definition in the xml mapping file for
>>>>> this?.*
>>>>>
>>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>>> get/write Entities directly.
>>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>>> [5] to compute the splits (start key---end key) with the location of the
>>>>> split to create a colection of partitions [6].
>>>>>
>>>>> So, if Kudu is allowed to perform computation using local kudu splits,
>>>>> then this method does the needed preparation to allow to "send the
>>>>> computation to where the data is locally".
>>>>>
>>>>> In any case, you can see that:
>>>>>
>>>>> - MongoDB store implementation does not implement splitting [7]
>>>>> - Cassandra store implementation does not implement splitting [8]
>>>>> - Aerospike store implementation does not implement splitting [9]
>>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>>
>>>>> If Kudu has a method to get the different splits for a table and its
>>>>> locations, then you will be able to implement the full feature.
>>>>>
>>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>>> much, so if you find you need more information let me know :)
>>>>>
>>>>>
>>>>>
>>>> I will check whether Kudu has these features in order to implement this
>>>> method. If not I will use the default implementation found in other
>>>> backends.
>>>>
>>>>
>>>>> About Queries, what I can tell is that Hbase only implements "Start
>>>>> key" + "End key" because it has only 2 operations: "get" and "scan", and
>>>>> the querying is for "scan" operation, were you want an interval (or all) of
>>>>> the rows. Does Kudu have more querying functionality?
>>>>>
>>>>>
>>>> Yes, Kudu implements a Scanner for querying data among with conditional
>>>> predicates for filtering. I am using those classes.
>>>>
>>>>
>>>>> About other topic, I am trying to install Kudu in standalone (all in 1
>>>>> node). Do you use a Cloudera installation or do you have a standalone
>>>>> installation? How do you do it? I found some instructions, but they talk
>>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>>> is unzip + execute "hbase start".
>>>>>
>>>>>
>>>> I am using an embedded mini-cluster which comes with compiled binaries
>>>> and can be used with maven[1] for testing my code. Once I get it mature
>>>> enough I think I will be testing the datastore with a docker container [2].
>>>> I could not find a unzip+execute bundle either and I am kinda noob for
>>>> compiling it myself.
>>>>
>>>> [1]
>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>>
>>>>
>>>>> Good job and thank you!! :)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>>
>>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>>> [2] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>>> [3] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>>> [4] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>>> [5] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>>> [6] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>>> [7] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>>> [8] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>>> [9] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>>> [10] -
>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>>
>>>>>
>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed my
>>>>>> last commits to my branch [2]. Please give it a look if you have time.
>>>>>>
>>>>>> This week, I will be continue working in the Queries implementation,
>>>>>> please reach me out if you have any suggestions.
>>>>>>
>>>>>> Also, while reviewing the datastore interface I noticed this method
>>>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>>>> method?, should I use the partition definition in the xml mapping file for
>>>>>> this?.
>>>>>>
>>>>>> Cheers,
>>>>>> John.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>>
>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I received my first evaluation from the Google Summer of Code
>>>>>>> program with a positive result. Thanks so much for your support and
>>>>>>> confidence to the project and me.
>>>>>>>
>>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my
>>>>>>> last commits to my branch [2].
>>>>>>>
>>>>>>> This week, I will be reviewing my the serialization/ deserialization
>>>>>>> process in order to identify optimizations specific for Kudu. Because I
>>>>>>> used a generic methods of other backends which probably could be better
>>>>>>> tuned for kudu. Also, I will start working on the Queries implementation.
>>>>>>>
>>>>>>> BTW, I added a question to the wiki about Date types. Please give it
>>>>>>> a look if you have time.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Cheers,
>>>>>>> John
>>>>>>>
>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Carlos.
>>>>>>>>
>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> John.
>>>>>>>>
>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi John
>>>>>>>>>
>>>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Carlos
>>>>>>>>>
>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>>
>>>>>>>>>> As I mentioned in the reports I would like to know how datastores
>>>>>>>>>> deal with flush(), should it work always manually executed?.
>>>>>>>>>>
>>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>>
>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi John
>>>>>>>>>>>
>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed
>>>>>>>>>>> that you have created an Enumeration for datatypes, which is very similar
>>>>>>>>>>> to the kudu-client's [2]. Probably you should replace [1] for [2] in order
>>>>>>>>>>> to avoid code duplication.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Carlos
>>>>>>>>>>>
>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>
>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that
>>>>>>>>>>>> my code is lacking some javadoc documentation I think I will be working on
>>>>>>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> John.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your
>>>>>>>>>>>>> comments.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0,
>>>>>>>>>>>>>> far from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from
>>>>>>>>>>>>>>> there.
>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and
>>>>>>>>>>>>>>> you will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>>>>> isn't it?
>>>>>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>>>>>> - I know it is in other backends, but avoid
>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the checked ones) like in
>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This
>>>>>>>>>>>>>>>> next week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last commits
>>>>>>>>>>>>>>>> are available here [2]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
I checked out your code and it looks good :)
I found that you use javafx, but that is not present in OpenJDK and fails
to compile, and since we don't stick to Oracle JVM I would suggest to
change it.
Good job, keep it going :)
Regards,
Alfonso Nishikawa
El sáb., 20 jul. 2019 a las 22:25, John Mora (<jh...@gmail.com>)
escribió:
> Hi.
>
> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
> branch [2]. Please give it a look if you have time.
>
> This week, I will give a look to the map reduce tests for DataStores.
>
> Please let me know if you have suggestions.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Thanks,
> John
>
> El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all
>>
>> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
>> branch [2]. Please give it a look if you have time.
>>
>> This week, I will be working in the getPartitions and deleteByQuery
>> methods and testing the other tests in the DataStoreTestBase class.
>>
>> Please let me know if you have suggestions.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Best,
>> John.
>>
>> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi Alfonso,
>>>
>>> Thanks so much for your time and support for this project. I will work
>>> on your comments. Responses inline :)
>>>
>>>
>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi, John.
>>>>
>>>> Sorry for the delay, I am changing work and I have been very busy :( I
>>>> will try to answer your questions :)
>>>>
>>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>>> intuitively assumed this is a date.). However, in the java world the
>>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>>
>>>> In Avro 1.8 were introduced "Logical Types" so there is a "date" type
>>>> with an underlying "int" [1]. It's the first time I read about because
>>>> until the last version upgrade of Avro this weren't there. I would suggest
>>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>>> the value is the unix epoch. After this first approach, a design
>>>> improvement would be great, though :)
>>>>
>>>> - Would be good to have in the mapping a "timestamp" type so KuduStore
>>>> converts between the Entity long field <-> Kudu timestamp storage?
>>>> - Is there any other approach?
>>>>
>>>
>>> I think that Entity long field <-> Kudu timestamp conversion that the
>>> best alternative right now. Because, I would add more compatible datatypes
>>> to the mapping parameters which users can use. And this conversion should
>>> not be dificult to implement in my opinion. Also, the new Date datatype of
>>> avro could be implemented in newer versions because it would need further
>>> analysis in other datastores too. I will work on that.
>>>
>>>
>>>>
>>>>
>>>> *> What is the Gora's policy regarding flush()? *
>>>> *> KuduClient has multiple flushing modes
>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>>> also can set time interval
>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>>> for automatic flush.*
>>>> *> Should theses behaviors be configurable using gora.properties file?
>>>> or just use the default configurations.*
>>>>
>>>> What we do in HBase is configure an autoflush option in gora.properties
>>>> [2] which is used when instanced the Table, but at the same time we
>>>> implement the flush() method to force the flush [3]. I would suggest to
>>>> follow that example, but adding the flushing options of Kudu. What flushing
>>>> mode (and time interval if it applies) do you suggest?
>>>>
>>>
>>> Well, IMHO the default flush mode (auto flush sync) will do the job for
>>> most use cases. But I will add a configuration in gora.properties for
>>> selecting the other modes and specifying a autoflush time if needed by
>>> the user.
>>>
>>>
>>>>
>>>> *> Also, while reviewing the datastore interface I noticed this method
>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>> method?, should I use the partition definition in the xml mapping file for
>>>> this?.*
>>>>
>>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>>> get/write Entities directly.
>>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>>> [5] to compute the splits (start key---end key) with the location of the
>>>> split to create a colection of partitions [6].
>>>>
>>>> So, if Kudu is allowed to perform computation using local kudu splits,
>>>> then this method does the needed preparation to allow to "send the
>>>> computation to where the data is locally".
>>>>
>>>> In any case, you can see that:
>>>>
>>>> - MongoDB store implementation does not implement splitting [7]
>>>> - Cassandra store implementation does not implement splitting [8]
>>>> - Aerospike store implementation does not implement splitting [9]
>>>> - Accumulo store implementation* does* implement splitting [10]
>>>>
>>>> If Kudu has a method to get the different splits for a table and its
>>>> locations, then you will be able to implement the full feature.
>>>>
>>>> This is Hadoop related and it is not trivial. I haven't elaborated
>>>> much, so if you find you need more information let me know :)
>>>>
>>>>
>>>>
>>> I will check whether Kudu has these features in order to implement this
>>> method. If not I will use the default implementation found in other
>>> backends.
>>>
>>>
>>>> About Queries, what I can tell is that Hbase only implements "Start
>>>> key" + "End key" because it has only 2 operations: "get" and "scan", and
>>>> the querying is for "scan" operation, were you want an interval (or all) of
>>>> the rows. Does Kudu have more querying functionality?
>>>>
>>>>
>>> Yes, Kudu implements a Scanner for querying data among with conditional
>>> predicates for filtering. I am using those classes.
>>>
>>>
>>>> About other topic, I am trying to install Kudu in standalone (all in 1
>>>> node). Do you use a Cloudera installation or do you have a standalone
>>>> installation? How do you do it? I found some instructions, but they talk
>>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>>> is unzip + execute "hbase start".
>>>>
>>>>
>>> I am using an embedded mini-cluster which comes with compiled binaries
>>> and can be used with maven[1] for testing my code. Once I get it mature
>>> enough I think I will be testing the datastore with a docker container [2].
>>> I could not find a unzip+execute bundle either and I am kinda noob for
>>> compiling it myself.
>>>
>>> [1]
>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>>
>>>
>>>> Good job and thank you!! :)
>>>>
>>>> Regards,
>>>>
>>>> Alfonso Nishikawa
>>>>
>>>>
>>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>>> [2] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>>> [3] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>>> [4] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>>> [5] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>>> [6] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>>> [7] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>>> [8] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>>> [9] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>>> [10] -
>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>>> [11] - https://kudu.apache.org/docs/installation.html
>>>>
>>>>
>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> As every week I updated my report in the Wiki[1]. Also, I pushed my
>>>>> last commits to my branch [2]. Please give it a look if you have time.
>>>>>
>>>>> This week, I will be continue working in the Queries implementation,
>>>>> please reach me out if you have any suggestions.
>>>>>
>>>>> Also, while reviewing the datastore interface I noticed this method
>>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>>> method?, should I use the partition definition in the xml mapping file for
>>>>> this?.
>>>>>
>>>>> Cheers,
>>>>> John.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>>
>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> I received my first evaluation from the Google Summer of Code program
>>>>>> with a positive result. Thanks so much for your support and confidence to
>>>>>> the project and me.
>>>>>>
>>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my
>>>>>> last commits to my branch [2].
>>>>>>
>>>>>> This week, I will be reviewing my the serialization/ deserialization
>>>>>> process in order to identify optimizations specific for Kudu. Because I
>>>>>> used a generic methods of other backends which probably could be better
>>>>>> tuned for kudu. Also, I will start working on the Queries implementation.
>>>>>>
>>>>>> BTW, I added a question to the wiki about Date types. Please give it
>>>>>> a look if you have time.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Cheers,
>>>>>> John
>>>>>>
>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi Carlos.
>>>>>>>
>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>>
>>>>>>> Best,
>>>>>>> John.
>>>>>>>
>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi John
>>>>>>>>
>>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Carlos
>>>>>>>>
>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>>
>>>>>>>>> As I mentioned in the reports I would like to know how datastores
>>>>>>>>> deal with flush(), should it work always manually executed?.
>>>>>>>>>
>>>>>>>>> Finally, This week I will be implementing object
>>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>>
>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi John
>>>>>>>>>>
>>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed
>>>>>>>>>> that you have created an Enumeration for datatypes, which is very similar
>>>>>>>>>> to the kudu-client's [2]. Probably you should replace [1] for [2] in order
>>>>>>>>>> to avoid code duplication.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Carlos
>>>>>>>>>>
>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that
>>>>>>>>>>> my code is lacking some javadoc documentation I think I will be working on
>>>>>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> John.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far
>>>>>>>>>>>>> from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and
>>>>>>>>>>>>>> you will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>>>> isn't it?
>>>>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions
>>>>>>>>>>>>>> (at least in Java since we have the checked ones) like in [11]. You can
>>>>>>>>>>>>>> wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This
>>>>>>>>>>>>>>> next week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last commits
>>>>>>>>>>>>>>> are available here [2]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi.
I updated my report in the Wiki[1]. Also, I pushed my last commits to my
branch [2]. Please give it a look if you have time.
This week, I will give a look to the map reduce tests for DataStores.
Please let me know if you have suggestions.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Thanks,
John
El sáb., 13 jul. 2019 a las 19:31, John Mora (<jh...@gmail.com>)
escribió:
> Hi all
>
> I updated my report in the Wiki[1]. Also, I pushed my last commits to my
> branch [2]. Please give it a look if you have time.
>
> This week, I will be working in the getPartitions and deleteByQuery
> methods and testing the other tests in the DataStoreTestBase class.
>
> Please let me know if you have suggestions.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Best,
> John.
>
> El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi Alfonso,
>>
>> Thanks so much for your time and support for this project. I will work on
>> your comments. Responses inline :)
>>
>>
>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> Sorry for the delay, I am changing work and I have been very busy :( I
>>> will try to answer your questions :)
>>>
>>> *> In the Employee example there is a field called 'dateOfBirth'. I
>>> tried to map that field with the UNIXTIME_MICROS datatype of Kudu (I
>>> intuitively assumed this is a date.). However, in the java world the
>>> Employee field is a Long value and the kudu datatype is a Timestamp. So, I
>>> was wondering whether I should force the usage of the UNIXTIME_MICROS
>>> datatype for this field or just use a LONG datatype in Kudu.*
>>>
>>> In Avro 1.8 were introduced "Logical Types" so there is a "date" type
>>> with an underlying "int" [1]. It's the first time I read about because
>>> until the last version upgrade of Avro this weren't there. I would suggest
>>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>>> the value is the unix epoch. After this first approach, a design
>>> improvement would be great, though :)
>>>
>>> - Would be good to have in the mapping a "timestamp" type so KuduStore
>>> converts between the Entity long field <-> Kudu timestamp storage?
>>> - Is there any other approach?
>>>
>>
>> I think that Entity long field <-> Kudu timestamp conversion that the
>> best alternative right now. Because, I would add more compatible datatypes
>> to the mapping parameters which users can use. And this conversion should
>> not be dificult to implement in my opinion. Also, the new Date datatype of
>> avro could be implemented in newer versions because it would need further
>> analysis in other datastores too. I will work on that.
>>
>>
>>>
>>>
>>> *> What is the Gora's policy regarding flush()? *
>>> *> KuduClient has multiple flushing modes
>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>>> also can set time interval
>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>>> for automatic flush.*
>>> *> Should theses behaviors be configurable using gora.properties file?
>>> or just use the default configurations.*
>>>
>>> What we do in HBase is configure an autoflush option in gora.properties
>>> [2] which is used when instanced the Table, but at the same time we
>>> implement the flush() method to force the flush [3]. I would suggest to
>>> follow that example, but adding the flushing options of Kudu. What flushing
>>> mode (and time interval if it applies) do you suggest?
>>>
>>
>> Well, IMHO the default flush mode (auto flush sync) will do the job for
>> most use cases. But I will add a configuration in gora.properties for
>> selecting the other modes and specifying a autoflush time if needed by
>> the user.
>>
>>
>>>
>>> *> Also, while reviewing the datastore interface I noticed this method
>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>> method?, should I use the partition definition in the xml mapping file for
>>> this?.*
>>>
>>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>>> get/write Entities directly.
>>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>>> [5] to compute the splits (start key---end key) with the location of the
>>> split to create a colection of partitions [6].
>>>
>>> So, if Kudu is allowed to perform computation using local kudu splits,
>>> then this method does the needed preparation to allow to "send the
>>> computation to where the data is locally".
>>>
>>> In any case, you can see that:
>>>
>>> - MongoDB store implementation does not implement splitting [7]
>>> - Cassandra store implementation does not implement splitting [8]
>>> - Aerospike store implementation does not implement splitting [9]
>>> - Accumulo store implementation* does* implement splitting [10]
>>>
>>> If Kudu has a method to get the different splits for a table and its
>>> locations, then you will be able to implement the full feature.
>>>
>>> This is Hadoop related and it is not trivial. I haven't elaborated much,
>>> so if you find you need more information let me know :)
>>>
>>>
>>>
>> I will check whether Kudu has these features in order to implement this
>> method. If not I will use the default implementation found in other
>> backends.
>>
>>
>>> About Queries, what I can tell is that Hbase only implements "Start key"
>>> + "End key" because it has only 2 operations: "get" and "scan", and the
>>> querying is for "scan" operation, were you want an interval (or all) of the
>>> rows. Does Kudu have more querying functionality?
>>>
>>>
>> Yes, Kudu implements a Scanner for querying data among with conditional
>> predicates for filtering. I am using those classes.
>>
>>
>>> About other topic, I am trying to install Kudu in standalone (all in 1
>>> node). Do you use a Cloudera installation or do you have a standalone
>>> installation? How do you do it? I found some instructions, but they talk
>>> about compiling Kudu [11]. I was looking for something like HBase, that it
>>> is unzip + execute "hbase start".
>>>
>>>
>> I am using an embedded mini-cluster which comes with compiled binaries
>> and can be used with maven[1] for testing my code. Once I get it mature
>> enough I think I will be testing the datastore with a docker container [2].
>> I could not find a unzip+execute bundle either and I am kinda noob for
>> compiling it myself.
>>
>> [1]
>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>>
>>
>>> Good job and thank you!! :)
>>>
>>> Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>>
>>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>>> [2] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>>> [3] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>>> [4] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>>> [5] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>>> [6] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>>> [7] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>>> [8] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>>> [9] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>>> [10] -
>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>>> [11] - https://kudu.apache.org/docs/installation.html
>>>
>>>
>>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> As every week I updated my report in the Wiki[1]. Also, I pushed my
>>>> last commits to my branch [2]. Please give it a look if you have time.
>>>>
>>>> This week, I will be continue working in the Queries implementation,
>>>> please reach me out if you have any suggestions.
>>>>
>>>> Also, while reviewing the datastore interface I noticed this method
>>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>>> method?, should I use the partition definition in the xml mapping file for
>>>> this?.
>>>>
>>>> Cheers,
>>>> John.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>>
>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> I received my first evaluation from the Google Summer of Code program
>>>>> with a positive result. Thanks so much for your support and confidence to
>>>>> the project and me.
>>>>>
>>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my
>>>>> last commits to my branch [2].
>>>>>
>>>>> This week, I will be reviewing my the serialization/ deserialization
>>>>> process in order to identify optimizations specific for Kudu. Because I
>>>>> used a generic methods of other backends which probably could be better
>>>>> tuned for kudu. Also, I will start working on the Queries implementation.
>>>>>
>>>>> BTW, I added a question to the wiki about Date types. Please give it a
>>>>> look if you have time.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Cheers,
>>>>> John
>>>>>
>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi Carlos.
>>>>>>
>>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>>
>>>>>> Best,
>>>>>> John.
>>>>>>
>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<
>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi John
>>>>>>>
>>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Carlos
>>>>>>>
>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I
>>>>>>>> pushed my last commits to my branch [2].
>>>>>>>>
>>>>>>>> As I mentioned in the reports I would like to know how datastores
>>>>>>>> deal with flush(), should it work always manually executed?.
>>>>>>>>
>>>>>>>> Finally, This week I will be implementing object
>>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>>
>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> John
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi John
>>>>>>>>>
>>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed that
>>>>>>>>> you have created an Enumeration for datatypes, which is very similar to the
>>>>>>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>>>>>>> avoid code duplication.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Carlos
>>>>>>>>>
>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<
>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that
>>>>>>>>>> my code is lacking some javadoc documentation I think I will be working on
>>>>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> John.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>>
>>>>>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>>> mapping file.
>>>>>>>>>>>>
>>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>>
>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin
>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>>
>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far
>>>>>>>>>>>> from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>>
>>>>>>>>>>>> [1] -
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you!
>>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you
>>>>>>>>>>>>> will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>>> isn't it?
>>>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions
>>>>>>>>>>>>> (at least in Java since we have the checked ones) like in [11]. You can
>>>>>>>>>>>>> wrap them in GoraException. An example is [12]
>>>>>>>>>>>>>
>>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>>> [2] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>>> [3] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>>> [4] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>>> [5] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>>> [6] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>>> [7] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>>> [8] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>>> [9] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>>> [10] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>>> [11] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>>> [12] -
>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next
>>>>>>>>>>>>>> week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let me know if you have suggestions, my last commits
>>>>>>>>>>>>>> are available here [2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all
I updated my report in the Wiki[1]. Also, I pushed my last commits to my
branch [2]. Please give it a look if you have time.
This week, I will be working in the getPartitions and deleteByQuery methods
and testing the other tests in the DataStoreTestBase class.
Please let me know if you have suggestions.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Best,
John.
El mié., 10 jul. 2019 a las 16:17, John Mora (<jh...@gmail.com>)
escribió:
> Hi Alfonso,
>
> Thanks so much for your time and support for this project. I will work on
> your comments. Responses inline :)
>
>
> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> Sorry for the delay, I am changing work and I have been very busy :( I
>> will try to answer your questions :)
>>
>> *> In the Employee example there is a field called 'dateOfBirth'. I tried
>> to map that field with the UNIXTIME_MICROS datatype of Kudu (I intuitively
>> assumed this is a date.). However, in the java world the Employee field is
>> a Long value and the kudu datatype is a Timestamp. So, I was wondering
>> whether I should force the usage of the UNIXTIME_MICROS datatype for this
>> field or just use a LONG datatype in Kudu.*
>>
>> In Avro 1.8 were introduced "Logical Types" so there is a "date" type
>> with an underlying "int" [1]. It's the first time I read about because
>> until the last version upgrade of Avro this weren't there. I would suggest
>> to ignore "dates" and map dateOfBirth as long, since in any case -in avro-
>> the value is the unix epoch. After this first approach, a design
>> improvement would be great, though :)
>>
>> - Would be good to have in the mapping a "timestamp" type so KuduStore
>> converts between the Entity long field <-> Kudu timestamp storage?
>> - Is there any other approach?
>>
>
> I think that Entity long field <-> Kudu timestamp conversion that the best
> alternative right now. Because, I would add more compatible datatypes to
> the mapping parameters which users can use. And this conversion should not
> be dificult to implement in my opinion. Also, the new Date datatype of avro
> could be implemented in newer versions because it would need further
> analysis in other datastores too. I will work on that.
>
>
>>
>>
>> *> What is the Gora's policy regarding flush()? *
>> *> KuduClient has multiple flushing modes
>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
>> also can set time interval
>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
>> for automatic flush.*
>> *> Should theses behaviors be configurable using gora.properties file? or
>> just use the default configurations.*
>>
>> What we do in HBase is configure an autoflush option in gora.properties
>> [2] which is used when instanced the Table, but at the same time we
>> implement the flush() method to force the flush [3]. I would suggest to
>> follow that example, but adding the flushing options of Kudu. What flushing
>> mode (and time interval if it applies) do you suggest?
>>
>
> Well, IMHO the default flush mode (auto flush sync) will do the job for
> most use cases. But I will add a configuration in gora.properties for
> selecting the other modes and specifying a autoflush time if needed by
> the user.
>
>
>>
>> *> Also, while reviewing the datastore interface I noticed this method
>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>> method?, should I use the partition definition in the xml mapping file for
>> this?.*
>>
>> The method getPartitions(Query) is related to Hadoop. Apache Gora
>> integrates with Hadoop implementing a custom Map and Reduce that allows to
>> get/write Entities directly.
>> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
>> [5] to compute the splits (start key---end key) with the location of the
>> split to create a colection of partitions [6].
>>
>> So, if Kudu is allowed to perform computation using local kudu splits,
>> then this method does the needed preparation to allow to "send the
>> computation to where the data is locally".
>>
>> In any case, you can see that:
>>
>> - MongoDB store implementation does not implement splitting [7]
>> - Cassandra store implementation does not implement splitting [8]
>> - Aerospike store implementation does not implement splitting [9]
>> - Accumulo store implementation* does* implement splitting [10]
>>
>> If Kudu has a method to get the different splits for a table and its
>> locations, then you will be able to implement the full feature.
>>
>> This is Hadoop related and it is not trivial. I haven't elaborated much,
>> so if you find you need more information let me know :)
>>
>>
>>
> I will check whether Kudu has these features in order to implement this
> method. If not I will use the default implementation found in other
> backends.
>
>
>> About Queries, what I can tell is that Hbase only implements "Start key"
>> + "End key" because it has only 2 operations: "get" and "scan", and the
>> querying is for "scan" operation, were you want an interval (or all) of the
>> rows. Does Kudu have more querying functionality?
>>
>>
> Yes, Kudu implements a Scanner for querying data among with conditional
> predicates for filtering. I am using those classes.
>
>
>> About other topic, I am trying to install Kudu in standalone (all in 1
>> node). Do you use a Cloudera installation or do you have a standalone
>> installation? How do you do it? I found some instructions, but they talk
>> about compiling Kudu [11]. I was looking for something like HBase, that it
>> is unzip + execute "hbase start".
>>
>>
> I am using an embedded mini-cluster which comes with compiled binaries and
> can be used with maven[1] for testing my code. Once I get it mature enough
> I think I will be testing the datastore with a docker container [2]. I
> could not find a unzip+execute bundle either and I am kinda noob for
> compiling it myself.
>
> [1]
> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
> [2] https://hub.docker.com/r/usuresearch/apache-kudu/
>
>
>> Good job and thank you!! :)
>>
>> Regards,
>>
>> Alfonso Nishikawa
>>
>>
>> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
>> [2] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
>> [3] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
>> [4] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
>> [5] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
>> [6] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
>> [7] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
>> [8] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
>> [9] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
>> [10] -
>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
>> [11] - https://kudu.apache.org/docs/installation.html
>>
>>
>> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> As every week I updated my report in the Wiki[1]. Also, I pushed my last
>>> commits to my branch [2]. Please give it a look if you have time.
>>>
>>> This week, I will be continue working in the Queries implementation,
>>> please reach me out if you have any suggestions.
>>>
>>> Also, while reviewing the datastore interface I noticed this method
>>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>>> method?, should I use the partition definition in the xml mapping file for
>>> this?.
>>>
>>> Cheers,
>>> John.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>>
>>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> I received my first evaluation from the Google Summer of Code program
>>>> with a positive result. Thanks so much for your support and confidence to
>>>> the project and me.
>>>>
>>>> I updated my report of this week in the Wiki[1]. Also, I pushed my last
>>>> commits to my branch [2].
>>>>
>>>> This week, I will be reviewing my the serialization/ deserialization
>>>> process in order to identify optimizations specific for Kudu. Because I
>>>> used a generic methods of other backends which probably could be better
>>>> tuned for kudu. Also, I will start working on the Queries implementation.
>>>>
>>>> BTW, I added a question to the wiki about Date types. Please give it a
>>>> look if you have time.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Cheers,
>>>> John
>>>>
>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Carlos.
>>>>>
>>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>>
>>>>> Best,
>>>>> John.
>>>>>
>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi John
>>>>>>
>>>>>> The first Google Summer of Code evaluation is due on June 28th.
>>>>>> Please make sure you submit your Mentors' evaluation on time.
>>>>>>
>>>>>> Regards,
>>>>>> Carlos
>>>>>>
>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed
>>>>>>> my last commits to my branch [2].
>>>>>>>
>>>>>>> As I mentioned in the reports I would like to know how datastores
>>>>>>> deal with flush(), should it work always manually executed?.
>>>>>>>
>>>>>>> Finally, This week I will be implementing object
>>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>>
>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Cheers,
>>>>>>> John
>>>>>>>
>>>>>>>
>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi John
>>>>>>>>
>>>>>>>> Your last changes look good to me. Keep it up. But, I noticed that
>>>>>>>> you have created an Enumeration for datatypes, which is very similar to the
>>>>>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>>>>>> avoid code duplication.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Carlos
>>>>>>>>
>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that my
>>>>>>>>> code is lacking some javadoc documentation I think I will be working on
>>>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> John.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>>>>>>> escribió:
>>>>>>>>>
>>>>>>>>>> Hi Alfonso.
>>>>>>>>>>
>>>>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi, John.
>>>>>>>>>>>
>>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - How to represent partitioning configurations on the
>>>>>>>>>>> mapping file.
>>>>>>>>>>>
>>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>>
>>>>>>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>>
>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far
>>>>>>>>>>> from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>>
>>>>>>>>>>> [1] -
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi, John.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you!
>>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>>
>>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>>> should not set the version there.
>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you
>>>>>>>>>>>> will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>>> isn't it?
>>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions
>>>>>>>>>>>> (at least in Java since we have the checked ones) like in [11]. You can
>>>>>>>>>>>> wrap them in GoraException. An example is [12]
>>>>>>>>>>>>
>>>>>>>>>>>> And nothing more :)
>>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>>> [2] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>>> [3] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>>> [4] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>>> [5] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>>> [6] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>>> [7] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>>> [8] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>>> [9] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>>> [10] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>>> [11] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>>> [12] -
>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next
>>>>>>>>>>>>> week I think I should be focusing on the create schema operation and
>>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if you have suggestions, my last commits
>>>>>>>>>>>>> are available here [2]
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Alfonso,
Thanks so much for your time and support for this project. I will work on
your comments. Responses inline :)
El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi, John.
>
> Sorry for the delay, I am changing work and I have been very busy :( I
> will try to answer your questions :)
>
> *> In the Employee example there is a field called 'dateOfBirth'. I tried
> to map that field with the UNIXTIME_MICROS datatype of Kudu (I intuitively
> assumed this is a date.). However, in the java world the Employee field is
> a Long value and the kudu datatype is a Timestamp. So, I was wondering
> whether I should force the usage of the UNIXTIME_MICROS datatype for this
> field or just use a LONG datatype in Kudu.*
>
> In Avro 1.8 were introduced "Logical Types" so there is a "date" type with
> an underlying "int" [1]. It's the first time I read about because until the
> last version upgrade of Avro this weren't there. I would suggest to ignore
> "dates" and map dateOfBirth as long, since in any case -in avro- the value
> is the unix epoch. After this first approach, a design improvement would be
> great, though :)
>
> - Would be good to have in the mapping a "timestamp" type so KuduStore
> converts between the Entity long field <-> Kudu timestamp storage?
> - Is there any other approach?
>
I think that Entity long field <-> Kudu timestamp conversion that the best
alternative right now. Because, I would add more compatible datatypes to
the mapping parameters which users can use. And this conversion should not
be dificult to implement in my opinion. Also, the new Date datatype of avro
could be implemented in newer versions because it would need further
analysis in other datastores too. I will work on that.
>
>
> *> What is the Gora's policy regarding flush()? *
> *> KuduClient has multiple flushing modes
> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
> also can set time interval
> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
> for automatic flush.*
> *> Should theses behaviors be configurable using gora.properties file? or
> just use the default configurations.*
>
> What we do in HBase is configure an autoflush option in gora.properties
> [2] which is used when instanced the Table, but at the same time we
> implement the flush() method to force the flush [3]. I would suggest to
> follow that example, but adding the flushing options of Kudu. What flushing
> mode (and time interval if it applies) do you suggest?
>
Well, IMHO the default flush mode (auto flush sync) will do the job for
most use cases. But I will add a configuration in gora.properties for
selecting the other modes and specifying a autoflush time if needed by
the user.
>
> *> Also, while reviewing the datastore interface I noticed this method
> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
> method?, should I use the partition definition in the xml mapping file for
> this?.*
>
> The method getPartitions(Query) is related to Hadoop. Apache Gora
> integrates with Hadoop implementing a custom Map and Reduce that allows to
> get/write Entities directly.
> You can take a look at HBase's implementation [4], which relies o.a.h.hbase.mapreduce.TableInputFormatBase
> [5] to compute the splits (start key---end key) with the location of the
> split to create a colection of partitions [6].
>
> So, if Kudu is allowed to perform computation using local kudu splits,
> then this method does the needed preparation to allow to "send the
> computation to where the data is locally".
>
> In any case, you can see that:
>
> - MongoDB store implementation does not implement splitting [7]
> - Cassandra store implementation does not implement splitting [8]
> - Aerospike store implementation does not implement splitting [9]
> - Accumulo store implementation* does* implement splitting [10]
>
> If Kudu has a method to get the different splits for a table and its
> locations, then you will be able to implement the full feature.
>
> This is Hadoop related and it is not trivial. I haven't elaborated much,
> so if you find you need more information let me know :)
>
>
>
I will check whether Kudu has these features in order to implement this
method. If not I will use the default implementation found in other
backends.
> About Queries, what I can tell is that Hbase only implements "Start key" +
> "End key" because it has only 2 operations: "get" and "scan", and the
> querying is for "scan" operation, were you want an interval (or all) of the
> rows. Does Kudu have more querying functionality?
>
>
Yes, Kudu implements a Scanner for querying data among with conditional
predicates for filtering. I am using those classes.
> About other topic, I am trying to install Kudu in standalone (all in 1
> node). Do you use a Cloudera installation or do you have a standalone
> installation? How do you do it? I found some instructions, but they talk
> about compiling Kudu [11]. I was looking for something like HBase, that it
> is unzip + execute "hbase start".
>
>
I am using an embedded mini-cluster which comes with compiled binaries and
can be used with maven[1] for testing my code. Once I get it mature enough
I think I will be testing the datastore with a docker container [2]. I
could not find a unzip+execute bundle either and I am kinda noob for
compiling it myself.
[1]
https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing
[2] https://hub.docker.com/r/usuresearch/apache-kudu/
> Good job and thank you!! :)
>
> Regards,
>
> Alfonso Nishikawa
>
>
> [1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
> [2] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
> [3] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
> [4] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
> [5] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
> [6] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
> [7] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
> [8] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
> [9] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
> [10] -
> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
> [11] - https://kudu.apache.org/docs/installation.html
>
>
> El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> As every week I updated my report in the Wiki[1]. Also, I pushed my last
>> commits to my branch [2]. Please give it a look if you have time.
>>
>> This week, I will be continue working in the Queries implementation,
>> please reach me out if you have any suggestions.
>>
>> Also, while reviewing the datastore interface I noticed this method
>> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
>> method?, should I use the partition definition in the xml mapping file for
>> this?.
>>
>> Cheers,
>> John.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>>
>> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> I received my first evaluation from the Google Summer of Code program
>>> with a positive result. Thanks so much for your support and confidence to
>>> the project and me.
>>>
>>> I updated my report of this week in the Wiki[1]. Also, I pushed my last
>>> commits to my branch [2].
>>>
>>> This week, I will be reviewing my the serialization/ deserialization
>>> process in order to identify optimizations specific for Kudu. Because I
>>> used a generic methods of other backends which probably could be better
>>> tuned for kudu. Also, I will start working on the Queries implementation.
>>>
>>> BTW, I added a question to the wiki about Date types. Please give it a
>>> look if you have time.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Cheers,
>>> John
>>>
>>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Carlos.
>>>>
>>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>>
>>>> Best,
>>>> John.
>>>>
>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi John
>>>>>
>>>>> The first Google Summer of Code evaluation is due on June 28th. Please
>>>>> make sure you submit your Mentors' evaluation on time.
>>>>>
>>>>> Regards,
>>>>> Carlos
>>>>>
>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed
>>>>>> my last commits to my branch [2].
>>>>>>
>>>>>> As I mentioned in the reports I would like to know how datastores
>>>>>> deal with flush(), should it work always manually executed?.
>>>>>>
>>>>>> Finally, This week I will be implementing object
>>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>>> you have any suggestions on how to proceed with this task?.
>>>>>>
>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Cheers,
>>>>>> John
>>>>>>
>>>>>>
>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<
>>>>>> carlosrmng@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi John
>>>>>>>
>>>>>>> Your last changes look good to me. Keep it up. But, I noticed that
>>>>>>> you have created an Enumeration for datatypes, which is very similar to the
>>>>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>>>>> avoid code duplication.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Carlos
>>>>>>>
>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that my
>>>>>>>> code is lacking some javadoc documentation I think I will be working on
>>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> John.
>>>>>>>>
>>>>>>>>
>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi Alfonso.
>>>>>>>>>
>>>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi, John.
>>>>>>>>>>
>>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - How to represent partitioning configurations on the mapping
>>>>>>>>>> file.
>>>>>>>>>>
>>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>>
>>>>>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>>
>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far
>>>>>>>>>> from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>>
>>>>>>>>>> [1] -
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi, John.
>>>>>>>>>>>
>>>>>>>>>>> Thank you!
>>>>>>>>>>> Things I have seen:
>>>>>>>>>>>
>>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>>> should not set the version there.
>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you
>>>>>>>>>>> will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>>> isn't it?
>>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions
>>>>>>>>>>> (at least in Java since we have the checked ones) like in [11]. You can
>>>>>>>>>>> wrap them in GoraException. An example is [12]
>>>>>>>>>>>
>>>>>>>>>>> And nothing more :)
>>>>>>>>>>> Keep going, good job.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>>> [2] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>>> [3] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>>> [4] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>>> [5] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>>> [6] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>>> [7] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>>> [8] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>>> [9] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>>> [10] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>>> [11] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>>> [12] -
>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all.
>>>>>>>>>>>>
>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next
>>>>>>>>>>>> week I think I should be focusing on the create schema operation and
>>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>>>>>>> available here [2]
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
Sorry for the delay, I am changing work and I have been very busy :( I will
try to answer your questions :)
*> In the Employee example there is a field called 'dateOfBirth'. I tried
to map that field with the UNIXTIME_MICROS datatype of Kudu (I intuitively
assumed this is a date.). However, in the java world the Employee field is
a Long value and the kudu datatype is a Timestamp. So, I was wondering
whether I should force the usage of the UNIXTIME_MICROS datatype for this
field or just use a LONG datatype in Kudu.*
In Avro 1.8 were introduced "Logical Types" so there is a "date" type with
an underlying "int" [1]. It's the first time I read about because until the
last version upgrade of Avro this weren't there. I would suggest to ignore
"dates" and map dateOfBirth as long, since in any case -in avro- the value
is the unix epoch. After this first approach, a design improvement would be
great, though :)
- Would be good to have in the mapping a "timestamp" type so KuduStore
converts between the Entity long field <-> Kudu timestamp storage?
- Is there any other approach?
*> What is the Gora's policy regarding flush()? *
*> KuduClient has multiple flushing modes
<https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and
also can set time interval
<https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int->
for automatic flush.*
*> Should theses behaviors be configurable using gora.properties file? or
just use the default configurations.*
What we do in HBase is configure an autoflush option in gora.properties [2]
which is used when instanced the Table, but at the same time we implement
the flush() method to force the flush [3]. I would suggest to follow that
example, but adding the flushing options of Kudu. What flushing mode (and
time interval if it applies) do you suggest?
*> Also, while reviewing the datastore interface I noticed this method
'getPartitions(Query<K, T> query)'. What is the expected behavior of this
method?, should I use the partition definition in the xml mapping file for
this?.*
The method getPartitions(Query) is related to Hadoop. Apache Gora
integrates with Hadoop implementing a custom Map and Reduce that allows to
get/write Entities directly.
You can take a look at HBase's implementation [4], which relies
o.a.h.hbase.mapreduce.TableInputFormatBase
[5] to compute the splits (start key---end key) with the location of the
split to create a colection of partitions [6].
So, if Kudu is allowed to perform computation using local kudu splits, then
this method does the needed preparation to allow to "send the computation
to where the data is locally".
In any case, you can see that:
- MongoDB store implementation does not implement splitting [7]
- Cassandra store implementation does not implement splitting [8]
- Aerospike store implementation does not implement splitting [9]
- Accumulo store implementation* does* implement splitting [10]
If Kudu has a method to get the different splits for a table and its
locations, then you will be able to implement the full feature.
This is Hadoop related and it is not trivial. I haven't elaborated much, so
if you find you need more information let me know :)
About Queries, what I can tell is that Hbase only implements "Start key" +
"End key" because it has only 2 operations: "get" and "scan", and the
querying is for "scan" operation, were you want an interval (or all) of the
rows. Does Kudu have more querying functionality?
About other topic, I am trying to install Kudu in standalone (all in 1
node). Do you use a Cloudera installation or do you have a standalone
installation? How do you do it? I found some instructions, but they talk
about compiling Kudu [11]. I was looking for something like HBase, that it
is unzip + execute "hbase start".
Good job and thank you!! :)
Regards,
Alfonso Nishikawa
[1] - https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types
[2] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175
[3] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458
[4] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472
[5] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479
[6] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517
[7] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533
[8] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292
[9] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369
[10] -
https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902
[11] - https://kudu.apache.org/docs/installation.html
El lun., 8 jul. 2019 a las 3:42, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> As every week I updated my report in the Wiki[1]. Also, I pushed my last
> commits to my branch [2]. Please give it a look if you have time.
>
> This week, I will be continue working in the Queries implementation,
> please reach me out if you have any suggestions.
>
> Also, while reviewing the datastore interface I noticed this method
> 'getPartitions(Query<K, T> query)'. What is the expected behavior of this
> method?, should I use the partition definition in the xml mapping file for
> this?.
>
> Cheers,
> John.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
>
> El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> I received my first evaluation from the Google Summer of Code program
>> with a positive result. Thanks so much for your support and confidence to
>> the project and me.
>>
>> I updated my report of this week in the Wiki[1]. Also, I pushed my last
>> commits to my branch [2].
>>
>> This week, I will be reviewing my the serialization/ deserialization
>> process in order to identify optimizations specific for Kudu. Because I
>> used a generic methods of other backends which probably could be better
>> tuned for kudu. Also, I will start working on the Queries implementation.
>>
>> BTW, I added a question to the wiki about Date types. Please give it a
>> look if you have time.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Cheers,
>> John
>>
>> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi Carlos.
>>>
>>> Thanks for the reminder. I submitted the form yesterday. :D
>>>
>>> Best,
>>> John.
>>>
>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
>>> escribió:
>>>
>>>> Hi John
>>>>
>>>> The first Google Summer of Code evaluation is due on June 28th. Please
>>>> make sure you submit your Mentors' evaluation on time.
>>>>
>>>> Regards,
>>>> Carlos
>>>>
>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed
>>>>> my last commits to my branch [2].
>>>>>
>>>>> As I mentioned in the reports I would like to know how datastores deal
>>>>> with flush(), should it work always manually executed?.
>>>>>
>>>>> Finally, This week I will be implementing object
>>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>>> you have any suggestions on how to proceed with this task?.
>>>>>
>>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Cheers,
>>>>> John
>>>>>
>>>>>
>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi John
>>>>>>
>>>>>> Your last changes look good to me. Keep it up. But, I noticed that
>>>>>> you have created an Enumeration for datatypes, which is very similar to the
>>>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>>>> avoid code duplication.
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Carlos
>>>>>>
>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I updated my report of this week on the Wiki[1]. I noticed that my
>>>>>>> code is lacking some javadoc documentation I think I will be working on
>>>>>>> that this week, also I would like to enable and check schema management
>>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>
>>>>>>> Cheers,
>>>>>>> John.
>>>>>>>
>>>>>>>
>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi Alfonso.
>>>>>>>>
>>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> John
>>>>>>>>
>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi, John.
>>>>>>>>>
>>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - How to represent partitioning configurations on the mapping
>>>>>>>>> file.
>>>>>>>>>
>>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>>
>>>>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>>
>>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far
>>>>>>>>> from 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>>
>>>>>>>>> [1] -
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Hi, John.
>>>>>>>>>>
>>>>>>>>>> Thank you!
>>>>>>>>>> Things I have seen:
>>>>>>>>>>
>>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>>> should not set the version there.
>>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you
>>>>>>>>>> will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>>> isn't it?
>>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>>> the one for HBase at [10]
>>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>>>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>>>>>>> them in GoraException. An example is [12]
>>>>>>>>>>
>>>>>>>>>> And nothing more :)
>>>>>>>>>> Keep going, good job.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>>> [2] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>>> [3] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>>> [4] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>>> [5] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>>> [6] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>>> [7] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>>> [8] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>>> [9] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>>> [10] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>>> [11] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>>> [12] -
>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<
>>>>>>>>>> jhnmora000@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next
>>>>>>>>>>> week I think I should be focusing on the create schema operation and
>>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>>>>>> available here [2]
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
As every week I updated my report in the Wiki[1]. Also, I pushed my last
commits to my branch [2]. Please give it a look if you have time.
This week, I will be continue working in the Queries implementation, please
reach me out if you have any suggestions.
Also, while reviewing the datastore interface I noticed this method
'getPartitions(Query<K, T> query)'. What is the expected behavior of this
method?, should I use the partition definition in the xml mapping file for
this?.
Cheers,
John.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
El dom., 30 jun. 2019 a las 16:56, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> I received my first evaluation from the Google Summer of Code program with
> a positive result. Thanks so much for your support and confidence to the
> project and me.
>
> I updated my report of this week in the Wiki[1]. Also, I pushed my last
> commits to my branch [2].
>
> This week, I will be reviewing my the serialization/ deserialization
> process in order to identify optimizations specific for Kudu. Because I
> used a generic methods of other backends which probably could be better
> tuned for kudu. Also, I will start working on the Queries implementation.
>
> BTW, I added a question to the wiki about Date types. Please give it a
> look if you have time.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Cheers,
> John
>
> El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi Carlos.
>>
>> Thanks for the reminder. I submitted the form yesterday. :D
>>
>> Best,
>> John.
>>
>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
>> escribió:
>>
>>> Hi John
>>>
>>> The first Google Summer of Code evaluation is due on June 28th. Please
>>> make sure you submit your Mentors' evaluation on time.
>>>
>>> Regards,
>>> Carlos
>>>
>>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed my
>>>> last commits to my branch [2].
>>>>
>>>> As I mentioned in the reports I would like to know how datastores deal
>>>> with flush(), should it work always manually executed?.
>>>>
>>>> Finally, This week I will be implementing object
>>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>>> you have any suggestions on how to proceed with this task?.
>>>>
>>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Cheers,
>>>> John
>>>>
>>>>
>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi John
>>>>>
>>>>> Your last changes look good to me. Keep it up. But, I noticed that you
>>>>> have created an Enumeration for datatypes, which is very similar to the
>>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>>> avoid code duplication.
>>>>>
>>>>> [1]
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>>
>>>>>
>>>>> Best,
>>>>> Carlos
>>>>>
>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> I updated my report of this week on the Wiki[1]. I noticed that my
>>>>>> code is lacking some javadoc documentation I think I will be working on
>>>>>> that this week, also I would like to enable and check schema management
>>>>>> tests (createSchema, existsSchema, etc.).
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>
>>>>>> Cheers,
>>>>>> John.
>>>>>>
>>>>>>
>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi Alfonso.
>>>>>>>
>>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>>
>>>>>>> Best,
>>>>>>> John
>>>>>>>
>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi, John.
>>>>>>>>
>>>>>>>> Regarding your questions at the report [1]:
>>>>>>>>
>>>>>>>>
>>>>>>>> - How to represent partitioning configurations on the mapping
>>>>>>>> file.
>>>>>>>>
>>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>>
>>>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>>
>>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>>>>>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>>
>>>>>>>> [1] -
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Alfonso Nishikawa
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi, John.
>>>>>>>>>
>>>>>>>>> Thank you!
>>>>>>>>> Things I have seen:
>>>>>>>>>
>>>>>>>>> - The version of a maven dependency [1] should go on the
>>>>>>>>> Dependency Management of the root pom [2]. Same for [3] and from there,
>>>>>>>>> should not set the version there.
>>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you
>>>>>>>>> will need root permission to run it. The default port for kudu is 7051,
>>>>>>>>> isn't it?
>>>>>>>>> - I would ask you to add the same functionality to load the
>>>>>>>>> mapping from configuration as in HBase's store [7] in you KuduStore [8].
>>>>>>>>> This will have implications on your readMapping at [9], so take a look at
>>>>>>>>> the one for HBase at [10]
>>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>>>>>> them in GoraException. An example is [12]
>>>>>>>>>
>>>>>>>>> And nothing more :)
>>>>>>>>> Keep going, good job.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>>> [2] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>>> [3] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>>> [4] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>>> [5] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>>> [6] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>>> [7] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>>> [8] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>>> [9] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>>> [10] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>>> [11] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>>> [12] -
>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Alfonso Nishikawa
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>>>>>>> escribió:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next
>>>>>>>>>> week I think I should be focusing on the create schema operation and
>>>>>>>>>> solving the issue of the partitioning configurations in the mapping file.
>>>>>>>>>>
>>>>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>>>>> available here [2]
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
I received my first evaluation from the Google Summer of Code program with
a positive result. Thanks so much for your support and confidence to the
project and me.
I updated my report of this week in the Wiki[1]. Also, I pushed my last
commits to my branch [2].
This week, I will be reviewing my the serialization/ deserialization
process in order to identify optimizations specific for Kudu. Because I
used a generic methods of other backends which probably could be better
tuned for kudu. Also, I will start working on the Queries implementation.
BTW, I added a question to the wiki about Date types. Please give it a look
if you have time.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Cheers,
John
El jue., 27 jun. 2019 a las 21:02, John Mora (<jh...@gmail.com>)
escribió:
> Hi Carlos.
>
> Thanks for the reminder. I submitted the form yesterday. :D
>
> Best,
> John.
>
> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
> escribió:
>
>> Hi John
>>
>> The first Google Summer of Code evaluation is due on June 28th. Please
>> make sure you submit your Mentors' evaluation on time.
>>
>> Regards,
>> Carlos
>>
>> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed my
>>> last commits to my branch [2].
>>>
>>> As I mentioned in the reports I would like to know how datastores deal
>>> with flush(), should it work always manually executed?.
>>>
>>> Finally, This week I will be implementing object
>>> serialization/deserialization in the methods put, get, delete, exists. Do
>>> you have any suggestions on how to proceed with this task?.
>>>
>>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Cheers,
>>> John
>>>
>>>
>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
>>> escribió:
>>>
>>>> Hi John
>>>>
>>>> Your last changes look good to me. Keep it up. But, I noticed that you
>>>> have created an Enumeration for datatypes, which is very similar to the
>>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>>> avoid code duplication.
>>>>
>>>> [1]
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>>
>>>>
>>>> Best,
>>>> Carlos
>>>>
>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> I updated my report of this week on the Wiki[1]. I noticed that my
>>>>> code is lacking some javadoc documentation I think I will be working on
>>>>> that this week, also I would like to enable and check schema management
>>>>> tests (createSchema, existsSchema, etc.).
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>
>>>>> Cheers,
>>>>> John.
>>>>>
>>>>>
>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi Alfonso.
>>>>>>
>>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>>
>>>>>> Best,
>>>>>> John
>>>>>>
>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi, John.
>>>>>>>
>>>>>>> Regarding your questions at the report [1]:
>>>>>>>
>>>>>>>
>>>>>>> - How to represent partitioning configurations on the mapping
>>>>>>> file.
>>>>>>>
>>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>>
>>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>>
>>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>>>>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>>
>>>>>>> [1] -
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Alfonso Nishikawa
>>>>>>>
>>>>>>>
>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Hi, John.
>>>>>>>>
>>>>>>>> Thank you!
>>>>>>>> Things I have seen:
>>>>>>>>
>>>>>>>> - The version of a maven dependency [1] should go on the Dependency
>>>>>>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>>>>>>> the version there.
>>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will
>>>>>>>> need root permission to run it. The default port for kudu is 7051, isn't it?
>>>>>>>> - I would ask you to add the same functionality to load the mapping
>>>>>>>> from configuration as in HBase's store [7] in you KuduStore [8]. This will
>>>>>>>> have implications on your readMapping at [9], so take a look at the one for
>>>>>>>> HBase at [10]
>>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>>>>> them in GoraException. An example is [12]
>>>>>>>>
>>>>>>>> And nothing more :)
>>>>>>>> Keep going, good job.
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>>> [3] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>>> [4] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>>> [5] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>>> [6] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>>> [7] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>>> [8] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>>> [9] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>>> [10] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>>> [11] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>>> [12] -
>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Alfonso Nishikawa
>>>>>>>>
>>>>>>>>
>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next week
>>>>>>>>> I think I should be focusing on the create schema operation and solving the
>>>>>>>>> issue of the partitioning configurations in the mapping file.
>>>>>>>>>
>>>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>>>> available here [2]
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Carlos.
Thanks for the reminder. I submitted the form yesterday. :D
Best,
John.
El jue., 27 jun. 2019 a las 17:34, carlos muñoz (<ca...@gmail.com>)
escribió:
> Hi John
>
> The first Google Summer of Code evaluation is due on June 28th. Please
> make sure you submit your Mentors' evaluation on time.
>
> Regards,
> Carlos
>
> El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed my
>> last commits to my branch [2].
>>
>> As I mentioned in the reports I would like to know how datastores deal
>> with flush(), should it work always manually executed?.
>>
>> Finally, This week I will be implementing object
>> serialization/deserialization in the methods put, get, delete, exists. Do
>> you have any suggestions on how to proceed with this task?.
>>
>> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Cheers,
>> John
>>
>>
>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
>> escribió:
>>
>>> Hi John
>>>
>>> Your last changes look good to me. Keep it up. But, I noticed that you
>>> have created an Enumeration for datatypes, which is very similar to the
>>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>>> avoid code duplication.
>>>
>>> [1]
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>>
>>>
>>> Best,
>>> Carlos
>>>
>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> I updated my report of this week on the Wiki[1]. I noticed that my code
>>>> is lacking some javadoc documentation I think I will be working on that
>>>> this week, also I would like to enable and check schema management tests
>>>> (createSchema, existsSchema, etc.).
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>
>>>> Cheers,
>>>> John.
>>>>
>>>>
>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi Alfonso.
>>>>>
>>>>> Thanks so much for your feedback. I am working on your comments.
>>>>>
>>>>> Best,
>>>>> John
>>>>>
>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi, John.
>>>>>>
>>>>>> Regarding your questions at the report [1]:
>>>>>>
>>>>>>
>>>>>> - How to represent partitioning configurations on the mapping
>>>>>> file.
>>>>>>
>>>>>> This was discussed in other emails, isn't it? :)
>>>>>>
>>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin,
>>>>>> which needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>>
>>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>>>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>>
>>>>>> [1] -
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>>
>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>>
>>>>>>> Hi, John.
>>>>>>>
>>>>>>> Thank you!
>>>>>>> Things I have seen:
>>>>>>>
>>>>>>> - The version of a maven dependency [1] should go on the Dependency
>>>>>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>>>>>> the version there.
>>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>>> - Missing "t" in "localhost" at [6].
>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will
>>>>>>> need root permission to run it. The default port for kudu is 7051, isn't it?
>>>>>>> - I would ask you to add the same functionality to load the mapping
>>>>>>> from configuration as in HBase's store [7] in you KuduStore [8]. This will
>>>>>>> have implications on your readMapping at [9], so take a look at the one for
>>>>>>> HBase at [10]
>>>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>>>> them in GoraException. An example is [12]
>>>>>>>
>>>>>>> And nothing more :)
>>>>>>> Keep going, good job.
>>>>>>>
>>>>>>>
>>>>>>> [1] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>>> [3] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>>> [4] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>>> [5] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>>> [6] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>>> [7] -
>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>>> [8] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>>> [9] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>>> [10] -
>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>>> [11] -
>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>>> [12] -
>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Alfonso Nishikawa
>>>>>>>
>>>>>>>
>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next week
>>>>>>>> I think I should be focusing on the create schema operation and solving the
>>>>>>>> issue of the partitioning configurations in the mapping file.
>>>>>>>>
>>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>>> available here [2]
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> John
>>>>>>>>
>>>>>>>>
Re: Kudu datastore reports
Posted by carlos muñoz <ca...@gmail.com>.
Hi John
The first Google Summer of Code evaluation is due on June 28th. Please make
sure you submit your Mentors' evaluation on time.
Regards,
Carlos
El dom., 23 jun. 2019 a las 18:29, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> FYI, I updated my report of this week on the Wiki[1]. Also, I pushed my
> last commits to my branch [2].
>
> As I mentioned in the reports I would like to know how datastores deal
> with flush(), should it work always manually executed?.
>
> Finally, This week I will be implementing object
> serialization/deserialization in the methods put, get, delete, exists. Do
> you have any suggestions on how to proceed with this task?.
>
> Footnote: Thanks for the feedback Carlos, I fixed the problem.
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Cheers,
> John
>
>
> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
> escribió:
>
>> Hi John
>>
>> Your last changes look good to me. Keep it up. But, I noticed that you
>> have created an Enumeration for datatypes, which is very similar to the
>> kudu-client's [2]. Probably you should replace [1] for [2] in order to
>> avoid code duplication.
>>
>> [1]
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
>> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>>
>>
>> Best,
>> Carlos
>>
>> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> I updated my report of this week on the Wiki[1]. I noticed that my code
>>> is lacking some javadoc documentation I think I will be working on that
>>> this week, also I would like to enable and check schema management tests
>>> (createSchema, existsSchema, etc.).
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>
>>> Cheers,
>>> John.
>>>
>>>
>>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi Alfonso.
>>>>
>>>> Thanks so much for your feedback. I am working on your comments.
>>>>
>>>> Best,
>>>> John
>>>>
>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi, John.
>>>>>
>>>>> Regarding your questions at the report [1]:
>>>>>
>>>>>
>>>>> - How to represent partitioning configurations on the mapping file.
>>>>>
>>>>> This was discussed in other emails, isn't it? :)
>>>>>
>>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin, which
>>>>> needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>>
>>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>>
>>>>> [1] -
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>>
>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>>
>>>>>> Hi, John.
>>>>>>
>>>>>> Thank you!
>>>>>> Things I have seen:
>>>>>>
>>>>>> - The version of a maven dependency [1] should go on the Dependency
>>>>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>>>>> the version there.
>>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>>> - Missing "t" in "localhost" at [6].
>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will
>>>>>> need root permission to run it. The default port for kudu is 7051, isn't it?
>>>>>> - I would ask you to add the same functionality to load the mapping
>>>>>> from configuration as in HBase's store [7] in you KuduStore [8]. This will
>>>>>> have implications on your readMapping at [9], so take a look at the one for
>>>>>> HBase at [10]
>>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>>> them in GoraException. An example is [12]
>>>>>>
>>>>>> And nothing more :)
>>>>>> Keep going, good job.
>>>>>>
>>>>>>
>>>>>> [1] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>>> [3] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>>> [4] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>>> [5] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>>> [6] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>>> [7] -
>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>>> [8] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>>> [9] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>>> [10] -
>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>>> [11] -
>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>>> [12] -
>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Alfonso Nishikawa
>>>>>>
>>>>>>
>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> I have just updated my weekly reports on Cwiki [1]. This next week I
>>>>>>> think I should be focusing on the create schema operation and solving the
>>>>>>> issue of the partitioning configurations in the mapping file.
>>>>>>>
>>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>>> available here [2]
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>>
>>>>>>> Best,
>>>>>>> John
>>>>>>>
>>>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
FYI, I updated my report of this week on the Wiki[1]. Also, I pushed my
last commits to my branch [2].
As I mentioned in the reports I would like to know how datastores deal with
flush(), should it work always manually executed?.
Finally, This week I will be implementing object
serialization/deserialization in the methods put, get, delete, exists. Do
you have any suggestions on how to proceed with this task?.
Footnote: Thanks for the feedback Carlos, I fixed the problem.
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
[2] https://github.com/jhnmora000/gora/tree/GORA-485
Cheers,
John
El lun., 17 jun. 2019 a las 22:58, carlos muñoz (<ca...@gmail.com>)
escribió:
> Hi John
>
> Your last changes look good to me. Keep it up. But, I noticed that you
> have created an Enumeration for datatypes, which is very similar to the
> kudu-client's [2]. Probably you should replace [1] for [2] in order to
> avoid code duplication.
>
> [1]
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
> [2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
>
>
> Best,
> Carlos
>
> El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> I updated my report of this week on the Wiki[1]. I noticed that my code
>> is lacking some javadoc documentation I think I will be working on that
>> this week, also I would like to enable and check schema management tests
>> (createSchema, existsSchema, etc.).
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>
>> Cheers,
>> John.
>>
>>
>> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi Alfonso.
>>>
>>> Thanks so much for your feedback. I am working on your comments.
>>>
>>> Best,
>>> John
>>>
>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi, John.
>>>>
>>>> Regarding your questions at the report [1]:
>>>>
>>>>
>>>> - How to represent partitioning configurations on the mapping file.
>>>>
>>>> This was discussed in other emails, isn't it? :)
>>>>
>>>> - KuduTestHarness requires the Maven plugin os-maven-plugin, which
>>>> needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>>
>>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>>
>>>> [1] -
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Alfonso Nishikawa
>>>>
>>>>
>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>>> alfonso.nishikawa@gmail.com>) escribió:
>>>>
>>>>> Hi, John.
>>>>>
>>>>> Thank you!
>>>>> Things I have seen:
>>>>>
>>>>> - The version of a maven dependency [1] should go on the Dependency
>>>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>>>> the version there.
>>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>>> - Set the indentation to 2 spaces for the pom [5]
>>>>> - Missing "t" in "localhost" at [6].
>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will
>>>>> need root permission to run it. The default port for kudu is 7051, isn't it?
>>>>> - I would ask you to add the same functionality to load the mapping
>>>>> from configuration as in HBase's store [7] in you KuduStore [8]. This will
>>>>> have implications on your readMapping at [9], so take a look at the one for
>>>>> HBase at [10]
>>>>> - I know it is in other backends, but avoid RuntimeExceptions (at
>>>>> least in Java since we have the checked ones) like in [11]. You can wrap
>>>>> them in GoraException. An example is [12]
>>>>>
>>>>> And nothing more :)
>>>>> Keep going, good job.
>>>>>
>>>>>
>>>>> [1] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>>> [3] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>>> [4] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>>> [5] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>>> [6] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>>> [7] -
>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>>> [8] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>>> [9] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>>> [10] -
>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>>> [11] -
>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>>> [12] -
>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>>
>>>>> Regards,
>>>>>
>>>>> Alfonso Nishikawa
>>>>>
>>>>>
>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>>> escribió:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> I have just updated my weekly reports on Cwiki [1]. This next week I
>>>>>> think I should be focusing on the create schema operation and solving the
>>>>>> issue of the partitioning configurations in the mapping file.
>>>>>>
>>>>>> Please let me know if you have suggestions, my last commits are
>>>>>> available here [2]
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>>
>>>>>> Best,
>>>>>> John
>>>>>>
>>>>>>
Re: Kudu datastore reports
Posted by carlos muñoz <ca...@gmail.com>.
Hi John
Your last changes look good to me. Keep it up. But, I noticed that you have
created an Enumeration for datatypes, which is very similar to the
kudu-client's [2]. Probably you should replace [1] for [2] in order to
avoid code duplication.
[1]
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76
[2] https://kudu.apache.org/apidocs/org/apache/kudu/Type.html
Best,
Carlos
El sáb., 15 jun. 2019 a las 12:01, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> I updated my report of this week on the Wiki[1]. I noticed that my code is
> lacking some javadoc documentation I think I will be working on that this
> week, also I would like to enable and check schema management tests
> (createSchema, existsSchema, etc.).
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>
> Cheers,
> John.
>
>
> El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi Alfonso.
>>
>> Thanks so much for your feedback. I am working on your comments.
>>
>> Best,
>> John
>>
>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> Regarding your questions at the report [1]:
>>>
>>>
>>> - How to represent partitioning configurations on the mapping file.
>>>
>>> This was discussed in other emails, isn't it? :)
>>>
>>> - KuduTestHarness requires the Maven plugin os-maven-plugin, which
>>> needs Maven 3.1.1+, is it a problem for Apache Gora?
>>>
>>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>>
>>> [1] -
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>
>>>
>>> Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>>
>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>>> alfonso.nishikawa@gmail.com>) escribió:
>>>
>>>> Hi, John.
>>>>
>>>> Thank you!
>>>> Things I have seen:
>>>>
>>>> - The version of a maven dependency [1] should go on the Dependency
>>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>>> the version there.
>>>> - Set test dependencies' scope to test, at [4] and from there.
>>>> - Set the indentation to 2 spaces for the pom [5]
>>>> - Missing "t" in "localhost" at [6].
>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will
>>>> need root permission to run it. The default port for kudu is 7051, isn't it?
>>>> - I would ask you to add the same functionality to load the mapping
>>>> from configuration as in HBase's store [7] in you KuduStore [8]. This will
>>>> have implications on your readMapping at [9], so take a look at the one for
>>>> HBase at [10]
>>>> - I know it is in other backends, but avoid RuntimeExceptions (at least
>>>> in Java since we have the checked ones) like in [11]. You can wrap them in
>>>> GoraException. An example is [12]
>>>>
>>>> And nothing more :)
>>>> Keep going, good job.
>>>>
>>>>
>>>> [1] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>>> [3] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>>> [4] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>>> [5] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>>> [6] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>>> [7] -
>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>>> [8] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>>> [9] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>>> [10] -
>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>>> [11] -
>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>>> [12] -
>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>>
>>>> Regards,
>>>>
>>>> Alfonso Nishikawa
>>>>
>>>>
>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>>> escribió:
>>>>
>>>>> Hi all.
>>>>>
>>>>> I have just updated my weekly reports on Cwiki [1]. This next week I
>>>>> think I should be focusing on the create schema operation and solving the
>>>>> issue of the partitioning configurations in the mapping file.
>>>>>
>>>>> Please let me know if you have suggestions, my last commits are
>>>>> available here [2]
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>>
>>>>> Best,
>>>>> John
>>>>>
>>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi all.
I updated my report of this week on the Wiki[1]. I noticed that my code is
lacking some javadoc documentation I think I will be working on that this
week, also I would like to enable and check schema management tests
(createSchema, existsSchema, etc.).
[1]
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
Cheers,
John.
El mar., 11 jun. 2019 a las 0:11, John Mora (<jh...@gmail.com>)
escribió:
> Hi Alfonso.
>
> Thanks so much for your feedback. I am working on your comments.
>
> Best,
> John
>
> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> Regarding your questions at the report [1]:
>>
>>
>> - How to represent partitioning configurations on the mapping file.
>>
>> This was discussed in other emails, isn't it? :)
>>
>> - KuduTestHarness requires the Maven plugin os-maven-plugin, which
>> needs Maven 3.1.1+, is it a problem for Apache Gora?
>>
>> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from
>> 3.1.1, and I assume everyone uses Maven 3 in a quite new version :)
>>
>> [1] -
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>
>>
>> Regards,
>>
>> Alfonso Nishikawa
>>
>>
>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
>> alfonso.nishikawa@gmail.com>) escribió:
>>
>>> Hi, John.
>>>
>>> Thank you!
>>> Things I have seen:
>>>
>>> - The version of a maven dependency [1] should go on the Dependency
>>> Management of the root pom [2]. Same for [3] and from there, should not set
>>> the version there.
>>> - Set test dependencies' scope to test, at [4] and from there.
>>> - Set the indentation to 2 spaces for the pom [5]
>>> - Missing "t" in "localhost" at [6].
>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will need
>>> root permission to run it. The default port for kudu is 7051, isn't it?
>>> - I would ask you to add the same functionality to load the mapping from
>>> configuration as in HBase's store [7] in you KuduStore [8]. This will have
>>> implications on your readMapping at [9], so take a look at the one for
>>> HBase at [10]
>>> - I know it is in other backends, but avoid RuntimeExceptions (at least
>>> in Java since we have the checked ones) like in [11]. You can wrap them in
>>> GoraException. An example is [12]
>>>
>>> And nothing more :)
>>> Keep going, good job.
>>>
>>>
>>> [1] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>>> [3] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>>> [4] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>>> [5] - https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>>> [6] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>>> [7] -
>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>>> [8] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>>> [9] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>>> [10] -
>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>>> [11] -
>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>>> [12] -
>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>>
>>> Regards,
>>>
>>> Alfonso Nishikawa
>>>
>>>
>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>>> escribió:
>>>
>>>> Hi all.
>>>>
>>>> I have just updated my weekly reports on Cwiki [1]. This next week I
>>>> think I should be focusing on the create schema operation and solving the
>>>> issue of the partitioning configurations in the mapping file.
>>>>
>>>> Please let me know if you have suggestions, my last commits are
>>>> available here [2]
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>>
>>>> Best,
>>>> John
>>>>
>>>>
Re: Kudu datastore reports
Posted by John Mora <jh...@gmail.com>.
Hi Alfonso.
Thanks so much for your feedback. I am working on your comments.
Best,
John
El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi, John.
>
> Regarding your questions at the report [1]:
>
>
> - How to represent partitioning configurations on the mapping file.
>
> This was discussed in other emails, isn't it? :)
>
> - KuduTestHarness requires the Maven plugin os-maven-plugin, which
> needs Maven 3.1.1+, is it a problem for Apache Gora?
>
> I believe it is not a problem. My Ubuntu comes with 3.6.0, far from 3.1.1,
> and I assume everyone uses Maven 3 in a quite new version :)
>
> [1] -
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>
>
> Regards,
>
> Alfonso Nishikawa
>
>
> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
> alfonso.nishikawa@gmail.com>) escribió:
>
>> Hi, John.
>>
>> Thank you!
>> Things I have seen:
>>
>> - The version of a maven dependency [1] should go on the Dependency
>> Management of the root pom [2]. Same for [3] and from there, should not set
>> the version there.
>> - Set test dependencies' scope to test, at [4] and from there.
>> - Set the indentation to 2 spaces for the pom [5]
>> - Missing "t" in "localhost" at [6].
>> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will need
>> root permission to run it. The default port for kudu is 7051, isn't it?
>> - I would ask you to add the same functionality to load the mapping from
>> configuration as in HBase's store [7] in you KuduStore [8]. This will have
>> implications on your readMapping at [9], so take a look at the one for
>> HBase at [10]
>> - I know it is in other backends, but avoid RuntimeExceptions (at least
>> in Java since we have the checked ones) like in [11]. You can wrap them in
>> GoraException. An example is [12]
>>
>> And nothing more :)
>> Keep going, good job.
>>
>>
>> [1] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
>> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
>> [3] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
>> [4] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
>> [5] - https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
>> [6] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
>> [7] -
>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
>> [8] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
>> [9] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
>> [10] -
>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
>> [11] -
>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
>> [12] -
>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>>
>> Regards,
>>
>> Alfonso Nishikawa
>>
>>
>> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
>> escribió:
>>
>>> Hi all.
>>>
>>> I have just updated my weekly reports on Cwiki [1]. This next week I
>>> think I should be focusing on the create schema operation and solving the
>>> issue of the partitioning configurations in the mapping file.
>>>
>>> Please let me know if you have suggestions, my last commits are
>>> available here [2]
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>>
>>> Best,
>>> John
>>>
>>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
Regarding your questions at the report [1]:
- How to represent partitioning configurations on the mapping file.
This was discussed in other emails, isn't it? :)
- KuduTestHarness requires the Maven plugin os-maven-plugin, which needs
Maven 3.1.1+, is it a problem for Apache Gora?
I believe it is not a problem. My Ubuntu comes with 3.6.0, far from 3.1.1,
and I assume everyone uses Maven 3 in a quite new version :)
[1] -
https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
Regards,
Alfonso Nishikawa
El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa (<
alfonso.nishikawa@gmail.com>) escribió:
> Hi, John.
>
> Thank you!
> Things I have seen:
>
> - The version of a maven dependency [1] should go on the Dependency
> Management of the root pom [2]. Same for [3] and from there, should not set
> the version there.
> - Set test dependencies' scope to test, at [4] and from there.
> - Set the indentation to 2 spaces for the pom [5]
> - Missing "t" in "localhost" at [6].
> - Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will need
> root permission to run it. The default port for kudu is 7051, isn't it?
> - I would ask you to add the same functionality to load the mapping from
> configuration as in HBase's store [7] in you KuduStore [8]. This will have
> implications on your readMapping at [9], so take a look at the one for
> HBase at [10]
> - I know it is in other backends, but avoid RuntimeExceptions (at least in
> Java since we have the checked ones) like in [11]. You can wrap them in
> GoraException. An example is [12]
>
> And nothing more :)
> Keep going, good job.
>
>
> [1] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
> [2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
> [3] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
> [4] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
> [5] - https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
> [6] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
> [7] -
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
> [8] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
> [9] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
> [10] -
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
> [11] -
> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
> [12] -
> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
>
> Regards,
>
> Alfonso Nishikawa
>
>
> El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
> escribió:
>
>> Hi all.
>>
>> I have just updated my weekly reports on Cwiki [1]. This next week I
>> think I should be focusing on the create schema operation and solving the
>> issue of the partitioning configurations in the mapping file.
>>
>> Please let me know if you have suggestions, my last commits are available
>> here [2]
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
>> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>>
>> Best,
>> John
>>
>>
Re: Kudu datastore reports
Posted by Alfonso Nishikawa <al...@gmail.com>.
Hi, John.
Thank you!
Things I have seen:
- The version of a maven dependency [1] should go on the Dependency
Management of the root pom [2]. Same for [3] and from there, should not set
the version there.
- Set test dependencies' scope to test, at [4] and from there.
- Set the indentation to 2 spaces for the pom [5]
- Missing "t" in "localhost" at [6].
- Port 13 for Kudu? That is "Daytime Protocol" RFC 867 and you will need
root permission to run it. The default port for kudu is 7051, isn't it?
- I would ask you to add the same functionality to load the mapping from
configuration as in HBase's store [7] in you KuduStore [8]. This will have
implications on your readMapping at [9], so take a look at the one for
HBase at [10]
- I know it is in other backends, but avoid RuntimeExceptions (at least in
Java since we have the checked ones) like in [11]. You can wrap them in
GoraException. An example is [12]
And nothing more :)
Keep going, good job.
[1] - https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98
[2] - https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890
[3] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121
[4] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180
[5] - https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml
[6] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18
[7] -
https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92
[8] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53
[9] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81
[10] -
https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822
[11] -
https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141
[12] -
https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268
Regards,
Alfonso Nishikawa
El sáb., 8 jun. 2019 a las 20:26, John Mora (<jh...@gmail.com>)
escribió:
> Hi all.
>
> I have just updated my weekly reports on Cwiki [1]. This next week I think
> I should be focusing on the create schema operation and solving the issue
> of the partitioning configurations in the mapping file.
>
> Please let me know if you have suggestions, my last commits are available
> here [2]
>
> [1]
> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports
> [2] https://github.com/jhnmora000/gora/tree/GORA-485
>
> Best,
> John
>
>