You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Ajay <aj...@gmail.com> on 2015/07/23 08:11:53 UTC
Lens driver for Cassandra
Hi,
I recently noticed about Apache Lens project. Currently we are building
REST APIs for Apache Cassandra (proprietary) as there no such Apache open
source project exists. Now as Lens supports REST APIs for Hadoop and JDBC,
I want to know is there any plan to support for Apache Cassandra as its
support CQL and Spark SQL (thru Spark connector) which are more SQL like.
If yes, I wish to know the details and contribute as well.
Thanks
Ajay
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
Yes with lombok plugin, these errors are gone.
Thanks
Ajay
On Fri, Jul 24, 2015 at 2:40 PM, Ajay <aj...@gmail.com> wrote:
> For example 1 such error in QueryExecutionServiceImp.java Lens Server
> project
>
>
> private LensResultSet getResultset(QueryHandle queryHandle) throws LensException {
> QueryContext ctx = allQueries.get(queryHandle);
> if (ctx == null) {
> return getResultsetFromDAO(queryHandle);
> } else {
> synchronized (ctx) {
> if (ctx.isFinishedQueryPersisted()) {
> return getResultsetFromDAO(queryHandle);
> }
> LensResultSet resultSet = resultSets.get(queryHandle);
> if (resultSet == null) {
> if (ctx.isPersistent() && ctx.getQueryOutputFormatter() != null) {
> resultSets
> .put(queryHandle,
> new LensPersistentResult(
> ctx.getQueryOutputFormatter().getMetadata(),
> ctx.getQueryOutputFormatter().getFinalOutputPath(),
> ctx.getQueryOutputFormatter().getNumRows()));
> } else if (allQueries.get(queryHandle).isResultAvailableInDriver()) {
> resultSet = allQueries.get(queryHandle).getSelectedDriver().fetchResultSet(allQueries.get(queryHandle));
> resultSets.put(queryHandle, resultSet);
> } else {
> throw new NotFoundException("Result set not available for query:" + queryHandle);
> }
> }
> }
> if (resultSets.get(queryHandle) instanceof InMemoryResultSet) {
> ((InMemoryResultSet) resultSets.get(queryHandle)).seekToStart();
> }
> return resultSets.get(queryHandle);
> }
> }
>
> There are no such methods in QueryContext() but annotated with @Getter and @Setter. Do we need any plugin to solve it?
>
> Thanks
>
> Ajay
>
>
>
> On Fri, Jul 24, 2015 at 2:33 PM, Rajat Khandelwal <ra...@gmail.com>
> wrote:
>
>> Yes, can you paste the exact error?
>>
>> On Fri, Jul 24, 2015 at 2:11 PM Yash Sharma <ya...@gmail.com> wrote:
>>
>> > Its certainly something wrt Env. Can you just copy paste couple of
>> errors ?
>> >
>> > On Fri, Jul 24, 2015 at 2:00 PM, Ajay <aj...@gmail.com> wrote:
>> >
>> > > I did the same from command line.
>> > >
>> > > mvn clean install -DskipTests -DskipCheck
>> > > mvn idea:idea
>> > >
>> > >
>> > > Java version is 1.8
>> > >
>> > > I tried opening the project directly and selected
>> > > /home/ajay/openSource/incubator-lens. Also I tried importing as maven
>> > > project. Both the cases I see compilation errors.
>> > >
>> > > -
>> > > Ajay
>> > >
>> > >
>> > > On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <
>> > rajatgupta59@gmail.com>
>> > > wrote:
>> > >
>> > > > I'm using idea too. There are no compilation issues. Can you do
>> > > >
>> > > > mvn clean install -DskipTests -DskipCheck
>> > > > mvn idea:idea
>> > > >
>> > > > from command line.
>> > > >
>> > > >
>> > > > Also, check your java version. Lens is configured for java >=7
>> > > >
>> > > > On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
>> > > >
>> > > > > From console, the build goes thru fine.
>> > > > >
>> > > > > From Idea IDE, it is giving compilation errors (say one such file
>> > > > > QueryExecutionServiceImp.java
>> > > > > Lens Server project).
>> > > > >
>> > > > > Are you guys using Eclipse?
>> > > > >
>> > > > >
>> > > > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Could you share the error. Might be env issues.
>> > > > > > Also can you build from the console directly ?
>> > > > > >
>> > > > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com>
>> > wrote:
>> > > > > >
>> > > > > > > Thanks.
>> > > > > > >
>> > > > > > > I am using intelliJ Idea IDE. I checked out the source from
>> GIT.
>> > > But
>> > > > I
>> > > > > am
>> > > > > > > seeing some compilation issues. Is there any documentation or
>> > notes
>> > > > on
>> > > > > > how
>> > > > > > > to set up the development envirnment?
>> > > > > > >
>> > > > > > > -
>> > > > > > > Ajay
>> > > > > > >
>> > > > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
>> > > > > amareshwari@gmail.com
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello Ajay,
>> > > > > > > >
>> > > > > > > > Nice to see your interest in exploring Lens.
>> > > > > > > >
>> > > > > > > > Looking at the requirement posted by you, seems you are only
>> > > > > interested
>> > > > > > > in
>> > > > > > > > REST api on top of cassandra. See the lens REST api for
>> query
>> > > > service
>> > > > > > > here
>> > > > > > > > -
>> > > > >
>> http://lens.incubator.apache.org/resource_QueryServiceResource.html
>>
>> > > > > > > >
>> > > > > > > > Lens provides REST api for query actions (submit, fetch
>> > results,
>> > > > > > cancel,
>> > > > > > > > getstatus), query history and OLAP data model.
>> > > > > > > >
>> > > > > > > > If you are not interested in OLAP data model, you can simple
>> > run
>> > > > > native
>> > > > > > > > queries on lens which would give REST api and history for
>> you
>> > as
>> > > > > well.
>> > > > > > > For
>> > > > > > > > more details on OLAP data model, you can read the doc on
>> > website.
>> > > > > > > >
>> > > > > > > > For new driver to be rewritten, it need not know anything
>> about
>> > > > OLAP
>> > > > > > data
>> > > > > > > > model. But all drivers are given HiveQL query to execute, if
>> > > driver
>> > > > > > > accepts
>> > > > > > > > Hive QL directly you dont have to do much on the query
>> itself.
>> > If
>> > > > > not,
>> > > > > > > the
>> > > > > > > > driver has to translate the query into something driver can
>> > > > > understand.
>> > > > > > > >
>> > > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can
>> have
>> > > > look
>> > > > > at
>> > > > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL,
>> > except
>> > > > it
>> > > > > > > might
>> > > > > > > > have to map some UDFs to SQL udfs.
>> > > > > > > >
>> > > > > > > > ok. Coming back to your questions, answers are inline.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <ajay.garga@gmail.com
>> >
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks.
>> > > > > > > > >
>> > > > > > > > > Couple of questions (that comes to my mind right now)
>> > > > > > > > >
>> > > > > > > > > 1) To create a Cluster instance to connect to a Cassandra
>> > > cluster
>> > > > > > > > > (Cassandra Java driver), we need to know the following:
>> > > > > > > > > a) ip address and port no of few instances of the cluster
>> > > > > > > > > b) authentication (if enabled)
>> > > > > > > > > c) few more configuration (loading balancing policy, ret
>> > policy
>> > > > and
>> > > > > > > > > compression etc..)
>> > > > > > > > >
>> > > > > > > > > Lens drivers are following xml based configuration for
>> > passing
>> > > > > > > > information. See
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
>> > > > > > > > and
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
>> > > > > > > > ,
>> > > > > > > > how driver specific configuration is put. Also read
>> > > > > > > > http://lens.incubator.apache.org/admin/config-server.html
>> to
>> > > > > > understand
>> > > > > > > > configuration in lens.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > Do we have any configuration support(yaml file or any
>> other)
>> > in
>> > > > > Lens.
>> > > > > > > If
>> > > > > > > > > yes, any pointers (java file names or WIKI)
>> > > > > > > > >
>> > > > > > > > > 2) Cassandra cluser/session and prepared statements are
>> multi
>> > > > > thread
>> > > > > > > safe
>> > > > > > > > > and recommended to be cached and reused.
>> > > > > > > > >
>> > > > > > > > > How does Lens support Caching (in memory or distributed)?.
>> > Any
>> > > > > > pointers
>> > > > > > > > > (java file names or WIKI)
>> > > > > > > > >
>> > > > > > > > > I did not understand the caching part here. But if you are
>> > > > talking
>> > > > > > > about
>> > > > > > > > cassandra sessions to be used across user queries, you can
>> look
>> > > at
>> > > > > how
>> > > > > > > hive
>> > > > > > > > sessions and connections are used in HiveDriver -
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <
>> > > yash360@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > That is great. We can probably pick the most recent Java
>> > > driver
>> > > > > and
>> > > > > > > > > > Cassandra version then. Since it addresses couple of old
>> > > > issues.
>> > > > > > > > > >
>> > > > > > > > > > Regarding Spark SQL for querying Cassandra I would let
>> > other
>> > > > > > > > contributors
>> > > > > > > > > > suggest.
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <
>> > ajay.garga@gmail.com
>>
>> > > >
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Thanks Yash.
>> > > > > > > > > > >
>> > > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra
>> > Java
>> > > > > driver
>> > > > > > > > [1].
>> > > > > > > > > > What
>> > > > > > > > > > > is the Cassandra and Java driver version we plan to
>> > > > support?. I
>> > > > > > > have
>> > > > > > > > > > worked
>> > > > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and
>> > > ALLOW
>> > > > > > > > FILTERING
>> > > > > > > > > > > worked.
>> > > > > > > > > > >
>> > > > > > > > > > > Secondly, as you mentioned I am aware of these
>> limitation
>> > > in
>> > > > > CQL.
>> > > > > > > But
>> > > > > > > > > > more
>> > > > > > > > > > > features recently added in Cassandra 2.2.x. Also,
>> other
>> > > > option
>> > > > > to
>> > > > > > > > work
>> > > > > > > > > > > around this to use Cassandra Spark connector and use
>> > Spark
>> > > > SQL.
>> > > > > > > > > > >
>> > > > > > > > > > > 1)
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
>> > > > > yash360@gmail.com
>>
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Just to add on...
>> > > > > > > > > > > >
>> > > > > > > > > > > > There will be few challenges to support entire SQL
>> on
>> > > > > Cassandra
>> > > > > > > > > > however.
>> > > > > > > > > > > > Last time I tried translating SQL to CQL I faced
>> couple
>> > > of
>> > > > > > > > scenarios
>> > > > > > > > > -
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1. Filtering non indexed column in Cassandra
>> > > > > > > > > > > > 2. Filtering by subset of primary key
>> > > > > > > > > > > > 3. OR condition in where clause
>> > > > > > > > > > > >
>> > > > > > > > > > > > You can probably start without these being a
>> blocker -
>> > In
>> > > > > > > parallel
>> > > > > > > > we
>> > > > > > > > > > can
>> > > > > > > > > > > > discuss how this can be implemented on our stack.
>> > Others
>> > > > can
>> > > > > > > > provide
>> > > > > > > > > > > their
>> > > > > > > > > > > > opinions here.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hope its helpful.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -----
>> > > > > > > > > > > > Examples:
>> > > > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
>> > > > pog_id)
>> > > > > > > where
>> > > > > > > > > > > > (id,rank) is primary key pair.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1.
>> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
>> > > > pog_id=10004 ;
>> > > > > > > > > > > > Bad Request: No indexed columns present in
>> by-columns
>> > > > clause
>> > > > > > with
>> > > > > > > > > Equal
>> > > > > > > > > > > > operator
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2.
>> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
>> rank=4;
>> > > > > > > > > > > > Bad Request: Cannot execute this query as it might
>> > > involve
>> > > > > data
>> > > > > > > > > > filtering
>> > > > > > > > > > > > and thus may have unpredictable performance. If you
>> > want
>> > > to
>> > > > > > > execute
>> > > > > > > > > > this
>> > > > > > > > > > > > query despite the performance unpredictability, use
>> > ALLOW
>> > > > > > > FILTERING
>> > > > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra
>> java
>> > > > > driver
>> > > > > > as
>> > > > > > > > of
>> > > > > > > > > > now.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 3.
>> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
>> rank=4
>> > or
>> > > > > > > > id='id0004';
>> > > > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
>> > > > > > > > > > > >
>> > > > > > > > > > > > 4. Valid Query:
>> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
>> > > id='id0004'
>> > > > > and
>> > > > > > > > > rank=4;
>> > > > > > > > > > > >
>> > > > > > > > > > > > id | rank | pog_id
>> > > > > > > > > > > > --------+------+--------
>> > > > > > > > > > > > id0004 | 4 | 10002
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
>> > > > > > yash360@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > The basic idea is to translate the Lens Query plan
>> > > (Which
>> > > > > is
>> > > > > > a
>> > > > > > > > Hive
>> > > > > > > > > > > AST)
>> > > > > > > > > > > > > to the DataStore specific Plan/Query.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The best example I can think of right now is the
>> > > Elastic
>> > > > > > Search
>> > > > > > > > > patch
>> > > > > > > > > > > by
>> > > > > > > > > > > > > Amruth. You can go through the JIRA [1] for the
>> > > detailed
>> > > > > > > > discussion
>> > > > > > > > > > and
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > Review Board [2] for code reference.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best regards
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
>> > > > > > > > > > > > > 2. https://reviews.apache.org/r/36434
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
>> > > > > ajay.garga@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >> Thanks Yash.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Is there any documentation or WIKI on the Lens
>> > > Driver. I
>> > > > > am
>> > > > > > > > going
>> > > > > > > > > > > > through
>> > > > > > > > > > > > >> the code as well.
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> Thanks
>> > > > > > > > > > > > >> Ajay
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
>> > > > > > > > yash360@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >> > Hi Ajay,
>> > > > > > > > > > > > >> > Welcome to the Lens Dev.
>> > > > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for
>> > Lens
>> > > > but
>> > > > > > no
>> > > > > > > > one
>> > > > > > > > > > has
>> > > > > > > > > > > > >> picked
>> > > > > > > > > > > > >> > it up yet. It would be great if you can pick up
>> > the
>> > > > task
>> > > > > > and
>> > > > > > > > > > submit
>> > > > > > > > > > > a
>> > > > > > > > > > > > >> patch
>> > > > > > > > > > > > >> > for review.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > Also drop a note on the list in case you
>> stumble
>> > > upon
>> > > > > any
>> > > > > > > > issue.
>> > > > > > > > > > > > Someone
>> > > > > > > > > > > > >> > will always be around to help you out.
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > 1.
>> https://issues.apache.org/jira/browse/LENS-654
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
>> > > > > > > ajay.garga@gmail.com>
>>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >> > > Hi,
>> > > > > > > > > > > > >> > >
>> > > > > > > > > > > > >> > > I recently noticed about Apache Lens project.
>> > > > > Currently
>> > > > > > we
>> > > > > > > > are
>> > > > > > > > > > > > >> building
>> > > > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary)
>> as
>> > > > there
>> > > > > no
>> > > > > > > > such
>> > > > > > > > > > > Apache
>> > > > > > > > > > > > >> open
>> > > > > > > > > > > > >> > > source project exists. Now as Lens supports
>> REST
>> > > > APIs
>> > > > > > for
>> > > > > > > > > Hadoop
>> > > > > > > > > > > and
>> > > > > > > > > > > > >> > JDBC,
>> > > > > > > > > > > > >> > > I want to know is there any plan to support
>> for
>> > > > Apache
>> > > > > > > > > Cassandra
>> > > > > > > > > > > as
>> > > > > > > > > > > > >> its
>> > > > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark
>> connector)
>> > > > which
>> > > > > > are
>> > > > > > > > > more
>> > > > > > > > > > > SQL
>> > > > > > > > > > > > >> like.
>> > > > > > > > > > > > >> > > If yes, I wish to know the details and
>> > contribute
>> > > as
>> > > > > > well.
>> > > > > > > > > > > > >> > >
>> > > > > > > > > > > > >> > > Thanks
>> > > > > > > > > > > > >> > > Ajay
>> > > > > > > > > > > > >> > >
>> > > > > > > > > > > > >> >
>> > > > > > > > > > > > >>
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
For example 1 such error in QueryExecutionServiceImp.java Lens Server
project
private LensResultSet getResultset(QueryHandle queryHandle) throws
LensException {
QueryContext ctx = allQueries.get(queryHandle);
if (ctx == null) {
return getResultsetFromDAO(queryHandle);
} else {
synchronized (ctx) {
if (ctx.isFinishedQueryPersisted()) {
return getResultsetFromDAO(queryHandle);
}
LensResultSet resultSet = resultSets.get(queryHandle);
if (resultSet == null) {
if (ctx.isPersistent() && ctx.getQueryOutputFormatter() != null) {
resultSets
.put(queryHandle,
new LensPersistentResult(
ctx.getQueryOutputFormatter().getMetadata(),
ctx.getQueryOutputFormatter().getFinalOutputPath(),
ctx.getQueryOutputFormatter().getNumRows()));
} else if (allQueries.get(queryHandle).isResultAvailableInDriver()) {
resultSet =
allQueries.get(queryHandle).getSelectedDriver().fetchResultSet(allQueries.get(queryHandle));
resultSets.put(queryHandle, resultSet);
} else {
throw new NotFoundException("Result set not available for
query:" + queryHandle);
}
}
}
if (resultSets.get(queryHandle) instanceof InMemoryResultSet) {
((InMemoryResultSet) resultSets.get(queryHandle)).seekToStart();
}
return resultSets.get(queryHandle);
}
}
There are no such methods in QueryContext() but annotated with @Getter
and @Setter. Do we need any plugin to solve it?
Thanks
Ajay
On Fri, Jul 24, 2015 at 2:33 PM, Rajat Khandelwal <ra...@gmail.com>
wrote:
> Yes, can you paste the exact error?
>
> On Fri, Jul 24, 2015 at 2:11 PM Yash Sharma <ya...@gmail.com> wrote:
>
> > Its certainly something wrt Env. Can you just copy paste couple of
> errors ?
> >
> > On Fri, Jul 24, 2015 at 2:00 PM, Ajay <aj...@gmail.com> wrote:
> >
> > > I did the same from command line.
> > >
> > > mvn clean install -DskipTests -DskipCheck
> > > mvn idea:idea
> > >
> > >
> > > Java version is 1.8
> > >
> > > I tried opening the project directly and selected
> > > /home/ajay/openSource/incubator-lens. Also I tried importing as maven
> > > project. Both the cases I see compilation errors.
> > >
> > > -
> > > Ajay
> > >
> > >
> > > On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <
> > rajatgupta59@gmail.com>
> > > wrote:
> > >
> > > > I'm using idea too. There are no compilation issues. Can you do
> > > >
> > > > mvn clean install -DskipTests -DskipCheck
> > > > mvn idea:idea
> > > >
> > > > from command line.
> > > >
> > > >
> > > > Also, check your java version. Lens is configured for java >=7
> > > >
> > > > On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
> > > >
> > > > > From console, the build goes thru fine.
> > > > >
> > > > > From Idea IDE, it is giving compilation errors (say one such file
> > > > > QueryExecutionServiceImp.java
> > > > > Lens Server project).
> > > > >
> > > > > Are you guys using Eclipse?
> > > > >
> > > > >
> > > > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Could you share the error. Might be env issues.
> > > > > > Also can you build from the console directly ?
> > > > > >
> > > > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > I am using intelliJ Idea IDE. I checked out the source from
> GIT.
> > > But
> > > > I
> > > > > am
> > > > > > > seeing some compilation issues. Is there any documentation or
> > notes
> > > > on
> > > > > > how
> > > > > > > to set up the development envirnment?
> > > > > > >
> > > > > > > -
> > > > > > > Ajay
> > > > > > >
> > > > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> > > > > amareshwari@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ajay,
> > > > > > > >
> > > > > > > > Nice to see your interest in exploring Lens.
> > > > > > > >
> > > > > > > > Looking at the requirement posted by you, seems you are only
> > > > > interested
> > > > > > > in
> > > > > > > > REST api on top of cassandra. See the lens REST api for query
> > > > service
> > > > > > > here
> > > > > > > > -
> > > > >
> http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > > > > > >
> > > > > > > > Lens provides REST api for query actions (submit, fetch
> > results,
> > > > > > cancel,
> > > > > > > > getstatus), query history and OLAP data model.
> > > > > > > >
> > > > > > > > If you are not interested in OLAP data model, you can simple
> > run
> > > > > native
> > > > > > > > queries on lens which would give REST api and history for you
> > as
> > > > > well.
> > > > > > > For
> > > > > > > > more details on OLAP data model, you can read the doc on
> > website.
> > > > > > > >
> > > > > > > > For new driver to be rewritten, it need not know anything
> about
> > > > OLAP
> > > > > > data
> > > > > > > > model. But all drivers are given HiveQL query to execute, if
> > > driver
> > > > > > > accepts
> > > > > > > > Hive QL directly you dont have to do much on the query
> itself.
> > If
> > > > > not,
> > > > > > > the
> > > > > > > > driver has to translate the query into something driver can
> > > > > understand.
> > > > > > > >
> > > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can
> have
> > > > look
> > > > > at
> > > > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL,
> > except
> > > > it
> > > > > > > might
> > > > > > > > have to map some UDFs to SQL udfs.
> > > > > > > >
> > > > > > > > ok. Coming back to your questions, answers are inline.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > Couple of questions (that comes to my mind right now)
> > > > > > > > >
> > > > > > > > > 1) To create a Cluster instance to connect to a Cassandra
> > > cluster
> > > > > > > > > (Cassandra Java driver), we need to know the following:
> > > > > > > > > a) ip address and port no of few instances of the cluster
> > > > > > > > > b) authentication (if enabled)
> > > > > > > > > c) few more configuration (loading balancing policy, ret
> > policy
> > > > and
> > > > > > > > > compression etc..)
> > > > > > > > >
> > > > > > > > > Lens drivers are following xml based configuration for
> > passing
> > > > > > > > information. See
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > > > > > and
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > > > > > ,
> > > > > > > > how driver specific configuration is put. Also read
> > > > > > > > http://lens.incubator.apache.org/admin/config-server.html to
> > > > > > understand
> > > > > > > > configuration in lens.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Do we have any configuration support(yaml file or any
> other)
> > in
> > > > > Lens.
> > > > > > > If
> > > > > > > > > yes, any pointers (java file names or WIKI)
> > > > > > > > >
> > > > > > > > > 2) Cassandra cluser/session and prepared statements are
> multi
> > > > > thread
> > > > > > > safe
> > > > > > > > > and recommended to be cached and reused.
> > > > > > > > >
> > > > > > > > > How does Lens support Caching (in memory or distributed)?.
> > Any
> > > > > > pointers
> > > > > > > > > (java file names or WIKI)
> > > > > > > > >
> > > > > > > > > I did not understand the caching part here. But if you are
> > > > talking
> > > > > > > about
> > > > > > > > cassandra sessions to be used across user queries, you can
> look
> > > at
> > > > > how
> > > > > > > hive
> > > > > > > > sessions and connections are used in HiveDriver -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <
> > > yash360@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > That is great. We can probably pick the most recent Java
> > > driver
> > > > > and
> > > > > > > > > > Cassandra version then. Since it addresses couple of old
> > > > issues.
> > > > > > > > > >
> > > > > > > > > > Regarding Spark SQL for querying Cassandra I would let
> > other
> > > > > > > > contributors
> > > > > > > > > > suggest.
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <
> > ajay.garga@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Yash.
> > > > > > > > > > >
> > > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra
> > Java
> > > > > driver
> > > > > > > > [1].
> > > > > > > > > > What
> > > > > > > > > > > is the Cassandra and Java driver version we plan to
> > > > support?. I
> > > > > > > have
> > > > > > > > > > worked
> > > > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and
> > > ALLOW
> > > > > > > > FILTERING
> > > > > > > > > > > worked.
> > > > > > > > > > >
> > > > > > > > > > > Secondly, as you mentioned I am aware of these
> limitation
> > > in
> > > > > CQL.
> > > > > > > But
> > > > > > > > > > more
> > > > > > > > > > > features recently added in Cassandra 2.2.x. Also, other
> > > > option
> > > > > to
> > > > > > > > work
> > > > > > > > > > > around this to use Cassandra Spark connector and use
> > Spark
> > > > SQL.
> > > > > > > > > > >
> > > > > > > > > > > 1)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> > > > > yash360@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Just to add on...
> > > > > > > > > > > >
> > > > > > > > > > > > There will be few challenges to support entire SQL on
> > > > > Cassandra
> > > > > > > > > > however.
> > > > > > > > > > > > Last time I tried translating SQL to CQL I faced
> couple
> > > of
> > > > > > > > scenarios
> > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > > > > > 3. OR condition in where clause
> > > > > > > > > > > >
> > > > > > > > > > > > You can probably start without these being a blocker
> -
> > In
> > > > > > > parallel
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > discuss how this can be implemented on our stack.
> > Others
> > > > can
> > > > > > > > provide
> > > > > > > > > > > their
> > > > > > > > > > > > opinions here.
> > > > > > > > > > > >
> > > > > > > > > > > > Hope its helpful.
> > > > > > > > > > > >
> > > > > > > > > > > > -----
> > > > > > > > > > > > Examples:
> > > > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
> > > > pog_id)
> > > > > > > where
> > > > > > > > > > > > (id,rank) is primary key pair.
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > > > pog_id=10004 ;
> > > > > > > > > > > > Bad Request: No indexed columns present in by-columns
> > > > clause
> > > > > > with
> > > > > > > > > Equal
> > > > > > > > > > > > operator
> > > > > > > > > > > >
> > > > > > > > > > > > 2.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> rank=4;
> > > > > > > > > > > > Bad Request: Cannot execute this query as it might
> > > involve
> > > > > data
> > > > > > > > > > filtering
> > > > > > > > > > > > and thus may have unpredictable performance. If you
> > want
> > > to
> > > > > > > execute
> > > > > > > > > > this
> > > > > > > > > > > > query despite the performance unpredictability, use
> > ALLOW
> > > > > > > FILTERING
> > > > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra
> java
> > > > > driver
> > > > > > as
> > > > > > > > of
> > > > > > > > > > now.
> > > > > > > > > > > >
> > > > > > > > > > > > 3.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4
> > or
> > > > > > > > id='id0004';
> > > > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > > > > > >
> > > > > > > > > > > > 4. Valid Query:
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > > id='id0004'
> > > > > and
> > > > > > > > > rank=4;
> > > > > > > > > > > >
> > > > > > > > > > > > id | rank | pog_id
> > > > > > > > > > > > --------+------+--------
> > > > > > > > > > > > id0004 | 4 | 10002
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > > > > > yash360@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The basic idea is to translate the Lens Query plan
> > > (Which
> > > > > is
> > > > > > a
> > > > > > > > Hive
> > > > > > > > > > > AST)
> > > > > > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The best example I can think of right now is the
> > > Elastic
> > > > > > Search
> > > > > > > > > patch
> > > > > > > > > > > by
> > > > > > > > > > > > > Amruth. You can go through the JIRA [1] for the
> > > detailed
> > > > > > > > discussion
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > Review Board [2] for code reference.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> > > > > ajay.garga@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Thanks Yash.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Is there any documentation or WIKI on the Lens
> > > Driver. I
> > > > > am
> > > > > > > > going
> > > > > > > > > > > > through
> > > > > > > > > > > > >> the code as well.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > >> Ajay
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > > > > > yash360@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Ajay,
> > > > > > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for
> > Lens
> > > > but
> > > > > > no
> > > > > > > > one
> > > > > > > > > > has
> > > > > > > > > > > > >> picked
> > > > > > > > > > > > >> > it up yet. It would be great if you can pick up
> > the
> > > > task
> > > > > > and
> > > > > > > > > > submit
> > > > > > > > > > > a
> > > > > > > > > > > > >> patch
> > > > > > > > > > > > >> > for review.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Also drop a note on the list in case you stumble
> > > upon
> > > > > any
> > > > > > > > issue.
> > > > > > > > > > > > Someone
> > > > > > > > > > > > >> > will always be around to help you out.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 1.
> https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > > > > > ajay.garga@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Hi,
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > I recently noticed about Apache Lens project.
> > > > > Currently
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > >> building
> > > > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary)
> as
> > > > there
> > > > > no
> > > > > > > > such
> > > > > > > > > > > Apache
> > > > > > > > > > > > >> open
> > > > > > > > > > > > >> > > source project exists. Now as Lens supports
> REST
> > > > APIs
> > > > > > for
> > > > > > > > > Hadoop
> > > > > > > > > > > and
> > > > > > > > > > > > >> > JDBC,
> > > > > > > > > > > > >> > > I want to know is there any plan to support
> for
> > > > Apache
> > > > > > > > > Cassandra
> > > > > > > > > > > as
> > > > > > > > > > > > >> its
> > > > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark
> connector)
> > > > which
> > > > > > are
> > > > > > > > > more
> > > > > > > > > > > SQL
> > > > > > > > > > > > >> like.
> > > > > > > > > > > > >> > > If yes, I wish to know the details and
> > contribute
> > > as
> > > > > > well.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Thanks
> > > > > > > > > > > > >> > > Ajay
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Rajat Khandelwal <ra...@gmail.com>.
Yes, can you paste the exact error?
On Fri, Jul 24, 2015 at 2:11 PM Yash Sharma <ya...@gmail.com> wrote:
> Its certainly something wrt Env. Can you just copy paste couple of errors ?
>
> On Fri, Jul 24, 2015 at 2:00 PM, Ajay <aj...@gmail.com> wrote:
>
> > I did the same from command line.
> >
> > mvn clean install -DskipTests -DskipCheck
> > mvn idea:idea
> >
> >
> > Java version is 1.8
> >
> > I tried opening the project directly and selected
> > /home/ajay/openSource/incubator-lens. Also I tried importing as maven
> > project. Both the cases I see compilation errors.
> >
> > -
> > Ajay
> >
> >
> > On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <
> rajatgupta59@gmail.com>
> > wrote:
> >
> > > I'm using idea too. There are no compilation issues. Can you do
> > >
> > > mvn clean install -DskipTests -DskipCheck
> > > mvn idea:idea
> > >
> > > from command line.
> > >
> > >
> > > Also, check your java version. Lens is configured for java >=7
> > >
> > > On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
> > >
> > > > From console, the build goes thru fine.
> > > >
> > > > From Idea IDE, it is giving compilation errors (say one such file
> > > > QueryExecutionServiceImp.java
> > > > Lens Server project).
> > > >
> > > > Are you guys using Eclipse?
> > > >
> > > >
> > > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com>
> > wrote:
> > > >
> > > > > Could you share the error. Might be env issues.
> > > > > Also can you build from the console directly ?
> > > > >
> > > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com>
> wrote:
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > I am using intelliJ Idea IDE. I checked out the source from GIT.
> > But
> > > I
> > > > am
> > > > > > seeing some compilation issues. Is there any documentation or
> notes
> > > on
> > > > > how
> > > > > > to set up the development envirnment?
> > > > > >
> > > > > > -
> > > > > > Ajay
> > > > > >
> > > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> > > > amareshwari@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Ajay,
> > > > > > >
> > > > > > > Nice to see your interest in exploring Lens.
> > > > > > >
> > > > > > > Looking at the requirement posted by you, seems you are only
> > > > interested
> > > > > > in
> > > > > > > REST api on top of cassandra. See the lens REST api for query
> > > service
> > > > > > here
> > > > > > > -
> > > > http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > > > > >
> > > > > > > Lens provides REST api for query actions (submit, fetch
> results,
> > > > > cancel,
> > > > > > > getstatus), query history and OLAP data model.
> > > > > > >
> > > > > > > If you are not interested in OLAP data model, you can simple
> run
> > > > native
> > > > > > > queries on lens which would give REST api and history for you
> as
> > > > well.
> > > > > > For
> > > > > > > more details on OLAP data model, you can read the doc on
> website.
> > > > > > >
> > > > > > > For new driver to be rewritten, it need not know anything about
> > > OLAP
> > > > > data
> > > > > > > model. But all drivers are given HiveQL query to execute, if
> > driver
> > > > > > accepts
> > > > > > > Hive QL directly you dont have to do much on the query itself.
> If
> > > > not,
> > > > > > the
> > > > > > > driver has to translate the query into something driver can
> > > > understand.
> > > > > > >
> > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can have
> > > look
> > > > at
> > > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL,
> except
> > > it
> > > > > > might
> > > > > > > have to map some UDFs to SQL udfs.
> > > > > > >
> > > > > > > ok. Coming back to your questions, answers are inline.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > > Couple of questions (that comes to my mind right now)
> > > > > > > >
> > > > > > > > 1) To create a Cluster instance to connect to a Cassandra
> > cluster
> > > > > > > > (Cassandra Java driver), we need to know the following:
> > > > > > > > a) ip address and port no of few instances of the cluster
> > > > > > > > b) authentication (if enabled)
> > > > > > > > c) few more configuration (loading balancing policy, ret
> policy
> > > and
> > > > > > > > compression etc..)
> > > > > > > >
> > > > > > > > Lens drivers are following xml based configuration for
> passing
> > > > > > > information. See
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > > > > and
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > > > > ,
> > > > > > > how driver specific configuration is put. Also read
> > > > > > > http://lens.incubator.apache.org/admin/config-server.html to
> > > > > understand
> > > > > > > configuration in lens.
> > > > > > >
> > > > > > >
> > > > > > > > Do we have any configuration support(yaml file or any other)
> in
> > > > Lens.
> > > > > > If
> > > > > > > > yes, any pointers (java file names or WIKI)
> > > > > > > >
> > > > > > > > 2) Cassandra cluser/session and prepared statements are multi
> > > > thread
> > > > > > safe
> > > > > > > > and recommended to be cached and reused.
> > > > > > > >
> > > > > > > > How does Lens support Caching (in memory or distributed)?.
> Any
> > > > > pointers
> > > > > > > > (java file names or WIKI)
> > > > > > > >
> > > > > > > > I did not understand the caching part here. But if you are
> > > talking
> > > > > > about
> > > > > > > cassandra sessions to be used across user queries, you can look
> > at
> > > > how
> > > > > > hive
> > > > > > > sessions and connections are used in HiveDriver -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <
> > yash360@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > That is great. We can probably pick the most recent Java
> > driver
> > > > and
> > > > > > > > > Cassandra version then. Since it addresses couple of old
> > > issues.
> > > > > > > > >
> > > > > > > > > Regarding Spark SQL for querying Cassandra I would let
> other
> > > > > > > contributors
> > > > > > > > > suggest.
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <
> ajay.garga@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Yash.
> > > > > > > > > >
> > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra
> Java
> > > > driver
> > > > > > > [1].
> > > > > > > > > What
> > > > > > > > > > is the Cassandra and Java driver version we plan to
> > > support?. I
> > > > > > have
> > > > > > > > > worked
> > > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and
> > ALLOW
> > > > > > > FILTERING
> > > > > > > > > > worked.
> > > > > > > > > >
> > > > > > > > > > Secondly, as you mentioned I am aware of these limitation
> > in
> > > > CQL.
> > > > > > But
> > > > > > > > > more
> > > > > > > > > > features recently added in Cassandra 2.2.x. Also, other
> > > option
> > > > to
> > > > > > > work
> > > > > > > > > > around this to use Cassandra Spark connector and use
> Spark
> > > SQL.
> > > > > > > > > >
> > > > > > > > > > 1)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> > > > yash360@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Just to add on...
> > > > > > > > > > >
> > > > > > > > > > > There will be few challenges to support entire SQL on
> > > > Cassandra
> > > > > > > > > however.
> > > > > > > > > > > Last time I tried translating SQL to CQL I faced couple
> > of
> > > > > > > scenarios
> > > > > > > > -
> > > > > > > > > > >
> > > > > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > > > > 3. OR condition in where clause
> > > > > > > > > > >
> > > > > > > > > > > You can probably start without these being a blocker -
> In
> > > > > > parallel
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > discuss how this can be implemented on our stack.
> Others
> > > can
> > > > > > > provide
> > > > > > > > > > their
> > > > > > > > > > > opinions here.
> > > > > > > > > > >
> > > > > > > > > > > Hope its helpful.
> > > > > > > > > > >
> > > > > > > > > > > -----
> > > > > > > > > > > Examples:
> > > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
> > > pog_id)
> > > > > > where
> > > > > > > > > > > (id,rank) is primary key pair.
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > > pog_id=10004 ;
> > > > > > > > > > > Bad Request: No indexed columns present in by-columns
> > > clause
> > > > > with
> > > > > > > > Equal
> > > > > > > > > > > operator
> > > > > > > > > > >
> > > > > > > > > > > 2.
> > > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > > > > > Bad Request: Cannot execute this query as it might
> > involve
> > > > data
> > > > > > > > > filtering
> > > > > > > > > > > and thus may have unpredictable performance. If you
> want
> > to
> > > > > > execute
> > > > > > > > > this
> > > > > > > > > > > query despite the performance unpredictability, use
> ALLOW
> > > > > > FILTERING
> > > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java
> > > > driver
> > > > > as
> > > > > > > of
> > > > > > > > > now.
> > > > > > > > > > >
> > > > > > > > > > > 3.
> > > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4
> or
> > > > > > > id='id0004';
> > > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > > > > >
> > > > > > > > > > > 4. Valid Query:
> > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > id='id0004'
> > > > and
> > > > > > > > rank=4;
> > > > > > > > > > >
> > > > > > > > > > > id | rank | pog_id
> > > > > > > > > > > --------+------+--------
> > > > > > > > > > > id0004 | 4 | 10002
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > > > > yash360@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The basic idea is to translate the Lens Query plan
> > (Which
> > > > is
> > > > > a
> > > > > > > Hive
> > > > > > > > > > AST)
> > > > > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > > > > >
> > > > > > > > > > > > The best example I can think of right now is the
> > Elastic
> > > > > Search
> > > > > > > > patch
> > > > > > > > > > by
> > > > > > > > > > > > Amruth. You can go through the JIRA [1] for the
> > detailed
> > > > > > > discussion
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > Review Board [2] for code reference.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > >
> > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> > > > ajay.garga@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Thanks Yash.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Is there any documentation or WIKI on the Lens
> > Driver. I
> > > > am
> > > > > > > going
> > > > > > > > > > > through
> > > > > > > > > > > >> the code as well.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks
> > > > > > > > > > > >> Ajay
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > > > > yash360@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Ajay,
> > > > > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for
> Lens
> > > but
> > > > > no
> > > > > > > one
> > > > > > > > > has
> > > > > > > > > > > >> picked
> > > > > > > > > > > >> > it up yet. It would be great if you can pick up
> the
> > > task
> > > > > and
> > > > > > > > > submit
> > > > > > > > > > a
> > > > > > > > > > > >> patch
> > > > > > > > > > > >> > for review.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Also drop a note on the list in case you stumble
> > upon
> > > > any
> > > > > > > issue.
> > > > > > > > > > > Someone
> > > > > > > > > > > >> > will always be around to help you out.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > > > > ajay.garga@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Hi,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > I recently noticed about Apache Lens project.
> > > > Currently
> > > > > we
> > > > > > > are
> > > > > > > > > > > >> building
> > > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as
> > > there
> > > > no
> > > > > > > such
> > > > > > > > > > Apache
> > > > > > > > > > > >> open
> > > > > > > > > > > >> > > source project exists. Now as Lens supports REST
> > > APIs
> > > > > for
> > > > > > > > Hadoop
> > > > > > > > > > and
> > > > > > > > > > > >> > JDBC,
> > > > > > > > > > > >> > > I want to know is there any plan to support for
> > > Apache
> > > > > > > > Cassandra
> > > > > > > > > > as
> > > > > > > > > > > >> its
> > > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector)
> > > which
> > > > > are
> > > > > > > > more
> > > > > > > > > > SQL
> > > > > > > > > > > >> like.
> > > > > > > > > > > >> > > If yes, I wish to know the details and
> contribute
> > as
> > > > > well.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Thanks
> > > > > > > > > > > >> > > Ajay
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
Its certainly something wrt Env. Can you just copy paste couple of errors ?
On Fri, Jul 24, 2015 at 2:00 PM, Ajay <aj...@gmail.com> wrote:
> I did the same from command line.
>
> mvn clean install -DskipTests -DskipCheck
> mvn idea:idea
>
>
> Java version is 1.8
>
> I tried opening the project directly and selected
> /home/ajay/openSource/incubator-lens. Also I tried importing as maven
> project. Both the cases I see compilation errors.
>
> -
> Ajay
>
>
> On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <ra...@gmail.com>
> wrote:
>
> > I'm using idea too. There are no compilation issues. Can you do
> >
> > mvn clean install -DskipTests -DskipCheck
> > mvn idea:idea
> >
> > from command line.
> >
> >
> > Also, check your java version. Lens is configured for java >=7
> >
> > On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
> >
> > > From console, the build goes thru fine.
> > >
> > > From Idea IDE, it is giving compilation errors (say one such file
> > > QueryExecutionServiceImp.java
> > > Lens Server project).
> > >
> > > Are you guys using Eclipse?
> > >
> > >
> > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com>
> wrote:
> > >
> > > > Could you share the error. Might be env issues.
> > > > Also can you build from the console directly ?
> > > >
> > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
> > > >
> > > > > Thanks.
> > > > >
> > > > > I am using intelliJ Idea IDE. I checked out the source from GIT.
> But
> > I
> > > am
> > > > > seeing some compilation issues. Is there any documentation or notes
> > on
> > > > how
> > > > > to set up the development envirnment?
> > > > >
> > > > > -
> > > > > Ajay
> > > > >
> > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> > > amareshwari@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Ajay,
> > > > > >
> > > > > > Nice to see your interest in exploring Lens.
> > > > > >
> > > > > > Looking at the requirement posted by you, seems you are only
> > > interested
> > > > > in
> > > > > > REST api on top of cassandra. See the lens REST api for query
> > service
> > > > > here
> > > > > > -
> > > http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > > > >
> > > > > > Lens provides REST api for query actions (submit, fetch results,
> > > > cancel,
> > > > > > getstatus), query history and OLAP data model.
> > > > > >
> > > > > > If you are not interested in OLAP data model, you can simple run
> > > native
> > > > > > queries on lens which would give REST api and history for you as
> > > well.
> > > > > For
> > > > > > more details on OLAP data model, you can read the doc on website.
> > > > > >
> > > > > > For new driver to be rewritten, it need not know anything about
> > OLAP
> > > > data
> > > > > > model. But all drivers are given HiveQL query to execute, if
> driver
> > > > > accepts
> > > > > > Hive QL directly you dont have to do much on the query itself. If
> > > not,
> > > > > the
> > > > > > driver has to translate the query into something driver can
> > > understand.
> > > > > >
> > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can have
> > look
> > > at
> > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except
> > it
> > > > > might
> > > > > > have to map some UDFs to SQL udfs.
> > > > > >
> > > > > > ok. Coming back to your questions, answers are inline.
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Couple of questions (that comes to my mind right now)
> > > > > > >
> > > > > > > 1) To create a Cluster instance to connect to a Cassandra
> cluster
> > > > > > > (Cassandra Java driver), we need to know the following:
> > > > > > > a) ip address and port no of few instances of the cluster
> > > > > > > b) authentication (if enabled)
> > > > > > > c) few more configuration (loading balancing policy, ret policy
> > and
> > > > > > > compression etc..)
> > > > > > >
> > > > > > > Lens drivers are following xml based configuration for passing
> > > > > > information. See
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > > > and
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > > > ,
> > > > > > how driver specific configuration is put. Also read
> > > > > > http://lens.incubator.apache.org/admin/config-server.html to
> > > > understand
> > > > > > configuration in lens.
> > > > > >
> > > > > >
> > > > > > > Do we have any configuration support(yaml file or any other) in
> > > Lens.
> > > > > If
> > > > > > > yes, any pointers (java file names or WIKI)
> > > > > > >
> > > > > > > 2) Cassandra cluser/session and prepared statements are multi
> > > thread
> > > > > safe
> > > > > > > and recommended to be cached and reused.
> > > > > > >
> > > > > > > How does Lens support Caching (in memory or distributed)?. Any
> > > > pointers
> > > > > > > (java file names or WIKI)
> > > > > > >
> > > > > > > I did not understand the caching part here. But if you are
> > talking
> > > > > about
> > > > > > cassandra sessions to be used across user queries, you can look
> at
> > > how
> > > > > hive
> > > > > > sessions and connections are used in HiveDriver -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <
> yash360@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > That is great. We can probably pick the most recent Java
> driver
> > > and
> > > > > > > > Cassandra version then. Since it addresses couple of old
> > issues.
> > > > > > > >
> > > > > > > > Regarding Spark SQL for querying Cassandra I would let other
> > > > > > contributors
> > > > > > > > suggest.
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <ajay.garga@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Yash.
> > > > > > > > >
> > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java
> > > driver
> > > > > > [1].
> > > > > > > > What
> > > > > > > > > is the Cassandra and Java driver version we plan to
> > support?. I
> > > > > have
> > > > > > > > worked
> > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and
> ALLOW
> > > > > > FILTERING
> > > > > > > > > worked.
> > > > > > > > >
> > > > > > > > > Secondly, as you mentioned I am aware of these limitation
> in
> > > CQL.
> > > > > But
> > > > > > > > more
> > > > > > > > > features recently added in Cassandra 2.2.x. Also, other
> > option
> > > to
> > > > > > work
> > > > > > > > > around this to use Cassandra Spark connector and use Spark
> > SQL.
> > > > > > > > >
> > > > > > > > > 1)
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> > > yash360@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Just to add on...
> > > > > > > > > >
> > > > > > > > > > There will be few challenges to support entire SQL on
> > > Cassandra
> > > > > > > > however.
> > > > > > > > > > Last time I tried translating SQL to CQL I faced couple
> of
> > > > > > scenarios
> > > > > > > -
> > > > > > > > > >
> > > > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > > > 3. OR condition in where clause
> > > > > > > > > >
> > > > > > > > > > You can probably start without these being a blocker - In
> > > > > parallel
> > > > > > we
> > > > > > > > can
> > > > > > > > > > discuss how this can be implemented on our stack. Others
> > can
> > > > > > provide
> > > > > > > > > their
> > > > > > > > > > opinions here.
> > > > > > > > > >
> > > > > > > > > > Hope its helpful.
> > > > > > > > > >
> > > > > > > > > > -----
> > > > > > > > > > Examples:
> > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
> > pog_id)
> > > > > where
> > > > > > > > > > (id,rank) is primary key pair.
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > pog_id=10004 ;
> > > > > > > > > > Bad Request: No indexed columns present in by-columns
> > clause
> > > > with
> > > > > > > Equal
> > > > > > > > > > operator
> > > > > > > > > >
> > > > > > > > > > 2.
> > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > > > > Bad Request: Cannot execute this query as it might
> involve
> > > data
> > > > > > > > filtering
> > > > > > > > > > and thus may have unpredictable performance. If you want
> to
> > > > > execute
> > > > > > > > this
> > > > > > > > > > query despite the performance unpredictability, use ALLOW
> > > > > FILTERING
> > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java
> > > driver
> > > > as
> > > > > > of
> > > > > > > > now.
> > > > > > > > > >
> > > > > > > > > > 3.
> > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > > > > > id='id0004';
> > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > > > >
> > > > > > > > > > 4. Valid Query:
> > > > > > > > > > cqlsh:recsys> select * from trending_now where
> id='id0004'
> > > and
> > > > > > > rank=4;
> > > > > > > > > >
> > > > > > > > > > id | rank | pog_id
> > > > > > > > > > --------+------+--------
> > > > > > > > > > id0004 | 4 | 10002
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > > > yash360@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The basic idea is to translate the Lens Query plan
> (Which
> > > is
> > > > a
> > > > > > Hive
> > > > > > > > > AST)
> > > > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > > > >
> > > > > > > > > > > The best example I can think of right now is the
> Elastic
> > > > Search
> > > > > > > patch
> > > > > > > > > by
> > > > > > > > > > > Amruth. You can go through the JIRA [1] for the
> detailed
> > > > > > discussion
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > Review Board [2] for code reference.
> > > > > > > > > > >
> > > > > > > > > > > Best regards
> > > > > > > > > > >
> > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> > > ajay.garga@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Thanks Yash.
> > > > > > > > > > >>
> > > > > > > > > > >> Is there any documentation or WIKI on the Lens
> Driver. I
> > > am
> > > > > > going
> > > > > > > > > > through
> > > > > > > > > > >> the code as well.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks
> > > > > > > > > > >> Ajay
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > > > yash360@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Ajay,
> > > > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens
> > but
> > > > no
> > > > > > one
> > > > > > > > has
> > > > > > > > > > >> picked
> > > > > > > > > > >> > it up yet. It would be great if you can pick up the
> > task
> > > > and
> > > > > > > > submit
> > > > > > > > > a
> > > > > > > > > > >> patch
> > > > > > > > > > >> > for review.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Also drop a note on the list in case you stumble
> upon
> > > any
> > > > > > issue.
> > > > > > > > > > Someone
> > > > > > > > > > >> > will always be around to help you out.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > > > ajay.garga@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > I recently noticed about Apache Lens project.
> > > Currently
> > > > we
> > > > > > are
> > > > > > > > > > >> building
> > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as
> > there
> > > no
> > > > > > such
> > > > > > > > > Apache
> > > > > > > > > > >> open
> > > > > > > > > > >> > > source project exists. Now as Lens supports REST
> > APIs
> > > > for
> > > > > > > Hadoop
> > > > > > > > > and
> > > > > > > > > > >> > JDBC,
> > > > > > > > > > >> > > I want to know is there any plan to support for
> > Apache
> > > > > > > Cassandra
> > > > > > > > > as
> > > > > > > > > > >> its
> > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector)
> > which
> > > > are
> > > > > > > more
> > > > > > > > > SQL
> > > > > > > > > > >> like.
> > > > > > > > > > >> > > If yes, I wish to know the details and contribute
> as
> > > > well.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks
> > > > > > > > > > >> > > Ajay
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
I did the same from command line.
mvn clean install -DskipTests -DskipCheck
mvn idea:idea
Java version is 1.8
I tried opening the project directly and selected
/home/ajay/openSource/incubator-lens. Also I tried importing as maven
project. Both the cases I see compilation errors.
-
Ajay
On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <ra...@gmail.com>
wrote:
> I'm using idea too. There are no compilation issues. Can you do
>
> mvn clean install -DskipTests -DskipCheck
> mvn idea:idea
>
> from command line.
>
>
> Also, check your java version. Lens is configured for java >=7
>
> On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
>
> > From console, the build goes thru fine.
> >
> > From Idea IDE, it is giving compilation errors (say one such file
> > QueryExecutionServiceImp.java
> > Lens Server project).
> >
> > Are you guys using Eclipse?
> >
> >
> > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com> wrote:
> >
> > > Could you share the error. Might be env issues.
> > > Also can you build from the console directly ?
> > >
> > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
> > >
> > > > Thanks.
> > > >
> > > > I am using intelliJ Idea IDE. I checked out the source from GIT. But
> I
> > am
> > > > seeing some compilation issues. Is there any documentation or notes
> on
> > > how
> > > > to set up the development envirnment?
> > > >
> > > > -
> > > > Ajay
> > > >
> > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> > amareshwari@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hello Ajay,
> > > > >
> > > > > Nice to see your interest in exploring Lens.
> > > > >
> > > > > Looking at the requirement posted by you, seems you are only
> > interested
> > > > in
> > > > > REST api on top of cassandra. See the lens REST api for query
> service
> > > > here
> > > > > -
> > http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > > >
> > > > > Lens provides REST api for query actions (submit, fetch results,
> > > cancel,
> > > > > getstatus), query history and OLAP data model.
> > > > >
> > > > > If you are not interested in OLAP data model, you can simple run
> > native
> > > > > queries on lens which would give REST api and history for you as
> > well.
> > > > For
> > > > > more details on OLAP data model, you can read the doc on website.
> > > > >
> > > > > For new driver to be rewritten, it need not know anything about
> OLAP
> > > data
> > > > > model. But all drivers are given HiveQL query to execute, if driver
> > > > accepts
> > > > > Hive QL directly you dont have to do much on the query itself. If
> > not,
> > > > the
> > > > > driver has to translate the query into something driver can
> > understand.
> > > > >
> > > > > For example : JDBCDriver converts HiveQL to SQL. - You can have
> look
> > at
> > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except
> it
> > > > might
> > > > > have to map some UDFs to SQL udfs.
> > > > >
> > > > > ok. Coming back to your questions, answers are inline.
> > > > >
> > > > >
> > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com>
> wrote:
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Couple of questions (that comes to my mind right now)
> > > > > >
> > > > > > 1) To create a Cluster instance to connect to a Cassandra cluster
> > > > > > (Cassandra Java driver), we need to know the following:
> > > > > > a) ip address and port no of few instances of the cluster
> > > > > > b) authentication (if enabled)
> > > > > > c) few more configuration (loading balancing policy, ret policy
> and
> > > > > > compression etc..)
> > > > > >
> > > > > > Lens drivers are following xml based configuration for passing
> > > > > information. See
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > > and
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > > ,
> > > > > how driver specific configuration is put. Also read
> > > > > http://lens.incubator.apache.org/admin/config-server.html to
> > > understand
> > > > > configuration in lens.
> > > > >
> > > > >
> > > > > > Do we have any configuration support(yaml file or any other) in
> > Lens.
> > > > If
> > > > > > yes, any pointers (java file names or WIKI)
> > > > > >
> > > > > > 2) Cassandra cluser/session and prepared statements are multi
> > thread
> > > > safe
> > > > > > and recommended to be cached and reused.
> > > > > >
> > > > > > How does Lens support Caching (in memory or distributed)?. Any
> > > pointers
> > > > > > (java file names or WIKI)
> > > > > >
> > > > > > I did not understand the caching part here. But if you are
> talking
> > > > about
> > > > > cassandra sessions to be used across user queries, you can look at
> > how
> > > > hive
> > > > > sessions and connections are used in HiveDriver -
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <yash360@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > That is great. We can probably pick the most recent Java driver
> > and
> > > > > > > Cassandra version then. Since it addresses couple of old
> issues.
> > > > > > >
> > > > > > > Regarding Spark SQL for querying Cassandra I would let other
> > > > > contributors
> > > > > > > suggest.
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks Yash.
> > > > > > > >
> > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java
> > driver
> > > > > [1].
> > > > > > > What
> > > > > > > > is the Cassandra and Java driver version we plan to
> support?. I
> > > > have
> > > > > > > worked
> > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> > > > > FILTERING
> > > > > > > > worked.
> > > > > > > >
> > > > > > > > Secondly, as you mentioned I am aware of these limitation in
> > CQL.
> > > > But
> > > > > > > more
> > > > > > > > features recently added in Cassandra 2.2.x. Also, other
> option
> > to
> > > > > work
> > > > > > > > around this to use Cassandra Spark connector and use Spark
> SQL.
> > > > > > > >
> > > > > > > > 1)
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> > yash360@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Just to add on...
> > > > > > > > >
> > > > > > > > > There will be few challenges to support entire SQL on
> > Cassandra
> > > > > > > however.
> > > > > > > > > Last time I tried translating SQL to CQL I faced couple of
> > > > > scenarios
> > > > > > -
> > > > > > > > >
> > > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > > 3. OR condition in where clause
> > > > > > > > >
> > > > > > > > > You can probably start without these being a blocker - In
> > > > parallel
> > > > > we
> > > > > > > can
> > > > > > > > > discuss how this can be implemented on our stack. Others
> can
> > > > > provide
> > > > > > > > their
> > > > > > > > > opinions here.
> > > > > > > > >
> > > > > > > > > Hope its helpful.
> > > > > > > > >
> > > > > > > > > -----
> > > > > > > > > Examples:
> > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
> pog_id)
> > > > where
> > > > > > > > > (id,rank) is primary key pair.
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > > cqlsh:recsys> select * from trending_now where
> pog_id=10004 ;
> > > > > > > > > Bad Request: No indexed columns present in by-columns
> clause
> > > with
> > > > > > Equal
> > > > > > > > > operator
> > > > > > > > >
> > > > > > > > > 2.
> > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > > > Bad Request: Cannot execute this query as it might involve
> > data
> > > > > > > filtering
> > > > > > > > > and thus may have unpredictable performance. If you want to
> > > > execute
> > > > > > > this
> > > > > > > > > query despite the performance unpredictability, use ALLOW
> > > > FILTERING
> > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java
> > driver
> > > as
> > > > > of
> > > > > > > now.
> > > > > > > > >
> > > > > > > > > 3.
> > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > > > > id='id0004';
> > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > > >
> > > > > > > > > 4. Valid Query:
> > > > > > > > > cqlsh:recsys> select * from trending_now where id='id0004'
> > and
> > > > > > rank=4;
> > > > > > > > >
> > > > > > > > > id | rank | pog_id
> > > > > > > > > --------+------+--------
> > > > > > > > > id0004 | 4 | 10002
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > > yash360@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The basic idea is to translate the Lens Query plan (Which
> > is
> > > a
> > > > > Hive
> > > > > > > > AST)
> > > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > > >
> > > > > > > > > > The best example I can think of right now is the Elastic
> > > Search
> > > > > > patch
> > > > > > > > by
> > > > > > > > > > Amruth. You can go through the JIRA [1] for the detailed
> > > > > discussion
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > Review Board [2] for code reference.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > >
> > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> > ajay.garga@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Thanks Yash.
> > > > > > > > > >>
> > > > > > > > > >> Is there any documentation or WIKI on the Lens Driver. I
> > am
> > > > > going
> > > > > > > > > through
> > > > > > > > > >> the code as well.
> > > > > > > > > >>
> > > > > > > > > >> Thanks
> > > > > > > > > >> Ajay
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > > yash360@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Ajay,
> > > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens
> but
> > > no
> > > > > one
> > > > > > > has
> > > > > > > > > >> picked
> > > > > > > > > >> > it up yet. It would be great if you can pick up the
> task
> > > and
> > > > > > > submit
> > > > > > > > a
> > > > > > > > > >> patch
> > > > > > > > > >> > for review.
> > > > > > > > > >> >
> > > > > > > > > >> > Also drop a note on the list in case you stumble upon
> > any
> > > > > issue.
> > > > > > > > > Someone
> > > > > > > > > >> > will always be around to help you out.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > > >> >
> > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > > ajay.garga@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi,
> > > > > > > > > >> > >
> > > > > > > > > >> > > I recently noticed about Apache Lens project.
> > Currently
> > > we
> > > > > are
> > > > > > > > > >> building
> > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as
> there
> > no
> > > > > such
> > > > > > > > Apache
> > > > > > > > > >> open
> > > > > > > > > >> > > source project exists. Now as Lens supports REST
> APIs
> > > for
> > > > > > Hadoop
> > > > > > > > and
> > > > > > > > > >> > JDBC,
> > > > > > > > > >> > > I want to know is there any plan to support for
> Apache
> > > > > > Cassandra
> > > > > > > > as
> > > > > > > > > >> its
> > > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector)
> which
> > > are
> > > > > > more
> > > > > > > > SQL
> > > > > > > > > >> like.
> > > > > > > > > >> > > If yes, I wish to know the details and contribute as
> > > well.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks
> > > > > > > > > >> > > Ajay
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Rajat Khandelwal <ra...@gmail.com>.
I'm using idea too. There are no compilation issues. Can you do
mvn clean install -DskipTests -DskipCheck
mvn idea:idea
from command line.
Also, check your java version. Lens is configured for java >=7
On Fri, Jul 24, 2015 at 1:36 PM Ajay <aj...@gmail.com> wrote:
> From console, the build goes thru fine.
>
> From Idea IDE, it is giving compilation errors (say one such file
> QueryExecutionServiceImp.java
> Lens Server project).
>
> Are you guys using Eclipse?
>
>
> On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > Could you share the error. Might be env issues.
> > Also can you build from the console directly ?
> >
> > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
> >
> > > Thanks.
> > >
> > > I am using intelliJ Idea IDE. I checked out the source from GIT. But I
> am
> > > seeing some compilation issues. Is there any documentation or notes on
> > how
> > > to set up the development envirnment?
> > >
> > > -
> > > Ajay
> > >
> > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> amareshwari@gmail.com
> > >
> > > wrote:
> > >
> > > > Hello Ajay,
> > > >
> > > > Nice to see your interest in exploring Lens.
> > > >
> > > > Looking at the requirement posted by you, seems you are only
> interested
> > > in
> > > > REST api on top of cassandra. See the lens REST api for query service
> > > here
> > > > -
> http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > >
> > > > Lens provides REST api for query actions (submit, fetch results,
> > cancel,
> > > > getstatus), query history and OLAP data model.
> > > >
> > > > If you are not interested in OLAP data model, you can simple run
> native
> > > > queries on lens which would give REST api and history for you as
> well.
> > > For
> > > > more details on OLAP data model, you can read the doc on website.
> > > >
> > > > For new driver to be rewritten, it need not know anything about OLAP
> > data
> > > > model. But all drivers are given HiveQL query to execute, if driver
> > > accepts
> > > > Hive QL directly you dont have to do much on the query itself. If
> not,
> > > the
> > > > driver has to translate the query into something driver can
> understand.
> > > >
> > > > For example : JDBCDriver converts HiveQL to SQL. - You can have look
> at
> > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except it
> > > might
> > > > have to map some UDFs to SQL udfs.
> > > >
> > > > ok. Coming back to your questions, answers are inline.
> > > >
> > > >
> > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
> > > >
> > > > > Thanks.
> > > > >
> > > > > Couple of questions (that comes to my mind right now)
> > > > >
> > > > > 1) To create a Cluster instance to connect to a Cassandra cluster
> > > > > (Cassandra Java driver), we need to know the following:
> > > > > a) ip address and port no of few instances of the cluster
> > > > > b) authentication (if enabled)
> > > > > c) few more configuration (loading balancing policy, ret policy and
> > > > > compression etc..)
> > > > >
> > > > > Lens drivers are following xml based configuration for passing
> > > > information. See
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > and
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > ,
> > > > how driver specific configuration is put. Also read
> > > > http://lens.incubator.apache.org/admin/config-server.html to
> > understand
> > > > configuration in lens.
> > > >
> > > >
> > > > > Do we have any configuration support(yaml file or any other) in
> Lens.
> > > If
> > > > > yes, any pointers (java file names or WIKI)
> > > > >
> > > > > 2) Cassandra cluser/session and prepared statements are multi
> thread
> > > safe
> > > > > and recommended to be cached and reused.
> > > > >
> > > > > How does Lens support Caching (in memory or distributed)?. Any
> > pointers
> > > > > (java file names or WIKI)
> > > > >
> > > > > I did not understand the caching part here. But if you are talking
> > > about
> > > > cassandra sessions to be used across user queries, you can look at
> how
> > > hive
> > > > sessions and connections are used in HiveDriver -
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > >
> > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > That is great. We can probably pick the most recent Java driver
> and
> > > > > > Cassandra version then. Since it addresses couple of old issues.
> > > > > >
> > > > > > Regarding Spark SQL for querying Cassandra I would let other
> > > > contributors
> > > > > > suggest.
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Thanks Yash.
> > > > > > >
> > > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java
> driver
> > > > [1].
> > > > > > What
> > > > > > > is the Cassandra and Java driver version we plan to support?. I
> > > have
> > > > > > worked
> > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> > > > FILTERING
> > > > > > > worked.
> > > > > > >
> > > > > > > Secondly, as you mentioned I am aware of these limitation in
> CQL.
> > > But
> > > > > > more
> > > > > > > features recently added in Cassandra 2.2.x. Also, other option
> to
> > > > work
> > > > > > > around this to use Cassandra Spark connector and use Spark SQL.
> > > > > > >
> > > > > > > 1)
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> yash360@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Just to add on...
> > > > > > > >
> > > > > > > > There will be few challenges to support entire SQL on
> Cassandra
> > > > > > however.
> > > > > > > > Last time I tried translating SQL to CQL I faced couple of
> > > > scenarios
> > > > > -
> > > > > > > >
> > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > 3. OR condition in where clause
> > > > > > > >
> > > > > > > > You can probably start without these being a blocker - In
> > > parallel
> > > > we
> > > > > > can
> > > > > > > > discuss how this can be implemented on our stack. Others can
> > > > provide
> > > > > > > their
> > > > > > > > opinions here.
> > > > > > > >
> > > > > > > > Hope its helpful.
> > > > > > > >
> > > > > > > > -----
> > > > > > > > Examples:
> > > > > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id)
> > > where
> > > > > > > > (id,rank) is primary key pair.
> > > > > > > >
> > > > > > > > 1.
> > > > > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > > > > > Bad Request: No indexed columns present in by-columns clause
> > with
> > > > > Equal
> > > > > > > > operator
> > > > > > > >
> > > > > > > > 2.
> > > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > > Bad Request: Cannot execute this query as it might involve
> data
> > > > > > filtering
> > > > > > > > and thus may have unpredictable performance. If you want to
> > > execute
> > > > > > this
> > > > > > > > query despite the performance unpredictability, use ALLOW
> > > FILTERING
> > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java
> driver
> > as
> > > > of
> > > > > > now.
> > > > > > > >
> > > > > > > > 3.
> > > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > > > id='id0004';
> > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > >
> > > > > > > > 4. Valid Query:
> > > > > > > > cqlsh:recsys> select * from trending_now where id='id0004'
> and
> > > > > rank=4;
> > > > > > > >
> > > > > > > > id | rank | pog_id
> > > > > > > > --------+------+--------
> > > > > > > > id0004 | 4 | 10002
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > yash360@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > The basic idea is to translate the Lens Query plan (Which
> is
> > a
> > > > Hive
> > > > > > > AST)
> > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > >
> > > > > > > > > The best example I can think of right now is the Elastic
> > Search
> > > > > patch
> > > > > > > by
> > > > > > > > > Amruth. You can go through the JIRA [1] for the detailed
> > > > discussion
> > > > > > and
> > > > > > > > the
> > > > > > > > > Review Board [2] for code reference.
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > >
> > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> ajay.garga@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks Yash.
> > > > > > > > >>
> > > > > > > > >> Is there any documentation or WIKI on the Lens Driver. I
> am
> > > > going
> > > > > > > > through
> > > > > > > > >> the code as well.
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >> Ajay
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > yash360@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Ajay,
> > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens but
> > no
> > > > one
> > > > > > has
> > > > > > > > >> picked
> > > > > > > > >> > it up yet. It would be great if you can pick up the task
> > and
> > > > > > submit
> > > > > > > a
> > > > > > > > >> patch
> > > > > > > > >> > for review.
> > > > > > > > >> >
> > > > > > > > >> > Also drop a note on the list in case you stumble upon
> any
> > > > issue.
> > > > > > > > Someone
> > > > > > > > >> > will always be around to help you out.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > >> >
> > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > ajay.garga@gmail.com>
> > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi,
> > > > > > > > >> > >
> > > > > > > > >> > > I recently noticed about Apache Lens project.
> Currently
> > we
> > > > are
> > > > > > > > >> building
> > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there
> no
> > > > such
> > > > > > > Apache
> > > > > > > > >> open
> > > > > > > > >> > > source project exists. Now as Lens supports REST APIs
> > for
> > > > > Hadoop
> > > > > > > and
> > > > > > > > >> > JDBC,
> > > > > > > > >> > > I want to know is there any plan to support for Apache
> > > > > Cassandra
> > > > > > > as
> > > > > > > > >> its
> > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector) which
> > are
> > > > > more
> > > > > > > SQL
> > > > > > > > >> like.
> > > > > > > > >> > > If yes, I wish to know the details and contribute as
> > well.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks
> > > > > > > > >> > > Ajay
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
I guess we use both. I am on Idea.
I simply build from console and import the project in Idea as a maven
project.
On Fri, Jul 24, 2015 at 1:35 PM, Ajay <aj...@gmail.com> wrote:
> From console, the build goes thru fine.
>
> From Idea IDE, it is giving compilation errors (say one such file
> QueryExecutionServiceImp.java
> Lens Server project).
>
> Are you guys using Eclipse?
>
>
> On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > Could you share the error. Might be env issues.
> > Also can you build from the console directly ?
> >
> > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
> >
> > > Thanks.
> > >
> > > I am using intelliJ Idea IDE. I checked out the source from GIT. But I
> am
> > > seeing some compilation issues. Is there any documentation or notes on
> > how
> > > to set up the development envirnment?
> > >
> > > -
> > > Ajay
> > >
> > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> amareshwari@gmail.com
> > >
> > > wrote:
> > >
> > > > Hello Ajay,
> > > >
> > > > Nice to see your interest in exploring Lens.
> > > >
> > > > Looking at the requirement posted by you, seems you are only
> interested
> > > in
> > > > REST api on top of cassandra. See the lens REST api for query service
> > > here
> > > > -
> http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > >
> > > > Lens provides REST api for query actions (submit, fetch results,
> > cancel,
> > > > getstatus), query history and OLAP data model.
> > > >
> > > > If you are not interested in OLAP data model, you can simple run
> native
> > > > queries on lens which would give REST api and history for you as
> well.
> > > For
> > > > more details on OLAP data model, you can read the doc on website.
> > > >
> > > > For new driver to be rewritten, it need not know anything about OLAP
> > data
> > > > model. But all drivers are given HiveQL query to execute, if driver
> > > accepts
> > > > Hive QL directly you dont have to do much on the query itself. If
> not,
> > > the
> > > > driver has to translate the query into something driver can
> understand.
> > > >
> > > > For example : JDBCDriver converts HiveQL to SQL. - You can have look
> at
> > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except it
> > > might
> > > > have to map some UDFs to SQL udfs.
> > > >
> > > > ok. Coming back to your questions, answers are inline.
> > > >
> > > >
> > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
> > > >
> > > > > Thanks.
> > > > >
> > > > > Couple of questions (that comes to my mind right now)
> > > > >
> > > > > 1) To create a Cluster instance to connect to a Cassandra cluster
> > > > > (Cassandra Java driver), we need to know the following:
> > > > > a) ip address and port no of few instances of the cluster
> > > > > b) authentication (if enabled)
> > > > > c) few more configuration (loading balancing policy, ret policy and
> > > > > compression etc..)
> > > > >
> > > > > Lens drivers are following xml based configuration for passing
> > > > information. See
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > and
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > ,
> > > > how driver specific configuration is put. Also read
> > > > http://lens.incubator.apache.org/admin/config-server.html to
> > understand
> > > > configuration in lens.
> > > >
> > > >
> > > > > Do we have any configuration support(yaml file or any other) in
> Lens.
> > > If
> > > > > yes, any pointers (java file names or WIKI)
> > > > >
> > > > > 2) Cassandra cluser/session and prepared statements are multi
> thread
> > > safe
> > > > > and recommended to be cached and reused.
> > > > >
> > > > > How does Lens support Caching (in memory or distributed)?. Any
> > pointers
> > > > > (java file names or WIKI)
> > > > >
> > > > > I did not understand the caching part here. But if you are talking
> > > about
> > > > cassandra sessions to be used across user queries, you can look at
> how
> > > hive
> > > > sessions and connections are used in HiveDriver -
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > >
> > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > That is great. We can probably pick the most recent Java driver
> and
> > > > > > Cassandra version then. Since it addresses couple of old issues.
> > > > > >
> > > > > > Regarding Spark SQL for querying Cassandra I would let other
> > > > contributors
> > > > > > suggest.
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Thanks Yash.
> > > > > > >
> > > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java
> driver
> > > > [1].
> > > > > > What
> > > > > > > is the Cassandra and Java driver version we plan to support?. I
> > > have
> > > > > > worked
> > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> > > > FILTERING
> > > > > > > worked.
> > > > > > >
> > > > > > > Secondly, as you mentioned I am aware of these limitation in
> CQL.
> > > But
> > > > > > more
> > > > > > > features recently added in Cassandra 2.2.x. Also, other option
> to
> > > > work
> > > > > > > around this to use Cassandra Spark connector and use Spark SQL.
> > > > > > >
> > > > > > > 1)
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> yash360@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Just to add on...
> > > > > > > >
> > > > > > > > There will be few challenges to support entire SQL on
> Cassandra
> > > > > > however.
> > > > > > > > Last time I tried translating SQL to CQL I faced couple of
> > > > scenarios
> > > > > -
> > > > > > > >
> > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > 3. OR condition in where clause
> > > > > > > >
> > > > > > > > You can probably start without these being a blocker - In
> > > parallel
> > > > we
> > > > > > can
> > > > > > > > discuss how this can be implemented on our stack. Others can
> > > > provide
> > > > > > > their
> > > > > > > > opinions here.
> > > > > > > >
> > > > > > > > Hope its helpful.
> > > > > > > >
> > > > > > > > -----
> > > > > > > > Examples:
> > > > > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id)
> > > where
> > > > > > > > (id,rank) is primary key pair.
> > > > > > > >
> > > > > > > > 1.
> > > > > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > > > > > Bad Request: No indexed columns present in by-columns clause
> > with
> > > > > Equal
> > > > > > > > operator
> > > > > > > >
> > > > > > > > 2.
> > > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > > Bad Request: Cannot execute this query as it might involve
> data
> > > > > > filtering
> > > > > > > > and thus may have unpredictable performance. If you want to
> > > execute
> > > > > > this
> > > > > > > > query despite the performance unpredictability, use ALLOW
> > > FILTERING
> > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java
> driver
> > as
> > > > of
> > > > > > now.
> > > > > > > >
> > > > > > > > 3.
> > > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > > > id='id0004';
> > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > >
> > > > > > > > 4. Valid Query:
> > > > > > > > cqlsh:recsys> select * from trending_now where id='id0004'
> and
> > > > > rank=4;
> > > > > > > >
> > > > > > > > id | rank | pog_id
> > > > > > > > --------+------+--------
> > > > > > > > id0004 | 4 | 10002
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > yash360@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > The basic idea is to translate the Lens Query plan (Which
> is
> > a
> > > > Hive
> > > > > > > AST)
> > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > >
> > > > > > > > > The best example I can think of right now is the Elastic
> > Search
> > > > > patch
> > > > > > > by
> > > > > > > > > Amruth. You can go through the JIRA [1] for the detailed
> > > > discussion
> > > > > > and
> > > > > > > > the
> > > > > > > > > Review Board [2] for code reference.
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > >
> > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> ajay.garga@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks Yash.
> > > > > > > > >>
> > > > > > > > >> Is there any documentation or WIKI on the Lens Driver. I
> am
> > > > going
> > > > > > > > through
> > > > > > > > >> the code as well.
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >> Ajay
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > yash360@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Ajay,
> > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens but
> > no
> > > > one
> > > > > > has
> > > > > > > > >> picked
> > > > > > > > >> > it up yet. It would be great if you can pick up the task
> > and
> > > > > > submit
> > > > > > > a
> > > > > > > > >> patch
> > > > > > > > >> > for review.
> > > > > > > > >> >
> > > > > > > > >> > Also drop a note on the list in case you stumble upon
> any
> > > > issue.
> > > > > > > > Someone
> > > > > > > > >> > will always be around to help you out.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > >> >
> > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > ajay.garga@gmail.com>
> > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi,
> > > > > > > > >> > >
> > > > > > > > >> > > I recently noticed about Apache Lens project.
> Currently
> > we
> > > > are
> > > > > > > > >> building
> > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there
> no
> > > > such
> > > > > > > Apache
> > > > > > > > >> open
> > > > > > > > >> > > source project exists. Now as Lens supports REST APIs
> > for
> > > > > Hadoop
> > > > > > > and
> > > > > > > > >> > JDBC,
> > > > > > > > >> > > I want to know is there any plan to support for Apache
> > > > > Cassandra
> > > > > > > as
> > > > > > > > >> its
> > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector) which
> > are
> > > > > more
> > > > > > > SQL
> > > > > > > > >> like.
> > > > > > > > >> > > If yes, I wish to know the details and contribute as
> > well.
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks
> > > > > > > > >> > > Ajay
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
>From console, the build goes thru fine.
>From Idea IDE, it is giving compilation errors (say one such file
QueryExecutionServiceImp.java
Lens Server project).
Are you guys using Eclipse?
On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <ya...@gmail.com> wrote:
> Could you share the error. Might be env issues.
> Also can you build from the console directly ?
>
> On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
>
> > Thanks.
> >
> > I am using intelliJ Idea IDE. I checked out the source from GIT. But I am
> > seeing some compilation issues. Is there any documentation or notes on
> how
> > to set up the development envirnment?
> >
> > -
> > Ajay
> >
> > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <amareshwari@gmail.com
> >
> > wrote:
> >
> > > Hello Ajay,
> > >
> > > Nice to see your interest in exploring Lens.
> > >
> > > Looking at the requirement posted by you, seems you are only interested
> > in
> > > REST api on top of cassandra. See the lens REST api for query service
> > here
> > > - http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > >
> > > Lens provides REST api for query actions (submit, fetch results,
> cancel,
> > > getstatus), query history and OLAP data model.
> > >
> > > If you are not interested in OLAP data model, you can simple run native
> > > queries on lens which would give REST api and history for you as well.
> > For
> > > more details on OLAP data model, you can read the doc on website.
> > >
> > > For new driver to be rewritten, it need not know anything about OLAP
> data
> > > model. But all drivers are given HiveQL query to execute, if driver
> > accepts
> > > Hive QL directly you dont have to do much on the query itself. If not,
> > the
> > > driver has to translate the query into something driver can understand.
> > >
> > > For example : JDBCDriver converts HiveQL to SQL. - You can have look at
> > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except it
> > might
> > > have to map some UDFs to SQL udfs.
> > >
> > > ok. Coming back to your questions, answers are inline.
> > >
> > >
> > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
> > >
> > > > Thanks.
> > > >
> > > > Couple of questions (that comes to my mind right now)
> > > >
> > > > 1) To create a Cluster instance to connect to a Cassandra cluster
> > > > (Cassandra Java driver), we need to know the following:
> > > > a) ip address and port no of few instances of the cluster
> > > > b) authentication (if enabled)
> > > > c) few more configuration (loading balancing policy, ret policy and
> > > > compression etc..)
> > > >
> > > > Lens drivers are following xml based configuration for passing
> > > information. See
> > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > and
> > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > ,
> > > how driver specific configuration is put. Also read
> > > http://lens.incubator.apache.org/admin/config-server.html to
> understand
> > > configuration in lens.
> > >
> > >
> > > > Do we have any configuration support(yaml file or any other) in Lens.
> > If
> > > > yes, any pointers (java file names or WIKI)
> > > >
> > > > 2) Cassandra cluser/session and prepared statements are multi thread
> > safe
> > > > and recommended to be cached and reused.
> > > >
> > > > How does Lens support Caching (in memory or distributed)?. Any
> pointers
> > > > (java file names or WIKI)
> > > >
> > > > I did not understand the caching part here. But if you are talking
> > about
> > > cassandra sessions to be used across user queries, you can look at how
> > hive
> > > sessions and connections are used in HiveDriver -
> > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > >
> > >
> > > >
> > > >
> > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com>
> > wrote:
> > > >
> > > > > That is great. We can probably pick the most recent Java driver and
> > > > > Cassandra version then. Since it addresses couple of old issues.
> > > > >
> > > > > Regarding Spark SQL for querying Cassandra I would let other
> > > contributors
> > > > > suggest.
> > > > >
> > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com>
> wrote:
> > > > >
> > > > > > Thanks Yash.
> > > > > >
> > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java driver
> > > [1].
> > > > > What
> > > > > > is the Cassandra and Java driver version we plan to support?. I
> > have
> > > > > worked
> > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> > > FILTERING
> > > > > > worked.
> > > > > >
> > > > > > Secondly, as you mentioned I am aware of these limitation in CQL.
> > But
> > > > > more
> > > > > > features recently added in Cassandra 2.2.x. Also, other option to
> > > work
> > > > > > around this to use Cassandra Spark connector and use Spark SQL.
> > > > > >
> > > > > > 1)
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <yash360@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Just to add on...
> > > > > > >
> > > > > > > There will be few challenges to support entire SQL on Cassandra
> > > > > however.
> > > > > > > Last time I tried translating SQL to CQL I faced couple of
> > > scenarios
> > > > -
> > > > > > >
> > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > 2. Filtering by subset of primary key
> > > > > > > 3. OR condition in where clause
> > > > > > >
> > > > > > > You can probably start without these being a blocker - In
> > parallel
> > > we
> > > > > can
> > > > > > > discuss how this can be implemented on our stack. Others can
> > > provide
> > > > > > their
> > > > > > > opinions here.
> > > > > > >
> > > > > > > Hope its helpful.
> > > > > > >
> > > > > > > -----
> > > > > > > Examples:
> > > > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id)
> > where
> > > > > > > (id,rank) is primary key pair.
> > > > > > >
> > > > > > > 1.
> > > > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > > > > Bad Request: No indexed columns present in by-columns clause
> with
> > > > Equal
> > > > > > > operator
> > > > > > >
> > > > > > > 2.
> > > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > > Bad Request: Cannot execute this query as it might involve data
> > > > > filtering
> > > > > > > and thus may have unpredictable performance. If you want to
> > execute
> > > > > this
> > > > > > > query despite the performance unpredictability, use ALLOW
> > FILTERING
> > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver
> as
> > > of
> > > > > now.
> > > > > > >
> > > > > > > 3.
> > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > > id='id0004';
> > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > >
> > > > > > > 4. Valid Query:
> > > > > > > cqlsh:recsys> select * from trending_now where id='id0004' and
> > > > rank=4;
> > > > > > >
> > > > > > > id | rank | pog_id
> > > > > > > --------+------+--------
> > > > > > > id0004 | 4 | 10002
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> yash360@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > The basic idea is to translate the Lens Query plan (Which is
> a
> > > Hive
> > > > > > AST)
> > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > >
> > > > > > > > The best example I can think of right now is the Elastic
> Search
> > > > patch
> > > > > > by
> > > > > > > > Amruth. You can go through the JIRA [1] for the detailed
> > > discussion
> > > > > and
> > > > > > > the
> > > > > > > > Review Board [2] for code reference.
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > >
> > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > >
> > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks Yash.
> > > > > > > >>
> > > > > > > >> Is there any documentation or WIKI on the Lens Driver. I am
> > > going
> > > > > > > through
> > > > > > > >> the code as well.
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >> Ajay
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > yash360@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Ajay,
> > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens but
> no
> > > one
> > > > > has
> > > > > > > >> picked
> > > > > > > >> > it up yet. It would be great if you can pick up the task
> and
> > > > > submit
> > > > > > a
> > > > > > > >> patch
> > > > > > > >> > for review.
> > > > > > > >> >
> > > > > > > >> > Also drop a note on the list in case you stumble upon any
> > > issue.
> > > > > > > Someone
> > > > > > > >> > will always be around to help you out.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > > >> >
> > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > ajay.garga@gmail.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi,
> > > > > > > >> > >
> > > > > > > >> > > I recently noticed about Apache Lens project. Currently
> we
> > > are
> > > > > > > >> building
> > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there no
> > > such
> > > > > > Apache
> > > > > > > >> open
> > > > > > > >> > > source project exists. Now as Lens supports REST APIs
> for
> > > > Hadoop
> > > > > > and
> > > > > > > >> > JDBC,
> > > > > > > >> > > I want to know is there any plan to support for Apache
> > > > Cassandra
> > > > > > as
> > > > > > > >> its
> > > > > > > >> > > support CQL and Spark SQL (thru Spark connector) which
> are
> > > > more
> > > > > > SQL
> > > > > > > >> like.
> > > > > > > >> > > If yes, I wish to know the details and contribute as
> well.
> > > > > > > >> > >
> > > > > > > >> > > Thanks
> > > > > > > >> > > Ajay
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
Could you share the error. Might be env issues.
Also can you build from the console directly ?
On Fri, Jul 24, 2015 at 12:54 PM, Ajay <aj...@gmail.com> wrote:
> Thanks.
>
> I am using intelliJ Idea IDE. I checked out the source from GIT. But I am
> seeing some compilation issues. Is there any documentation or notes on how
> to set up the development envirnment?
>
> -
> Ajay
>
> On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <am...@gmail.com>
> wrote:
>
> > Hello Ajay,
> >
> > Nice to see your interest in exploring Lens.
> >
> > Looking at the requirement posted by you, seems you are only interested
> in
> > REST api on top of cassandra. See the lens REST api for query service
> here
> > - http://lens.incubator.apache.org/resource_QueryServiceResource.html
> >
> > Lens provides REST api for query actions (submit, fetch results, cancel,
> > getstatus), query history and OLAP data model.
> >
> > If you are not interested in OLAP data model, you can simple run native
> > queries on lens which would give REST api and history for you as well.
> For
> > more details on OLAP data model, you can read the doc on website.
> >
> > For new driver to be rewritten, it need not know anything about OLAP data
> > model. But all drivers are given HiveQL query to execute, if driver
> accepts
> > Hive QL directly you dont have to do much on the query itself. If not,
> the
> > driver has to translate the query into something driver can understand.
> >
> > For example : JDBCDriver converts HiveQL to SQL. - You can have look at
> > JDBCDriver for converting into SQL. Hive QL is almost SQL, except it
> might
> > have to map some UDFs to SQL udfs.
> >
> > ok. Coming back to your questions, answers are inline.
> >
> >
> > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
> >
> > > Thanks.
> > >
> > > Couple of questions (that comes to my mind right now)
> > >
> > > 1) To create a Cluster instance to connect to a Cassandra cluster
> > > (Cassandra Java driver), we need to know the following:
> > > a) ip address and port no of few instances of the cluster
> > > b) authentication (if enabled)
> > > c) few more configuration (loading balancing policy, ret policy and
> > > compression etc..)
> > >
> > > Lens drivers are following xml based configuration for passing
> > information. See
> >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > and
> >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > ,
> > how driver specific configuration is put. Also read
> > http://lens.incubator.apache.org/admin/config-server.html to understand
> > configuration in lens.
> >
> >
> > > Do we have any configuration support(yaml file or any other) in Lens.
> If
> > > yes, any pointers (java file names or WIKI)
> > >
> > > 2) Cassandra cluser/session and prepared statements are multi thread
> safe
> > > and recommended to be cached and reused.
> > >
> > > How does Lens support Caching (in memory or distributed)?. Any pointers
> > > (java file names or WIKI)
> > >
> > > I did not understand the caching part here. But if you are talking
> about
> > cassandra sessions to be used across user queries, you can look at how
> hive
> > sessions and connections are used in HiveDriver -
> >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> >
> >
> > >
> > >
> > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com>
> wrote:
> > >
> > > > That is great. We can probably pick the most recent Java driver and
> > > > Cassandra version then. Since it addresses couple of old issues.
> > > >
> > > > Regarding Spark SQL for querying Cassandra I would let other
> > contributors
> > > > suggest.
> > > >
> > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com> wrote:
> > > >
> > > > > Thanks Yash.
> > > > >
> > > > > In Between, ALLOW FILTERING is supported in Cassandra Java driver
> > [1].
> > > > What
> > > > > is the Cassandra and Java driver version we plan to support?. I
> have
> > > > worked
> > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> > FILTERING
> > > > > worked.
> > > > >
> > > > > Secondly, as you mentioned I am aware of these limitation in CQL.
> But
> > > > more
> > > > > features recently added in Cassandra 2.2.x. Also, other option to
> > work
> > > > > around this to use Cassandra Spark connector and use Spark SQL.
> > > > >
> > > > > 1)
> > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Just to add on...
> > > > > >
> > > > > > There will be few challenges to support entire SQL on Cassandra
> > > > however.
> > > > > > Last time I tried translating SQL to CQL I faced couple of
> > scenarios
> > > -
> > > > > >
> > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > 2. Filtering by subset of primary key
> > > > > > 3. OR condition in where clause
> > > > > >
> > > > > > You can probably start without these being a blocker - In
> parallel
> > we
> > > > can
> > > > > > discuss how this can be implemented on our stack. Others can
> > provide
> > > > > their
> > > > > > opinions here.
> > > > > >
> > > > > > Hope its helpful.
> > > > > >
> > > > > > -----
> > > > > > Examples:
> > > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id)
> where
> > > > > > (id,rank) is primary key pair.
> > > > > >
> > > > > > 1.
> > > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > > > Bad Request: No indexed columns present in by-columns clause with
> > > Equal
> > > > > > operator
> > > > > >
> > > > > > 2.
> > > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > > Bad Request: Cannot execute this query as it might involve data
> > > > filtering
> > > > > > and thus may have unpredictable performance. If you want to
> execute
> > > > this
> > > > > > query despite the performance unpredictability, use ALLOW
> FILTERING
> > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver as
> > of
> > > > now.
> > > > > >
> > > > > > 3.
> > > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> > id='id0004';
> > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > >
> > > > > > 4. Valid Query:
> > > > > > cqlsh:recsys> select * from trending_now where id='id0004' and
> > > rank=4;
> > > > > >
> > > > > > id | rank | pog_id
> > > > > > --------+------+--------
> > > > > > id0004 | 4 | 10002
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > The basic idea is to translate the Lens Query plan (Which is a
> > Hive
> > > > > AST)
> > > > > > > to the DataStore specific Plan/Query.
> > > > > > >
> > > > > > > The best example I can think of right now is the Elastic Search
> > > patch
> > > > > by
> > > > > > > Amruth. You can go through the JIRA [1] for the detailed
> > discussion
> > > > and
> > > > > > the
> > > > > > > Review Board [2] for code reference.
> > > > > > >
> > > > > > > Best regards
> > > > > > >
> > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > >
> > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> Thanks Yash.
> > > > > > >>
> > > > > > >> Is there any documentation or WIKI on the Lens Driver. I am
> > going
> > > > > > through
> > > > > > >> the code as well.
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >> Ajay
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > yash360@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Ajay,
> > > > > > >> > Welcome to the Lens Dev.
> > > > > > >> > We do have plans for a Cassandra driver [1] for Lens but no
> > one
> > > > has
> > > > > > >> picked
> > > > > > >> > it up yet. It would be great if you can pick up the task and
> > > > submit
> > > > > a
> > > > > > >> patch
> > > > > > >> > for review.
> > > > > > >> >
> > > > > > >> > Also drop a note on the list in case you stumble upon any
> > issue.
> > > > > > Someone
> > > > > > >> > will always be around to help you out.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > > >> >
> > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> ajay.garga@gmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi,
> > > > > > >> > >
> > > > > > >> > > I recently noticed about Apache Lens project. Currently we
> > are
> > > > > > >> building
> > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there no
> > such
> > > > > Apache
> > > > > > >> open
> > > > > > >> > > source project exists. Now as Lens supports REST APIs for
> > > Hadoop
> > > > > and
> > > > > > >> > JDBC,
> > > > > > >> > > I want to know is there any plan to support for Apache
> > > Cassandra
> > > > > as
> > > > > > >> its
> > > > > > >> > > support CQL and Spark SQL (thru Spark connector) which are
> > > more
> > > > > SQL
> > > > > > >> like.
> > > > > > >> > > If yes, I wish to know the details and contribute as well.
> > > > > > >> > >
> > > > > > >> > > Thanks
> > > > > > >> > > Ajay
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
Thanks.
I am using intelliJ Idea IDE. I checked out the source from GIT. But I am
seeing some compilation issues. Is there any documentation or notes on how
to set up the development envirnment?
-
Ajay
On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <am...@gmail.com>
wrote:
> Hello Ajay,
>
> Nice to see your interest in exploring Lens.
>
> Looking at the requirement posted by you, seems you are only interested in
> REST api on top of cassandra. See the lens REST api for query service here
> - http://lens.incubator.apache.org/resource_QueryServiceResource.html
>
> Lens provides REST api for query actions (submit, fetch results, cancel,
> getstatus), query history and OLAP data model.
>
> If you are not interested in OLAP data model, you can simple run native
> queries on lens which would give REST api and history for you as well. For
> more details on OLAP data model, you can read the doc on website.
>
> For new driver to be rewritten, it need not know anything about OLAP data
> model. But all drivers are given HiveQL query to execute, if driver accepts
> Hive QL directly you dont have to do much on the query itself. If not, the
> driver has to translate the query into something driver can understand.
>
> For example : JDBCDriver converts HiveQL to SQL. - You can have look at
> JDBCDriver for converting into SQL. Hive QL is almost SQL, except it might
> have to map some UDFs to SQL udfs.
>
> ok. Coming back to your questions, answers are inline.
>
>
> On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
>
> > Thanks.
> >
> > Couple of questions (that comes to my mind right now)
> >
> > 1) To create a Cluster instance to connect to a Cassandra cluster
> > (Cassandra Java driver), we need to know the following:
> > a) ip address and port no of few instances of the cluster
> > b) authentication (if enabled)
> > c) few more configuration (loading balancing policy, ret policy and
> > compression etc..)
> >
> > Lens drivers are following xml based configuration for passing
> information. See
>
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> and
>
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> ,
> how driver specific configuration is put. Also read
> http://lens.incubator.apache.org/admin/config-server.html to understand
> configuration in lens.
>
>
> > Do we have any configuration support(yaml file or any other) in Lens. If
> > yes, any pointers (java file names or WIKI)
> >
> > 2) Cassandra cluser/session and prepared statements are multi thread safe
> > and recommended to be cached and reused.
> >
> > How does Lens support Caching (in memory or distributed)?. Any pointers
> > (java file names or WIKI)
> >
> > I did not understand the caching part here. But if you are talking about
> cassandra sessions to be used across user queries, you can look at how hive
> sessions and connections are used in HiveDriver -
>
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
>
>
> >
> >
> > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com> wrote:
> >
> > > That is great. We can probably pick the most recent Java driver and
> > > Cassandra version then. Since it addresses couple of old issues.
> > >
> > > Regarding Spark SQL for querying Cassandra I would let other
> contributors
> > > suggest.
> > >
> > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com> wrote:
> > >
> > > > Thanks Yash.
> > > >
> > > > In Between, ALLOW FILTERING is supported in Cassandra Java driver
> [1].
> > > What
> > > > is the Cassandra and Java driver version we plan to support?. I have
> > > worked
> > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW
> FILTERING
> > > > worked.
> > > >
> > > > Secondly, as you mentioned I am aware of these limitation in CQL. But
> > > more
> > > > features recently added in Cassandra 2.2.x. Also, other option to
> work
> > > > around this to use Cassandra Spark connector and use Spark SQL.
> > > >
> > > > 1)
> > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > >
> > > >
> > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com>
> > wrote:
> > > >
> > > > > Just to add on...
> > > > >
> > > > > There will be few challenges to support entire SQL on Cassandra
> > > however.
> > > > > Last time I tried translating SQL to CQL I faced couple of
> scenarios
> > -
> > > > >
> > > > > 1. Filtering non indexed column in Cassandra
> > > > > 2. Filtering by subset of primary key
> > > > > 3. OR condition in where clause
> > > > >
> > > > > You can probably start without these being a blocker - In parallel
> we
> > > can
> > > > > discuss how this can be implemented on our stack. Others can
> provide
> > > > their
> > > > > opinions here.
> > > > >
> > > > > Hope its helpful.
> > > > >
> > > > > -----
> > > > > Examples:
> > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id) where
> > > > > (id,rank) is primary key pair.
> > > > >
> > > > > 1.
> > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > > Bad Request: No indexed columns present in by-columns clause with
> > Equal
> > > > > operator
> > > > >
> > > > > 2.
> > > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > > Bad Request: Cannot execute this query as it might involve data
> > > filtering
> > > > > and thus may have unpredictable performance. If you want to execute
> > > this
> > > > > query despite the performance unpredictability, use ALLOW FILTERING
> > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver as
> of
> > > now.
> > > > >
> > > > > 3.
> > > > > cqlsh:recsys> select * from trending_now where rank=4 or
> id='id0004';
> > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > >
> > > > > 4. Valid Query:
> > > > > cqlsh:recsys> select * from trending_now where id='id0004' and
> > rank=4;
> > > > >
> > > > > id | rank | pog_id
> > > > > --------+------+--------
> > > > > id0004 | 4 | 10002
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > The basic idea is to translate the Lens Query plan (Which is a
> Hive
> > > > AST)
> > > > > > to the DataStore specific Plan/Query.
> > > > > >
> > > > > > The best example I can think of right now is the Elastic Search
> > patch
> > > > by
> > > > > > Amruth. You can go through the JIRA [1] for the detailed
> discussion
> > > and
> > > > > the
> > > > > > Review Board [2] for code reference.
> > > > > >
> > > > > > Best regards
> > > > > >
> > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > 2. https://reviews.apache.org/r/36434
> > > > > >
> > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com>
> > wrote:
> > > > > >
> > > > > >> Thanks Yash.
> > > > > >>
> > > > > >> Is there any documentation or WIKI on the Lens Driver. I am
> going
> > > > > through
> > > > > >> the code as well.
> > > > > >>
> > > > > >> Thanks
> > > > > >> Ajay
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> yash360@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi Ajay,
> > > > > >> > Welcome to the Lens Dev.
> > > > > >> > We do have plans for a Cassandra driver [1] for Lens but no
> one
> > > has
> > > > > >> picked
> > > > > >> > it up yet. It would be great if you can pick up the task and
> > > submit
> > > > a
> > > > > >> patch
> > > > > >> > for review.
> > > > > >> >
> > > > > >> > Also drop a note on the list in case you stumble upon any
> issue.
> > > > > Someone
> > > > > >> > will always be around to help you out.
> > > > > >> >
> > > > > >> >
> > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > > >> >
> > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > Hi,
> > > > > >> > >
> > > > > >> > > I recently noticed about Apache Lens project. Currently we
> are
> > > > > >> building
> > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there no
> such
> > > > Apache
> > > > > >> open
> > > > > >> > > source project exists. Now as Lens supports REST APIs for
> > Hadoop
> > > > and
> > > > > >> > JDBC,
> > > > > >> > > I want to know is there any plan to support for Apache
> > Cassandra
> > > > as
> > > > > >> its
> > > > > >> > > support CQL and Spark SQL (thru Spark connector) which are
> > more
> > > > SQL
> > > > > >> like.
> > > > > >> > > If yes, I wish to know the details and contribute as well.
> > > > > >> > >
> > > > > >> > > Thanks
> > > > > >> > > Ajay
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by "amareshwarisr ." <am...@gmail.com>.
Hello Ajay,
Nice to see your interest in exploring Lens.
Looking at the requirement posted by you, seems you are only interested in
REST api on top of cassandra. See the lens REST api for query service here
- http://lens.incubator.apache.org/resource_QueryServiceResource.html
Lens provides REST api for query actions (submit, fetch results, cancel,
getstatus), query history and OLAP data model.
If you are not interested in OLAP data model, you can simple run native
queries on lens which would give REST api and history for you as well. For
more details on OLAP data model, you can read the doc on website.
For new driver to be rewritten, it need not know anything about OLAP data
model. But all drivers are given HiveQL query to execute, if driver accepts
Hive QL directly you dont have to do much on the query itself. If not, the
driver has to translate the query into something driver can understand.
For example : JDBCDriver converts HiveQL to SQL. - You can have look at
JDBCDriver for converting into SQL. Hive QL is almost SQL, except it might
have to map some UDFs to SQL udfs.
ok. Coming back to your questions, answers are inline.
On Fri, Jul 24, 2015 at 9:08 AM, Ajay <aj...@gmail.com> wrote:
> Thanks.
>
> Couple of questions (that comes to my mind right now)
>
> 1) To create a Cluster instance to connect to a Cassandra cluster
> (Cassandra Java driver), we need to know the following:
> a) ip address and port no of few instances of the cluster
> b) authentication (if enabled)
> c) few more configuration (loading balancing policy, ret policy and
> compression etc..)
>
> Lens drivers are following xml based configuration for passing
information. See
https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
and
https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml,
how driver specific configuration is put. Also read
http://lens.incubator.apache.org/admin/config-server.html to understand
configuration in lens.
> Do we have any configuration support(yaml file or any other) in Lens. If
> yes, any pointers (java file names or WIKI)
>
> 2) Cassandra cluser/session and prepared statements are multi thread safe
> and recommended to be cached and reused.
>
> How does Lens support Caching (in memory or distributed)?. Any pointers
> (java file names or WIKI)
>
> I did not understand the caching part here. But if you are talking about
cassandra sessions to be used across user queries, you can look at how hive
sessions and connections are used in HiveDriver -
https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
>
>
> On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > That is great. We can probably pick the most recent Java driver and
> > Cassandra version then. Since it addresses couple of old issues.
> >
> > Regarding Spark SQL for querying Cassandra I would let other contributors
> > suggest.
> >
> > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com> wrote:
> >
> > > Thanks Yash.
> > >
> > > In Between, ALLOW FILTERING is supported in Cassandra Java driver [1].
> > What
> > > is the Cassandra and Java driver version we plan to support?. I have
> > worked
> > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW FILTERING
> > > worked.
> > >
> > > Secondly, as you mentioned I am aware of these limitation in CQL. But
> > more
> > > features recently added in Cassandra 2.2.x. Also, other option to work
> > > around this to use Cassandra Spark connector and use Spark SQL.
> > >
> > > 1)
> > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > >
> > >
> > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com>
> wrote:
> > >
> > > > Just to add on...
> > > >
> > > > There will be few challenges to support entire SQL on Cassandra
> > however.
> > > > Last time I tried translating SQL to CQL I faced couple of scenarios
> -
> > > >
> > > > 1. Filtering non indexed column in Cassandra
> > > > 2. Filtering by subset of primary key
> > > > 3. OR condition in where clause
> > > >
> > > > You can probably start without these being a blocker - In parallel we
> > can
> > > > discuss how this can be implemented on our stack. Others can provide
> > > their
> > > > opinions here.
> > > >
> > > > Hope its helpful.
> > > >
> > > > -----
> > > > Examples:
> > > > Here 'trending_now' is a dummy table with (id, rank, pog_id) where
> > > > (id,rank) is primary key pair.
> > > >
> > > > 1.
> > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > > Bad Request: No indexed columns present in by-columns clause with
> Equal
> > > > operator
> > > >
> > > > 2.
> > > > cqlsh:recsys> select * from trending_now where rank=4;
> > > > Bad Request: Cannot execute this query as it might involve data
> > filtering
> > > > and thus may have unpredictable performance. If you want to execute
> > this
> > > > query despite the performance unpredictability, use ALLOW FILTERING
> > > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver as of
> > now.
> > > >
> > > > 3.
> > > > cqlsh:recsys> select * from trending_now where rank=4 or id='id0004';
> > > > Bad Request: line 1:40 missing EOF at 'or'
> > > >
> > > > 4. Valid Query:
> > > > cqlsh:recsys> select * from trending_now where id='id0004' and
> rank=4;
> > > >
> > > > id | rank | pog_id
> > > > --------+------+--------
> > > > id0004 | 4 | 10002
> > > >
> > > >
> > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com>
> > wrote:
> > > >
> > > > > The basic idea is to translate the Lens Query plan (Which is a Hive
> > > AST)
> > > > > to the DataStore specific Plan/Query.
> > > > >
> > > > > The best example I can think of right now is the Elastic Search
> patch
> > > by
> > > > > Amruth. You can go through the JIRA [1] for the detailed discussion
> > and
> > > > the
> > > > > Review Board [2] for code reference.
> > > > >
> > > > > Best regards
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > 2. https://reviews.apache.org/r/36434
> > > > >
> > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com>
> wrote:
> > > > >
> > > > >> Thanks Yash.
> > > > >>
> > > > >> Is there any documentation or WIKI on the Lens Driver. I am going
> > > > through
> > > > >> the code as well.
> > > > >>
> > > > >> Thanks
> > > > >> Ajay
> > > > >>
> > > > >>
> > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > Hi Ajay,
> > > > >> > Welcome to the Lens Dev.
> > > > >> > We do have plans for a Cassandra driver [1] for Lens but no one
> > has
> > > > >> picked
> > > > >> > it up yet. It would be great if you can pick up the task and
> > submit
> > > a
> > > > >> patch
> > > > >> > for review.
> > > > >> >
> > > > >> > Also drop a note on the list in case you stumble upon any issue.
> > > > Someone
> > > > >> > will always be around to help you out.
> > > > >> >
> > > > >> >
> > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > > >> >
> > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com>
> > > wrote:
> > > > >> >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > I recently noticed about Apache Lens project. Currently we are
> > > > >> building
> > > > >> > > REST APIs for Apache Cassandra (proprietary) as there no such
> > > Apache
> > > > >> open
> > > > >> > > source project exists. Now as Lens supports REST APIs for
> Hadoop
> > > and
> > > > >> > JDBC,
> > > > >> > > I want to know is there any plan to support for Apache
> Cassandra
> > > as
> > > > >> its
> > > > >> > > support CQL and Spark SQL (thru Spark connector) which are
> more
> > > SQL
> > > > >> like.
> > > > >> > > If yes, I wish to know the details and contribute as well.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > > Ajay
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
Thanks.
Couple of questions (that comes to my mind right now)
1) To create a Cluster instance to connect to a Cassandra cluster
(Cassandra Java driver), we need to know the following:
a) ip address and port no of few instances of the cluster
b) authentication (if enabled)
c) few more configuration (loading balancing policy, ret policy and
compression etc..)
Do we have any configuration support(yaml file or any other) in Lens. If
yes, any pointers (java file names or WIKI)
2) Cassandra cluser/session and prepared statements are multi thread safe
and recommended to be cached and reused.
How does Lens support Caching (in memory or distributed)?. Any pointers
(java file names or WIKI)
Thanks
Ajay
On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <ya...@gmail.com> wrote:
> That is great. We can probably pick the most recent Java driver and
> Cassandra version then. Since it addresses couple of old issues.
>
> Regarding Spark SQL for querying Cassandra I would let other contributors
> suggest.
>
> On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com> wrote:
>
> > Thanks Yash.
> >
> > In Between, ALLOW FILTERING is supported in Cassandra Java driver [1].
> What
> > is the Cassandra and Java driver version we plan to support?. I have
> worked
> > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW FILTERING
> > worked.
> >
> > Secondly, as you mentioned I am aware of these limitation in CQL. But
> more
> > features recently added in Cassandra 2.2.x. Also, other option to work
> > around this to use Cassandra Spark connector and use Spark SQL.
> >
> > 1)
> >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> >
> >
> > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com> wrote:
> >
> > > Just to add on...
> > >
> > > There will be few challenges to support entire SQL on Cassandra
> however.
> > > Last time I tried translating SQL to CQL I faced couple of scenarios -
> > >
> > > 1. Filtering non indexed column in Cassandra
> > > 2. Filtering by subset of primary key
> > > 3. OR condition in where clause
> > >
> > > You can probably start without these being a blocker - In parallel we
> can
> > > discuss how this can be implemented on our stack. Others can provide
> > their
> > > opinions here.
> > >
> > > Hope its helpful.
> > >
> > > -----
> > > Examples:
> > > Here 'trending_now' is a dummy table with (id, rank, pog_id) where
> > > (id,rank) is primary key pair.
> > >
> > > 1.
> > > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > > Bad Request: No indexed columns present in by-columns clause with Equal
> > > operator
> > >
> > > 2.
> > > cqlsh:recsys> select * from trending_now where rank=4;
> > > Bad Request: Cannot execute this query as it might involve data
> filtering
> > > and thus may have unpredictable performance. If you want to execute
> this
> > > query despite the performance unpredictability, use ALLOW FILTERING
> > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver as of
> now.
> > >
> > > 3.
> > > cqlsh:recsys> select * from trending_now where rank=4 or id='id0004';
> > > Bad Request: line 1:40 missing EOF at 'or'
> > >
> > > 4. Valid Query:
> > > cqlsh:recsys> select * from trending_now where id='id0004' and rank=4;
> > >
> > > id | rank | pog_id
> > > --------+------+--------
> > > id0004 | 4 | 10002
> > >
> > >
> > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com>
> wrote:
> > >
> > > > The basic idea is to translate the Lens Query plan (Which is a Hive
> > AST)
> > > > to the DataStore specific Plan/Query.
> > > >
> > > > The best example I can think of right now is the Elastic Search patch
> > by
> > > > Amruth. You can go through the JIRA [1] for the detailed discussion
> and
> > > the
> > > > Review Board [2] for code reference.
> > > >
> > > > Best regards
> > > >
> > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > 2. https://reviews.apache.org/r/36434
> > > >
> > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
> > > >
> > > >> Thanks Yash.
> > > >>
> > > >> Is there any documentation or WIKI on the Lens Driver. I am going
> > > through
> > > >> the code as well.
> > > >>
> > > >> Thanks
> > > >> Ajay
> > > >>
> > > >>
> > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Ajay,
> > > >> > Welcome to the Lens Dev.
> > > >> > We do have plans for a Cassandra driver [1] for Lens but no one
> has
> > > >> picked
> > > >> > it up yet. It would be great if you can pick up the task and
> submit
> > a
> > > >> patch
> > > >> > for review.
> > > >> >
> > > >> > Also drop a note on the list in case you stumble upon any issue.
> > > Someone
> > > >> > will always be around to help you out.
> > > >> >
> > > >> >
> > > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > > >> >
> > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com>
> > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > I recently noticed about Apache Lens project. Currently we are
> > > >> building
> > > >> > > REST APIs for Apache Cassandra (proprietary) as there no such
> > Apache
> > > >> open
> > > >> > > source project exists. Now as Lens supports REST APIs for Hadoop
> > and
> > > >> > JDBC,
> > > >> > > I want to know is there any plan to support for Apache Cassandra
> > as
> > > >> its
> > > >> > > support CQL and Spark SQL (thru Spark connector) which are more
> > SQL
> > > >> like.
> > > >> > > If yes, I wish to know the details and contribute as well.
> > > >> > >
> > > >> > > Thanks
> > > >> > > Ajay
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
That is great. We can probably pick the most recent Java driver and
Cassandra version then. Since it addresses couple of old issues.
Regarding Spark SQL for querying Cassandra I would let other contributors
suggest.
On Thu, Jul 23, 2015 at 11:04 PM, Ajay <aj...@gmail.com> wrote:
> Thanks Yash.
>
> In Between, ALLOW FILTERING is supported in Cassandra Java driver [1]. What
> is the Cassandra and Java driver version we plan to support?. I have worked
> on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW FILTERING
> worked.
>
> Secondly, as you mentioned I am aware of these limitation in CQL. But more
> features recently added in Cassandra 2.2.x. Also, other option to work
> around this to use Cassandra Spark connector and use Spark SQL.
>
> 1)
>
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
>
>
> On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > Just to add on...
> >
> > There will be few challenges to support entire SQL on Cassandra however.
> > Last time I tried translating SQL to CQL I faced couple of scenarios -
> >
> > 1. Filtering non indexed column in Cassandra
> > 2. Filtering by subset of primary key
> > 3. OR condition in where clause
> >
> > You can probably start without these being a blocker - In parallel we can
> > discuss how this can be implemented on our stack. Others can provide
> their
> > opinions here.
> >
> > Hope its helpful.
> >
> > -----
> > Examples:
> > Here 'trending_now' is a dummy table with (id, rank, pog_id) where
> > (id,rank) is primary key pair.
> >
> > 1.
> > cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> > Bad Request: No indexed columns present in by-columns clause with Equal
> > operator
> >
> > 2.
> > cqlsh:recsys> select * from trending_now where rank=4;
> > Bad Request: Cannot execute this query as it might involve data filtering
> > and thus may have unpredictable performance. If you want to execute this
> > query despite the performance unpredictability, use ALLOW FILTERING
> > P.S. ALLOW FILTERING is not permitted in Cassandra java driver as of now.
> >
> > 3.
> > cqlsh:recsys> select * from trending_now where rank=4 or id='id0004';
> > Bad Request: line 1:40 missing EOF at 'or'
> >
> > 4. Valid Query:
> > cqlsh:recsys> select * from trending_now where id='id0004' and rank=4;
> >
> > id | rank | pog_id
> > --------+------+--------
> > id0004 | 4 | 10002
> >
> >
> > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com> wrote:
> >
> > > The basic idea is to translate the Lens Query plan (Which is a Hive
> AST)
> > > to the DataStore specific Plan/Query.
> > >
> > > The best example I can think of right now is the Elastic Search patch
> by
> > > Amruth. You can go through the JIRA [1] for the detailed discussion and
> > the
> > > Review Board [2] for code reference.
> > >
> > > Best regards
> > >
> > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > 2. https://reviews.apache.org/r/36434
> > >
> > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
> > >
> > >> Thanks Yash.
> > >>
> > >> Is there any documentation or WIKI on the Lens Driver. I am going
> > through
> > >> the code as well.
> > >>
> > >> Thanks
> > >> Ajay
> > >>
> > >>
> > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Ajay,
> > >> > Welcome to the Lens Dev.
> > >> > We do have plans for a Cassandra driver [1] for Lens but no one has
> > >> picked
> > >> > it up yet. It would be great if you can pick up the task and submit
> a
> > >> patch
> > >> > for review.
> > >> >
> > >> > Also drop a note on the list in case you stumble upon any issue.
> > Someone
> > >> > will always be around to help you out.
> > >> >
> > >> >
> > >> > 1. https://issues.apache.org/jira/browse/LENS-654
> > >> >
> > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com>
> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I recently noticed about Apache Lens project. Currently we are
> > >> building
> > >> > > REST APIs for Apache Cassandra (proprietary) as there no such
> Apache
> > >> open
> > >> > > source project exists. Now as Lens supports REST APIs for Hadoop
> and
> > >> > JDBC,
> > >> > > I want to know is there any plan to support for Apache Cassandra
> as
> > >> its
> > >> > > support CQL and Spark SQL (thru Spark connector) which are more
> SQL
> > >> like.
> > >> > > If yes, I wish to know the details and contribute as well.
> > >> > >
> > >> > > Thanks
> > >> > > Ajay
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
Thanks Yash.
In Between, ALLOW FILTERING is supported in Cassandra Java driver [1]. What
is the Cassandra and Java driver version we plan to support?. I have worked
on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW FILTERING
worked.
Secondly, as you mentioned I am aware of these limitation in CQL. But more
features recently added in Cassandra 2.2.x. Also, other option to work
around this to use Cassandra Spark connector and use Spark SQL.
1)
http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <ya...@gmail.com> wrote:
> Just to add on...
>
> There will be few challenges to support entire SQL on Cassandra however.
> Last time I tried translating SQL to CQL I faced couple of scenarios -
>
> 1. Filtering non indexed column in Cassandra
> 2. Filtering by subset of primary key
> 3. OR condition in where clause
>
> You can probably start without these being a blocker - In parallel we can
> discuss how this can be implemented on our stack. Others can provide their
> opinions here.
>
> Hope its helpful.
>
> -----
> Examples:
> Here 'trending_now' is a dummy table with (id, rank, pog_id) where
> (id,rank) is primary key pair.
>
> 1.
> cqlsh:recsys> select * from trending_now where pog_id=10004 ;
> Bad Request: No indexed columns present in by-columns clause with Equal
> operator
>
> 2.
> cqlsh:recsys> select * from trending_now where rank=4;
> Bad Request: Cannot execute this query as it might involve data filtering
> and thus may have unpredictable performance. If you want to execute this
> query despite the performance unpredictability, use ALLOW FILTERING
> P.S. ALLOW FILTERING is not permitted in Cassandra java driver as of now.
>
> 3.
> cqlsh:recsys> select * from trending_now where rank=4 or id='id0004';
> Bad Request: line 1:40 missing EOF at 'or'
>
> 4. Valid Query:
> cqlsh:recsys> select * from trending_now where id='id0004' and rank=4;
>
> id | rank | pog_id
> --------+------+--------
> id0004 | 4 | 10002
>
>
> On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > The basic idea is to translate the Lens Query plan (Which is a Hive AST)
> > to the DataStore specific Plan/Query.
> >
> > The best example I can think of right now is the Elastic Search patch by
> > Amruth. You can go through the JIRA [1] for the detailed discussion and
> the
> > Review Board [2] for code reference.
> >
> > Best regards
> >
> > 1. https://issues.apache.org/jira/browse/LENS-252
> > 2. https://reviews.apache.org/r/36434
> >
> > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
> >
> >> Thanks Yash.
> >>
> >> Is there any documentation or WIKI on the Lens Driver. I am going
> through
> >> the code as well.
> >>
> >> Thanks
> >> Ajay
> >>
> >>
> >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com>
> wrote:
> >>
> >> > Hi Ajay,
> >> > Welcome to the Lens Dev.
> >> > We do have plans for a Cassandra driver [1] for Lens but no one has
> >> picked
> >> > it up yet. It would be great if you can pick up the task and submit a
> >> patch
> >> > for review.
> >> >
> >> > Also drop a note on the list in case you stumble upon any issue.
> Someone
> >> > will always be around to help you out.
> >> >
> >> >
> >> > 1. https://issues.apache.org/jira/browse/LENS-654
> >> >
> >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I recently noticed about Apache Lens project. Currently we are
> >> building
> >> > > REST APIs for Apache Cassandra (proprietary) as there no such Apache
> >> open
> >> > > source project exists. Now as Lens supports REST APIs for Hadoop and
> >> > JDBC,
> >> > > I want to know is there any plan to support for Apache Cassandra as
> >> its
> >> > > support CQL and Spark SQL (thru Spark connector) which are more SQL
> >> like.
> >> > > If yes, I wish to know the details and contribute as well.
> >> > >
> >> > > Thanks
> >> > > Ajay
> >> > >
> >> >
> >>
> >
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
Just to add on...
There will be few challenges to support entire SQL on Cassandra however.
Last time I tried translating SQL to CQL I faced couple of scenarios -
1. Filtering non indexed column in Cassandra
2. Filtering by subset of primary key
3. OR condition in where clause
You can probably start without these being a blocker - In parallel we can
discuss how this can be implemented on our stack. Others can provide their
opinions here.
Hope its helpful.
-----
Examples:
Here 'trending_now' is a dummy table with (id, rank, pog_id) where
(id,rank) is primary key pair.
1.
cqlsh:recsys> select * from trending_now where pog_id=10004 ;
Bad Request: No indexed columns present in by-columns clause with Equal
operator
2.
cqlsh:recsys> select * from trending_now where rank=4;
Bad Request: Cannot execute this query as it might involve data filtering
and thus may have unpredictable performance. If you want to execute this
query despite the performance unpredictability, use ALLOW FILTERING
P.S. ALLOW FILTERING is not permitted in Cassandra java driver as of now.
3.
cqlsh:recsys> select * from trending_now where rank=4 or id='id0004';
Bad Request: line 1:40 missing EOF at 'or'
4. Valid Query:
cqlsh:recsys> select * from trending_now where id='id0004' and rank=4;
id | rank | pog_id
--------+------+--------
id0004 | 4 | 10002
On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <ya...@gmail.com> wrote:
> The basic idea is to translate the Lens Query plan (Which is a Hive AST)
> to the DataStore specific Plan/Query.
>
> The best example I can think of right now is the Elastic Search patch by
> Amruth. You can go through the JIRA [1] for the detailed discussion and the
> Review Board [2] for code reference.
>
> Best regards
>
> 1. https://issues.apache.org/jira/browse/LENS-252
> 2. https://reviews.apache.org/r/36434
>
> On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
>
>> Thanks Yash.
>>
>> Is there any documentation or WIKI on the Lens Driver. I am going through
>> the code as well.
>>
>> Thanks
>> Ajay
>>
>>
>> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com> wrote:
>>
>> > Hi Ajay,
>> > Welcome to the Lens Dev.
>> > We do have plans for a Cassandra driver [1] for Lens but no one has
>> picked
>> > it up yet. It would be great if you can pick up the task and submit a
>> patch
>> > for review.
>> >
>> > Also drop a note on the list in case you stumble upon any issue. Someone
>> > will always be around to help you out.
>> >
>> >
>> > 1. https://issues.apache.org/jira/browse/LENS-654
>> >
>> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > I recently noticed about Apache Lens project. Currently we are
>> building
>> > > REST APIs for Apache Cassandra (proprietary) as there no such Apache
>> open
>> > > source project exists. Now as Lens supports REST APIs for Hadoop and
>> > JDBC,
>> > > I want to know is there any plan to support for Apache Cassandra as
>> its
>> > > support CQL and Spark SQL (thru Spark connector) which are more SQL
>> like.
>> > > If yes, I wish to know the details and contribute as well.
>> > >
>> > > Thanks
>> > > Ajay
>> > >
>> >
>>
>
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
The basic idea is to translate the Lens Query plan (Which is a Hive AST) to
the DataStore specific Plan/Query.
The best example I can think of right now is the Elastic Search patch by
Amruth. You can go through the JIRA [1] for the detailed discussion and the
Review Board [2] for code reference.
Best regards
1. https://issues.apache.org/jira/browse/LENS-252
2. https://reviews.apache.org/r/36434
On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
> Thanks Yash.
>
> Is there any documentation or WIKI on the Lens Driver. I am going through
> the code as well.
>
> Thanks
> Ajay
>
>
> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > Hi Ajay,
> > Welcome to the Lens Dev.
> > We do have plans for a Cassandra driver [1] for Lens but no one has
> picked
> > it up yet. It would be great if you can pick up the task and submit a
> patch
> > for review.
> >
> > Also drop a note on the list in case you stumble upon any issue. Someone
> > will always be around to help you out.
> >
> >
> > 1. https://issues.apache.org/jira/browse/LENS-654
> >
> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I recently noticed about Apache Lens project. Currently we are building
> > > REST APIs for Apache Cassandra (proprietary) as there no such Apache
> open
> > > source project exists. Now as Lens supports REST APIs for Hadoop and
> > JDBC,
> > > I want to know is there any plan to support for Apache Cassandra as its
> > > support CQL and Spark SQL (thru Spark connector) which are more SQL
> like.
> > > If yes, I wish to know the details and contribute as well.
> > >
> > > Thanks
> > > Ajay
> > >
> >
>
Re: Lens driver for Cassandra
Posted by Raju Bairishetti <ra...@gmail.com>.
Hi Ajay,
You can find driver information at LensDriver-Wiki
<https://cwiki.apache.org/confluence/display/LENS/Lens+Driver>.
On Thu, Jul 23, 2015 at 9:30 PM, Ajay <aj...@gmail.com> wrote:
> Thanks Yash.
>
> Is there any documentation or WIKI on the Lens Driver. I am going through
> the code as well.
>
> Thanks
> Ajay
>
>
> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com> wrote:
>
> > Hi Ajay,
> > Welcome to the Lens Dev.
> > We do have plans for a Cassandra driver [1] for Lens but no one has
> picked
> > it up yet. It would be great if you can pick up the task and submit a
> patch
> > for review.
> >
> > Also drop a note on the list in case you stumble upon any issue. Someone
> > will always be around to help you out.
> >
> >
> > 1. https://issues.apache.org/jira/browse/LENS-654
> >
> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I recently noticed about Apache Lens project. Currently we are building
> > > REST APIs for Apache Cassandra (proprietary) as there no such Apache
> open
> > > source project exists. Now as Lens supports REST APIs for Hadoop and
> > JDBC,
> > > I want to know is there any plan to support for Apache Cassandra as its
> > > support CQL and Spark SQL (thru Spark connector) which are more SQL
> like.
> > > If yes, I wish to know the details and contribute as well.
> > >
> > > Thanks
> > > Ajay
> > >
> >
>
--
Thanks
Raju Bairishetti,
www.inmobi.com
*2012 ME Best Mobile Ad Network Award Winner
[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png]
Re: Lens driver for Cassandra
Posted by Ajay <aj...@gmail.com>.
Thanks Yash.
Is there any documentation or WIKI on the Lens Driver. I am going through
the code as well.
Thanks
Ajay
On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <ya...@gmail.com> wrote:
> Hi Ajay,
> Welcome to the Lens Dev.
> We do have plans for a Cassandra driver [1] for Lens but no one has picked
> it up yet. It would be great if you can pick up the task and submit a patch
> for review.
>
> Also drop a note on the list in case you stumble upon any issue. Someone
> will always be around to help you out.
>
>
> 1. https://issues.apache.org/jira/browse/LENS-654
>
> On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
>
> > Hi,
> >
> > I recently noticed about Apache Lens project. Currently we are building
> > REST APIs for Apache Cassandra (proprietary) as there no such Apache open
> > source project exists. Now as Lens supports REST APIs for Hadoop and
> JDBC,
> > I want to know is there any plan to support for Apache Cassandra as its
> > support CQL and Spark SQL (thru Spark connector) which are more SQL like.
> > If yes, I wish to know the details and contribute as well.
> >
> > Thanks
> > Ajay
> >
>
Re: Lens driver for Cassandra
Posted by Yash Sharma <ya...@gmail.com>.
Hi Ajay,
Welcome to the Lens Dev.
We do have plans for a Cassandra driver [1] for Lens but no one has picked
it up yet. It would be great if you can pick up the task and submit a patch
for review.
Also drop a note on the list in case you stumble upon any issue. Someone
will always be around to help you out.
1. https://issues.apache.org/jira/browse/LENS-654
On Thu, Jul 23, 2015 at 11:41 AM, Ajay <aj...@gmail.com> wrote:
> Hi,
>
> I recently noticed about Apache Lens project. Currently we are building
> REST APIs for Apache Cassandra (proprietary) as there no such Apache open
> source project exists. Now as Lens supports REST APIs for Hadoop and JDBC,
> I want to know is there any plan to support for Apache Cassandra as its
> support CQL and Spark SQL (thru Spark connector) which are more SQL like.
> If yes, I wish to know the details and contribute as well.
>
> Thanks
> Ajay
>