You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Siddharth Verma <ve...@snapdeal.com> on 2016/04/27 19:41:56 UTC
Query regarding spark on cassandra
Hi,
I dont know, if someone has faced this problem or not.
I am running a job where some data is loaded from cassandra table. From
that data, i make some insert and delete statements.
and execute it (using forEach)
Code snippet:
boolean deleteStatus= connector.openSession().execute(delete).wasApplied();
boolean insertStatus =
connector.openSession().execute(insert).wasApplied();
System.out.println(delete+":"+deleteStatus);
System.out.println(insert+":"+insertStatus);
When i run it locally, i see the respective results in the table.
However when i run it on a cluster, sometimes the result is displayed and
sometime the changes don't take place.
I saw the stdout from web-ui of spark, and the query along with true was
printed for both the queries.
I can't understand, what could be the issue.
Any help would be appreciated.
Thanks,
Siddharth Verma
Re: Query regarding spark on cassandra
Posted by Siddharth Verma <ve...@snapdeal.com>.
Edit:
1. dc2 node has been removed.
nodetool status shows only active nodes.
2. Repair done on all nodes.
3. Cassandra restarted
Still it doesn't solve the problem.
On Thu, Apr 28, 2016 at 9:00 AM, Siddharth Verma <
verma.siddharth@snapdeal.com> wrote:
> Hi, If the info could be used
> we are using two DCs
> dc1 - 3 nodes
> dc2 - 1 node
> however, dc2 has been down for 3-4 weeks, and we haven't removed it yet.
>
> spark slaves on same machines as the cassandra nodes.
> each node has two instances of slaves.
>
> spark master on a separate machine.
>
> If anyone could provide insight to the problem, it would be helpful.
>
> Thanks
>
> On Wed, Apr 27, 2016 at 11:11 PM, Siddharth Verma <
> verma.siddharth@snapdeal.com> wrote:
>
>> Hi,
>> I dont know, if someone has faced this problem or not.
>> I am running a job where some data is loaded from cassandra table. From
>> that data, i make some insert and delete statements.
>> and execute it (using forEach)
>>
>> Code snippet:
>> boolean deleteStatus=
>> connector.openSession().execute(delete).wasApplied();
>> boolean insertStatus =
>> connector.openSession().execute(insert).wasApplied();
>> System.out.println(delete+":"+deleteStatus);
>> System.out.println(insert+":"+insertStatus);
>>
>> When i run it locally, i see the respective results in the table.
>>
>> However when i run it on a cluster, sometimes the result is displayed and
>> sometime the changes don't take place.
>> I saw the stdout from web-ui of spark, and the query along with true was
>> printed for both the queries.
>>
>> I can't understand, what could be the issue.
>>
>> Any help would be appreciated.
>>
>> Thanks,
>> Siddharth Verma
>>
>
>
Re: Query regarding spark on cassandra
Posted by Siddharth Verma <ve...@snapdeal.com>.
Hi, If the info could be used
we are using two DCs
dc1 - 3 nodes
dc2 - 1 node
however, dc2 has been down for 3-4 weeks, and we haven't removed it yet.
spark slaves on same machines as the cassandra nodes.
each node has two instances of slaves.
spark master on a separate machine.
If anyone could provide insight to the problem, it would be helpful.
Thanks
On Wed, Apr 27, 2016 at 11:11 PM, Siddharth Verma <
verma.siddharth@snapdeal.com> wrote:
> Hi,
> I dont know, if someone has faced this problem or not.
> I am running a job where some data is loaded from cassandra table. From
> that data, i make some insert and delete statements.
> and execute it (using forEach)
>
> Code snippet:
> boolean deleteStatus= connector.openSession().execute(delete).wasApplied();
> boolean insertStatus =
> connector.openSession().execute(insert).wasApplied();
> System.out.println(delete+":"+deleteStatus);
> System.out.println(insert+":"+insertStatus);
>
> When i run it locally, i see the respective results in the table.
>
> However when i run it on a cluster, sometimes the result is displayed and
> sometime the changes don't take place.
> I saw the stdout from web-ui of spark, and the query along with true was
> printed for both the queries.
>
> I can't understand, what could be the issue.
>
> Any help would be appreciated.
>
> Thanks,
> Siddharth Verma
>
Re: Query regarding spark on cassandra
Posted by Siddharth Verma <ve...@snapdeal.com>.
Anyways, thanks for your reply.
On Thu, Apr 28, 2016 at 1:59 PM, Hannu Kröger <hk...@gmail.com> wrote:
> Ok, then I don’t understand the problem.
>
> Hannu
>
> On 28 Apr 2016, at 11:19, Siddharth Verma <ve...@snapdeal.com>
> wrote:
>
> Hi Hannu,
>
> Had the issue been caused due to read, the insert, and delete statement
> would have been erroneous.
> "I saw the stdout from web-ui of spark, and the query along with true was
> printed for both the queries.".
> The statements were correct as seen on the UI.
> Thanks,
> Siddharth Verma
>
>
>
> On Thu, Apr 28, 2016 at 1:22 PM, Hannu Kröger <hk...@gmail.com> wrote:
>
>> Hi,
>>
>> could it be consistency level issue? If you use ONE for reads and writes,
>> might be that sometimes you don't get what you are writing.
>>
>> See:
>>
>> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
>>
>> Br,
>> Hannu
>>
>>
>> 2016-04-27 20:41 GMT+03:00 Siddharth Verma <ve...@snapdeal.com>
>> :
>>
>>> Hi,
>>> I dont know, if someone has faced this problem or not.
>>> I am running a job where some data is loaded from cassandra table. From
>>> that data, i make some insert and delete statements.
>>> and execute it (using forEach)
>>>
>>> Code snippet:
>>> boolean deleteStatus=
>>> connector.openSession().execute(delete).wasApplied();
>>> boolean insertStatus =
>>> connector.openSession().execute(insert).wasApplied();
>>> System.out.println(delete+":"+deleteStatus);
>>> System.out.println(insert+":"+insertStatus);
>>>
>>> When i run it locally, i see the respective results in the table.
>>>
>>> However when i run it on a cluster, sometimes the result is displayed
>>> and sometime the changes don't take place.
>>> I saw the stdout from web-ui of spark, and the query along with true was
>>> printed for both the queries.
>>>
>>> I can't understand, what could be the issue.
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks,
>>> Siddharth Verma
>>>
>>
>>
>
>
Re: Query regarding spark on cassandra
Posted by Hannu Kröger <hk...@gmail.com>.
Ok, then I don’t understand the problem.
Hannu
> On 28 Apr 2016, at 11:19, Siddharth Verma <ve...@snapdeal.com> wrote:
>
> Hi Hannu,
>
> Had the issue been caused due to read, the insert, and delete statement would have been erroneous.
> "I saw the stdout from web-ui of spark, and the query along with true was printed for both the queries.".
> The statements were correct as seen on the UI.
> Thanks,
> Siddharth Verma
>
>
>
> On Thu, Apr 28, 2016 at 1:22 PM, Hannu Kröger <hkroger@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> could it be consistency level issue? If you use ONE for reads and writes, might be that sometimes you don't get what you are writing.
>
> See:
> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html <https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html>
>
> Br,
> Hannu
>
>
> 2016-04-27 20:41 GMT+03:00 Siddharth Verma <verma.siddharth@snapdeal.com <ma...@snapdeal.com>>:
> Hi,
> I dont know, if someone has faced this problem or not.
> I am running a job where some data is loaded from cassandra table. From that data, i make some insert and delete statements.
> and execute it (using forEach)
>
> Code snippet:
> boolean deleteStatus= connector.openSession().execute(delete).wasApplied();
> boolean insertStatus = connector.openSession().execute(insert).wasApplied();
> System.out.println(delete+":"+deleteStatus);
> System.out.println(insert+":"+insertStatus);
>
> When i run it locally, i see the respective results in the table.
>
> However when i run it on a cluster, sometimes the result is displayed and sometime the changes don't take place.
> I saw the stdout from web-ui of spark, and the query along with true was printed for both the queries.
>
> I can't understand, what could be the issue.
>
> Any help would be appreciated.
>
> Thanks,
> Siddharth Verma
>
>
Re: Query regarding spark on cassandra
Posted by Siddharth Verma <ve...@snapdeal.com>.
Hi Hannu,
Had the issue been caused due to read, the insert, and delete statement
would have been erroneous.
"I saw the stdout from web-ui of spark, and the query along with true was
printed for both the queries.".
The statements were correct as seen on the UI.
Thanks,
Siddharth Verma
On Thu, Apr 28, 2016 at 1:22 PM, Hannu Kröger <hk...@gmail.com> wrote:
> Hi,
>
> could it be consistency level issue? If you use ONE for reads and writes,
> might be that sometimes you don't get what you are writing.
>
> See:
>
> https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
>
> Br,
> Hannu
>
>
> 2016-04-27 20:41 GMT+03:00 Siddharth Verma <ve...@snapdeal.com>:
>
>> Hi,
>> I dont know, if someone has faced this problem or not.
>> I am running a job where some data is loaded from cassandra table. From
>> that data, i make some insert and delete statements.
>> and execute it (using forEach)
>>
>> Code snippet:
>> boolean deleteStatus=
>> connector.openSession().execute(delete).wasApplied();
>> boolean insertStatus =
>> connector.openSession().execute(insert).wasApplied();
>> System.out.println(delete+":"+deleteStatus);
>> System.out.println(insert+":"+insertStatus);
>>
>> When i run it locally, i see the respective results in the table.
>>
>> However when i run it on a cluster, sometimes the result is displayed and
>> sometime the changes don't take place.
>> I saw the stdout from web-ui of spark, and the query along with true was
>> printed for both the queries.
>>
>> I can't understand, what could be the issue.
>>
>> Any help would be appreciated.
>>
>> Thanks,
>> Siddharth Verma
>>
>
>
Re: Query regarding spark on cassandra
Posted by Hannu Kröger <hk...@gmail.com>.
Hi,
could it be consistency level issue? If you use ONE for reads and writes,
might be that sometimes you don't get what you are writing.
See:
https://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
Br,
Hannu
2016-04-27 20:41 GMT+03:00 Siddharth Verma <ve...@snapdeal.com>:
> Hi,
> I dont know, if someone has faced this problem or not.
> I am running a job where some data is loaded from cassandra table. From
> that data, i make some insert and delete statements.
> and execute it (using forEach)
>
> Code snippet:
> boolean deleteStatus= connector.openSession().execute(delete).wasApplied();
> boolean insertStatus =
> connector.openSession().execute(insert).wasApplied();
> System.out.println(delete+":"+deleteStatus);
> System.out.println(insert+":"+insertStatus);
>
> When i run it locally, i see the respective results in the table.
>
> However when i run it on a cluster, sometimes the result is displayed and
> sometime the changes don't take place.
> I saw the stdout from web-ui of spark, and the query along with true was
> printed for both the queries.
>
> I can't understand, what could be the issue.
>
> Any help would be appreciated.
>
> Thanks,
> Siddharth Verma
>