You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Vishnu Viswanath <vi...@gmail.com> on 2015/01/16 04:46:15 UTC
solr indexing using pig script
Hi All,
I am in indexing data into solr using pig script.
I have two such scripts, and I tried combining these two scripts into a
single one.
i.e., i have script 1 that does
--------------------
a)SET solr collection info for collection 1
b)LOAD data
c)FILTER data for SOLR collection number 1
d)STORE data to solr
and script 2 that does
-------------------
a)SET solr collection info for collection 2
b)LOAD data
c)FILTER data for SOLR collection number 2
d)STORE data to solr
combined script looks something like
--------------
a)SET solr collection info for collection 1
b)LOAD data
c)FILTER data from (b) for SOLR collection number 1
d)STORE data to solr
e)SET solr collection info for collection 2
f)FILTER data from (b) for SOLR collection number 2
g)STORE data to solr
But the store function fails when I run the combined script where as it
runs fine if I run scripts 1 and 2 separately.
Any idea?
Regards,
Vishnu
Re: solr indexing using pig script
Posted by Pradeep Gollakota <pr...@gmail.com>.
Just out of curiosity, why are you using SET to set the solr collection?
I'm not sure if you're using an out of the box Load/Store Func, but if I
were to design it, I would use the "location" of a Load/Store Func to
specify which solr collection to write to.
Is it possible for you to redesign this way?
On Thu, Jan 15, 2015 at 9:41 PM, Vishnu Viswanath <
vishnu.viswanath25@gmail.com> wrote:
> Thanks
>
> SET sets the SOLR collection name. When the STORE is invoked, the data
> will be ingested into the collection name set before.
>
> So, the problem must be because the second set is overriding the
> collection name and the STORE is failing.
>
> Is there any way to overcome this? Because most of the processing time is
> taken in the load and I don't want to do it twice.
>
> Regards,
> Vishnu Viswanath
>
> > On 16-Jan-2015, at 09:29, Cheolsoo Park <pi...@gmail.com> wrote:
> >
> > What does "SET" do for Solr? Pig pre-processes all the set commands in
> the
> > entire script before executing any query, and values are overwritten if
> the
> > same key is set more than once. In your example, you have two set
> commands.
> > If you're thinking that different values will be applied in each section,
> > that's not the case. e) will overwrite a).
> >
> >
> > On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath <
> > vishnu.viswanath25@gmail.com> wrote:
> >
> >> Hi All,
> >>
> >> I am in indexing data into solr using pig script.
> >> I have two such scripts, and I tried combining these two scripts into a
> >> single one.
> >>
> >> i.e., i have script 1 that does
> >> --------------------
> >> a)SET solr collection info for collection 1
> >> b)LOAD data
> >> c)FILTER data for SOLR collection number 1
> >> d)STORE data to solr
> >>
> >>
> >> and script 2 that does
> >> -------------------
> >> a)SET solr collection info for collection 2
> >> b)LOAD data
> >> c)FILTER data for SOLR collection number 2
> >> d)STORE data to solr
> >>
> >>
> >> combined script looks something like
> >> --------------
> >> a)SET solr collection info for collection 1
> >> b)LOAD data
> >> c)FILTER data from (b) for SOLR collection number 1
> >> d)STORE data to solr
> >> e)SET solr collection info for collection 2
> >> f)FILTER data from (b) for SOLR collection number 2
> >> g)STORE data to solr
> >>
> >> But the store function fails when I run the combined script where as it
> >> runs fine if I run scripts 1 and 2 separately.
> >>
> >> Any idea?
> >>
> >> Regards,
> >> Vishnu
> >>
>
Re: solr indexing using pig script
Posted by Vishnu Viswanath <vi...@gmail.com>.
Thanks
SET sets the SOLR collection name. When the STORE is invoked, the data will be ingested into the collection name set before.
So, the problem must be because the second set is overriding the collection name and the STORE is failing.
Is there any way to overcome this? Because most of the processing time is taken in the load and I don't want to do it twice.
Regards,
Vishnu Viswanath
> On 16-Jan-2015, at 09:29, Cheolsoo Park <pi...@gmail.com> wrote:
>
> What does "SET" do for Solr? Pig pre-processes all the set commands in the
> entire script before executing any query, and values are overwritten if the
> same key is set more than once. In your example, you have two set commands.
> If you're thinking that different values will be applied in each section,
> that's not the case. e) will overwrite a).
>
>
> On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath <
> vishnu.viswanath25@gmail.com> wrote:
>
>> Hi All,
>>
>> I am in indexing data into solr using pig script.
>> I have two such scripts, and I tried combining these two scripts into a
>> single one.
>>
>> i.e., i have script 1 that does
>> --------------------
>> a)SET solr collection info for collection 1
>> b)LOAD data
>> c)FILTER data for SOLR collection number 1
>> d)STORE data to solr
>>
>>
>> and script 2 that does
>> -------------------
>> a)SET solr collection info for collection 2
>> b)LOAD data
>> c)FILTER data for SOLR collection number 2
>> d)STORE data to solr
>>
>>
>> combined script looks something like
>> --------------
>> a)SET solr collection info for collection 1
>> b)LOAD data
>> c)FILTER data from (b) for SOLR collection number 1
>> d)STORE data to solr
>> e)SET solr collection info for collection 2
>> f)FILTER data from (b) for SOLR collection number 2
>> g)STORE data to solr
>>
>> But the store function fails when I run the combined script where as it
>> runs fine if I run scripts 1 and 2 separately.
>>
>> Any idea?
>>
>> Regards,
>> Vishnu
>>
Re: solr indexing using pig script
Posted by Cheolsoo Park <pi...@gmail.com>.
What does "SET" do for Solr? Pig pre-processes all the set commands in the
entire script before executing any query, and values are overwritten if the
same key is set more than once. In your example, you have two set commands.
If you're thinking that different values will be applied in each section,
that's not the case. e) will overwrite a).
On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath <
vishnu.viswanath25@gmail.com> wrote:
> Hi All,
>
> I am in indexing data into solr using pig script.
> I have two such scripts, and I tried combining these two scripts into a
> single one.
>
> i.e., i have script 1 that does
> --------------------
> a)SET solr collection info for collection 1
> b)LOAD data
> c)FILTER data for SOLR collection number 1
> d)STORE data to solr
>
>
> and script 2 that does
> -------------------
> a)SET solr collection info for collection 2
> b)LOAD data
> c)FILTER data for SOLR collection number 2
> d)STORE data to solr
>
>
> combined script looks something like
> --------------
> a)SET solr collection info for collection 1
> b)LOAD data
> c)FILTER data from (b) for SOLR collection number 1
> d)STORE data to solr
> e)SET solr collection info for collection 2
> f)FILTER data from (b) for SOLR collection number 2
> g)STORE data to solr
>
> But the store function fails when I run the combined script where as it
> runs fine if I run scripts 1 and 2 separately.
>
> Any idea?
>
> Regards,
> Vishnu
>
Re: solr indexing using pig script
Posted by Cheolsoo Park <pi...@gmail.com>.
What does "SET" do for Solr? Pig pre-processes all the set commands in the
entire script before executing any query, and values are overwritten if the
same key is set more than once. In your example, you have two set commands.
If you're thinking that different values will be applied in each section,
that's not the case. e) will overwrite a).
On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath <
vishnu.viswanath25@gmail.com> wrote:
> Hi All,
>
> I am in indexing data into solr using pig script.
> I have two such scripts, and I tried combining these two scripts into a
> single one.
>
> i.e., i have script 1 that does
> --------------------
> a)SET solr collection info for collection 1
> b)LOAD data
> c)FILTER data for SOLR collection number 1
> d)STORE data to solr
>
>
> and script 2 that does
> -------------------
> a)SET solr collection info for collection 2
> b)LOAD data
> c)FILTER data for SOLR collection number 2
> d)STORE data to solr
>
>
> combined script looks something like
> --------------
> a)SET solr collection info for collection 1
> b)LOAD data
> c)FILTER data from (b) for SOLR collection number 1
> d)STORE data to solr
> e)SET solr collection info for collection 2
> f)FILTER data from (b) for SOLR collection number 2
> g)STORE data to solr
>
> But the store function fails when I run the combined script where as it
> runs fine if I run scripts 1 and 2 separately.
>
> Any idea?
>
> Regards,
> Vishnu
>