Field Size per document in Solr

[KNitin <ni...@gmail.com>]

Hi,

 I want to get the size of individual fields per document (or per index) in
solrcloud. Is there a way to do this using exiting solr or lucene api?

*Use case*: I have a few dynamic fields which may or may not be populated
everyday depending on certain conditions. I also do faceting and some
custom processing on these fields (using custom solr components). I want to
be able to plot the per field size of an index in realtime so that I can
try to identify the trend between fields & latencies.

Thanks a lot in advance!
Nitin

Re: Field Size per document in Solr

[Upayavira <uv...@odoko.co.uk>]

The field is not stored in a discrete place, rather it is mixed up with
all other field/document data. Therefore, I would suggest that
attempting to discern the disk space consumed by a single field would be
a futile endeavour.

Upayavira

On Tue, Jan 5, 2016, at 12:04 PM, KNitin wrote:
> I want to get the field size (in kb or mb) as is It is stored on disk.
> That
> approach might not give that info.
> 
> On Monday, January 4, 2016, Upayavira <uv...@odoko.co.uk> wrote:
> 
> >
> > Solr does store the term positions, but you won't find it easy to
> > extract them, as they are stored against terms not fields.
> >
> > Your best bet is to index field lengths into Solr alongside the field
> > values. You could use an UpdateProcessor to do this if you want to do it
> > in Solr.
> >
> > Upayavira
> >
> > On Tue, Jan 5, 2016, at 12:39 AM, KNitin wrote:
> > > Hi,
> > >
> > >  I want to get the size of individual fields per document (or per index)
> > >  in
> > > solrcloud. Is there a way to do this using exiting solr or lucene api?
> > >
> > > *Use case*: I have a few dynamic fields which may or may not be populated
> > > everyday depending on certain conditions. I also do faceting and some
> > > custom processing on these fields (using custom solr components). I want
> > > to
> > > be able to plot the per field size of an index in realtime so that I can
> > > try to identify the trend between fields & latencies.
> > >
> > > Thanks a lot in advance!
> > > Nitin
> >

Re: Field Size per document in Solr

[KNitin <ni...@gmail.com>]

I want to get the field size (in kb or mb) as is It is stored on disk. That
approach might not give that info.

On Monday, January 4, 2016, Upayavira <uv...@odoko.co.uk> wrote:

>
> Solr does store the term positions, but you won't find it easy to
> extract them, as they are stored against terms not fields.
>
> Your best bet is to index field lengths into Solr alongside the field
> values. You could use an UpdateProcessor to do this if you want to do it
> in Solr.
>
> Upayavira
>
> On Tue, Jan 5, 2016, at 12:39 AM, KNitin wrote:
> > Hi,
> >
> >  I want to get the size of individual fields per document (or per index)
> >  in
> > solrcloud. Is there a way to do this using exiting solr or lucene api?
> >
> > *Use case*: I have a few dynamic fields which may or may not be populated
> > everyday depending on certain conditions. I also do faceting and some
> > custom processing on these fields (using custom solr components). I want
> > to
> > be able to plot the per field size of an index in realtime so that I can
> > try to identify the trend between fields & latencies.
> >
> > Thanks a lot in advance!
> > Nitin
>

Re: Field Size per document in Solr

[Upayavira <uv...@odoko.co.uk>]

Solr does store the term positions, but you won't find it easy to
extract them, as they are stored against terms not fields.

Your best bet is to index field lengths into Solr alongside the field
values. You could use an UpdateProcessor to do this if you want to do it
in Solr.

Upayavira 

On Tue, Jan 5, 2016, at 12:39 AM, KNitin wrote:
> Hi,
> 
>  I want to get the size of individual fields per document (or per index)
>  in
> solrcloud. Is there a way to do this using exiting solr or lucene api?
> 
> *Use case*: I have a few dynamic fields which may or may not be populated
> everyday depending on certain conditions. I also do faceting and some
> custom processing on these fields (using custom solr components). I want
> to
> be able to plot the per field size of an index in realtime so that I can
> try to identify the trend between fields & latencies.
> 
> Thanks a lot in advance!
> Nitin