Multiple segments with bin/pinot-admin.sh LaunchDataIngestionJob

[katneni ravikiran <ka...@gmail.com>]

Hi,

I am trying to find information on how segmentation works on the data
ingested using "bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile ".

I have used the basic table configuration with no indexes defined and the
segmentation section only has

"segmentsConfig" : { "replication" : "2", "schemaName" : "Customer" }

When I use this command to upload data of size 7GB, it is creating only a
single segment with 52 million rows.

Can any one help me in finding the configuration to specify the
segmentation column? or any other configuration by which I can control the
number of segments?

Thanks for your help in advance.

Ravikiran

Re: Multiple segments with bin/pinot-admin.sh LaunchDataIngestionJob

[Neha Pawar <ne...@gmail.com>]

Hi Ravikiran,

There is no config to control the number of segments. But you can control
them by splitting your input file into multiple files. The data ingestion
job will generate as many segments as the number of input files in your
input folder.
Added this question to FAQs:
https://docs.pinot.apache.org/basics/getting-started/frequent-questions#how-to-control-number-of-segments-generated

Thanks,
Neha Pawar

<pa...@illinois.edu>


On Thu, Jul 23, 2020 at 9:33 AM katneni ravikiran <
katneniravikiran@gmail.com> wrote:

> Hi,
>
> I am trying to find information on how segmentation works on the data
> ingested using "bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile ".
>
> I have used the basic table configuration with no indexes defined and the
> segmentation section only has
>
> "segmentsConfig" : { "replication" : "2", "schemaName" : "Customer" }
>
> When I use this command to upload data of size 7GB, it is creating only a
> single segment with 52 million rows.
>
> Can any one help me in finding the configuration to specify the
> segmentation column? or any other configuration by which I can control the
> number of segments?
>
> Thanks for your help in advance.
>
> Ravikiran
>