replicated join gets extra job

[Dexin Wang <wa...@gmail.com>]

Hi,

I'm running a job like this:

raw_large = LOAD 'lots_of_files' AS (...);
raw_filtered = FILTER raw_large BY ...;
large_table = FOREACH raw_filtered GENERATE f1, f2, f3,....;

joined_1 = JOIN large_table BY (key1) LEFT, config_table_1  BY (key2) USING
'replicated';
joined_2 = JOIN join1          BY (key3) LEFT, config_table_2  BY (key4)
USING 'replicated';
joined_3 = JOIN join2          BY (key5) LEFT, config_table_3  BY (key6)
USING 'replicated';
joined_4 = JOIN join4          BY (key7) LEFT, config_table_3  BY (key8)
USING 'replicated';

basically left join a large table with 4 relatively small tables using the
replicated join.

I see a first load job has 120 mapper tasks and no reducer, and this job
seems to be doing the load and filtering. And there is another job
following that has 26 mapper tasks that seem to be doing the joins.

Shouldn't there be only one job and the joins being done in the mapper
phase of the first job?

The 4 config tables (files) have these sizes respectively:

3MB
220kB
2kB
100kB

these are running on AWS EMR Pig 0.92 on xlarge instances which has 15GB
memory.

Thanks!

Re: replicated join gets extra job

[Pradeep Gollakota <pr...@gmail.com>]

Use the ILLUSTRATE or EXPLAIN keywords to look at the details of the
physical execution plan... from first glance it doesn't look like you'd
need a 2nd job to do the joins, but if you can post the output of
ILLUSTRATE/EXPLAIN, we can look into it.


On Mon, Nov 11, 2013 at 4:36 PM, Dexin Wang <wa...@gmail.com> wrote:

> Hi,
>
> I'm running a job like this:
>
> raw_large = LOAD 'lots_of_files' AS (...);
> raw_filtered = FILTER raw_large BY ...;
> large_table = FOREACH raw_filtered GENERATE f1, f2, f3,....;
>
> joined_1 = JOIN large_table BY (key1) LEFT, config_table_1  BY (key2) USING
> 'replicated';
> joined_2 = JOIN join1          BY (key3) LEFT, config_table_2  BY (key4)
> USING 'replicated';
> joined_3 = JOIN join2          BY (key5) LEFT, config_table_3  BY (key6)
> USING 'replicated';
> joined_4 = JOIN join4          BY (key7) LEFT, config_table_3  BY (key8)
> USING 'replicated';
>
> basically left join a large table with 4 relatively small tables using the
> replicated join.
>
> I see a first load job has 120 mapper tasks and no reducer, and this job
> seems to be doing the load and filtering. And there is another job
> following that has 26 mapper tasks that seem to be doing the joins.
>
> Shouldn't there be only one job and the joins being done in the mapper
> phase of the first job?
>
> The 4 config tables (files) have these sizes respectively:
>
> 3MB
> 220kB
> 2kB
> 100kB
>
> these are running on AWS EMR Pig 0.92 on xlarge instances which has 15GB
> memory.
>
> Thanks!
>