You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2015/02/20 21:24:12 UTC

[jira] [Created] (PIG-4427) Log a warning if the PARALLEL specified is not a prime number

Rohini Palaniswamy created PIG-4427:
---------------------------------------

             Summary: Log a warning if the PARALLEL specified is not a prime number
                 Key: PIG-4427
                 URL: https://issues.apache.org/jira/browse/PIG-4427
             Project: Pig
          Issue Type: Improvement
            Reporter: Rohini Palaniswamy


  Most of the time users specify default_parallel or PARALLEL in multiples of 10. This causes data skew and is not that effective. For eg: Had a user specify 1000 and all records went into 1 reducer, before it was changed to 999. But in some cases where user wants exact number of output files a non-prime number is desired. We should log a warning message if we see a non-prime number, so that it at least makes the user re-look and change config if it was not intentional and was done for sake of ease.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)