You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by 吴晓菊 <ch...@gmail.com> on 2018/07/30 14:03:30 UTC

Why percentile and distinct are not done in one job?

I noticed that in column analyzing, 2 jobs will run separately to calculate
percentiles and then distinct. Why not combine into one job since
HyperLogLog also supports merge?

Chrysan Wu
Phone：+86 17717640807

Re: Why percentile and distinct are not done in one job?

Posted by 吴晓菊 <ch...@gmail.com>.

I mean in AnalyzeColumnCommand.scala the first one to compute percentiles
and the second one to compute columnStats.


Chrysan Wu
吴晓菊
Phone：+86 17717640807


2018-07-30 23:28 GMT+08:00 Reynold Xin <rx...@databricks.com>:

> Which API are you talking about?
>
> On Mon, Jul 30, 2018 at 7:03 AM 吴晓菊 <ch...@gmail.com> wrote:
>
>> I noticed that in column analyzing, 2 jobs will run separately to
>> calculate percentiles and then distinct. Why not combine into one job since
>> HyperLogLog also supports merge?
>>
>> Chrysan Wu
>> Phone：+86 17717640807
>>
>>

Re: Why percentile and distinct are not done in one job?

Posted by Reynold Xin <rx...@databricks.com>.

Which API are you talking about?

On Mon, Jul 30, 2018 at 7:03 AM 吴晓菊 <ch...@gmail.com> wrote:

> I noticed that in column analyzing, 2 jobs will run separately to
> calculate percentiles and then distinct. Why not combine into one job since
> HyperLogLog also supports merge?
>
> Chrysan Wu
> Phone：+86 17717640807
>
>