You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Prabhakar Bhosale <bh...@gmail.com> on 2022/07/08 10:11:50 UTC
Query performance
Hi Team,
We are querying the compressed json files. These files have simple json
records (1 record / row, no complex json).
Total files - 287
Average file size - 17,100 KB
Drill version - 1.20.1
Drill mode - embedded
The query text is
SELECT col1, col2, col3 from hugedata.TestTable
WHERE ID='01U00033' and SECTOR='01'
AND TO_DATE(exam_Date,'yyyy-MM-dd') >= TO_DATE('2018-01-01','yyyy-MM-dd')
AND TO_DATE(exam_Date,'yyyy-MM-dd') <= TO_DATE('2018-05-30','yyyy-MM-dd')
AND FLG <> 'Y' ORDER BY exam_Date,col1, col2 ASC
The query took 35 mins to give output. Below is the profile snapshot.
Please advise on how query performance can be improved
Fragment Profiles
Overview
<http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#fragment-overview>
Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast
EndMin RuntimeAvg RuntimeMax Runtime% BusyLast UpdateLast ProgressMax Peak
Memory
00-xx-xx 1 / 1 5.774s 5.774s 35m35s 35m35s 35m29s 35m29s 35m29s 86.59%
2.570s 35m28s 9MB
01-xx-xx 1 / 1 5.808s 5.808s 35m35s 35m35s 35m29s 35m29s 35m29s 0.15% 2.570s
34m43s 2MB
02-xx-xx 3 / 3 5.810s 5.818s 35m35s 35m35s 35m29s 35m29s 35m29s 99.71%
2.570s 2.571s 7MB
Operator ProfilesShow/Hide Estimated Rows
Overview
<http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#operator-overview>
WARNING: Query result was automatically limited to 1,000 rows
Operator IDTypeAvg Setup TimeMax Setup TimeAvg Process TimeMax Process TimeMin
Wait TimeAvg Wait TimeMax Wait Time% Fragment Time% Query TimeRowsAvg Peak
MemoryMax Peak Memory
00-xx-00 SCREEN 0.000s 0.000s 0.039s 0.039s 0.001s 0.001s 0.001s 54.33%
0.00% 0 - -
00-xx-01 PROJECT 0.003s 0.003s 0.001s 0.001s 0.000s 0.000s 0.000s 0.91%
0.00% 0 1MB 1MB
00-xx-02 SELECTION_VECTOR_REMOVER 0.000s 0.000s 0.001s 0.001s 0.000s 0.000s
0.000s 1.74% 0.00% 0 1MB 1MB
00-xx-03 LIMIT 0.000s 0.000s 0.001s 0.001s 0.000s 0.000s 0.000s 1.92% 0.00%
0 1MB 1MB
00-xx-04 LIMIT 0.001s 0.001s 0.000s 0.000s 0.000s 0.000s 0.000s 0.37% 0.00%
0 1MB 1MB
00-xx-05 SELECTION_VECTOR_REMOVER 0.008s 0.008s 0.002s 0.002s 0.000s 0.000s
0.000s 2.43% 0.00% 0 1MB 1MB
00-xx-06 TOP_N_SORT 0.000s 0.000s 0.024s 0.024s 0.000s 0.000s 0.000s 33.84%
0.00% 0 1MB 1MB
00-xx-07 PROJECT 0.004s 0.004s 0.002s 0.002s 0.000s 0.000s 0.000s 2.37%
0.00% 0 1MB 1MB
00-xx-08 UNORDERED_RECEIVER 0.000s 0.000s 0.001s 0.001s 0.009s 0.009s 0.009s
2.10% 0.00% 0 - -
01-xx-00 HASH_PARTITION_SENDER 0.299s 0.299s 0.009s 0.009s 0.103s 0.103s
0.103s 12.93% 0.00% 2 202KB 202KB
01-xx-01 UNORDERED_RECEIVER 0.000s 0.000s 0.060s 0.060s 44.424s 44.424s
44.424s 87.07% 0.00% 2 4KB 4KB
02-xx-00 JSON_SUB_SCAN 0.000s 0.000s 17m37s schedule35m15s 0.071s 3.052s
schedule6.257s 99.70% 99.70% 264,412,354 1MB 2MB
02-xx-01 PROJECT 0.070s 0.081s 0.009s 0.013s 0.000s 0.000s 0.000s 0.00%
0.00% 2 1MB 1MB
02-xx-02 PROJECT 0.040s 0.044s 0.020s 0.042s 0.000s 0.000s 0.000s 0.00%
0.00% 2 1MB 1MB
02-xx-03 SELECTION_VECTOR_REMOVER 0.010s 0.015s 1.151s 1.166s 0.000s 0.000s
0.000s 0.05% 0.05% 2 1MB 1MB
02-xx-04 FILTER 0.937s 0.956s 5.084s 5.190s 0.000s 0.000s 0.000s 0.24% 0.24%
264,412,352 2MB 2MB
Regards
Prabhakar
Re: Query performance
Posted by luoc <lu...@apache.org>.
Good job. Any new feedback is welcome.
> On Jul 11, 2022, at 22:53, Prabhakar Bhosale <bh...@gmail.com> wrote:
>
> Hi Luoc,
> Thanks for your reply. I have tried the same data in parquet format and
> results are really surprising. The query on that huge data which took 48
> mins on json files , returned me the results in 11 sec in paquet format.
> I am trying a few more scenarios. Thanks
>
> regards
> Prabhakar
>
>> On Mon, Jul 11, 2022 at 3:47 PM luoc <lu...@apache.org> wrote:
>>
>> Hello Prabhakar,
>>
>> For the first question, I recommend you try the performance tool like the
>> "nmon" to check the cost of the machine's CPU.
>>
>> I think you can get the results as how many nodes, once you scale the
>> number of drillbits to 2 or 3, because you can look at the velocity changes.
>>
>> You can try to reduce the file size to 512 MB, but we can't make sure
>> there's a big impact, but we can reduce the cost of the JVM’ GC.
>>
>> Parquet is good for data analysis but not for data queries, because the
>> Parquet is the columnar storage.
>>
>> If you like using the "select * from table1", then the Parquet is not a
>> good idea.
>>
>> If you like using the `select max(f1), min(f2) from table1` query text,
>> then the Parquet is a great solution.
>>
>> Next, you may have to choose between CTAS or direct query costs.
>>
>>
>> - luoc
>>
>>> 2022年7月10日 下午1:22，Prabhakar Bhosale <bh...@gmail.com> 写道：
>>>
>>> Dear Luoc,
>>> Thanks for the insights. This is just a week's data. Production will
>> have 15 times more data. So inline with it, I have following queries
>>>
>>> 1. Is there any template or calculator which will help me size the
>> production server (CPU, memory and IO) based on size of data?
>>> 2. For such a huge size of data, what are the best practices to be
>> followed to store and retrieve the data?
>>> 2. What should be the optimal size of the file? Currently the
>> uncompressed size of the file is 2GB. So how do we balance between the
>> number of files and file size?
>>> 3. Do you think the parquet format will perform better than JSON?
>>> 4. Is there any way in Drill to detect the "File Create" event and then
>> convert JSON to parquet using CTAS?
>>>
>>> Thanks And Regards
>>> Prabhakar
>>>
>>> On Sat, Jul 9, 2022 at 8:41 PM luoc <luoc@apache.org <mailto:
>> luoc@apache.org>> wrote:
>>>
>>> Hello Prabhakar,
>>>
>>> I will present my check process and hope to give you advice :
>>>
>>> 1. I imported the file in your attachment using the `View` button on the
>> right side of the `Profile` page.
>>>
>>> 2. The fragment profile record that: the major fragment (02-xx-xx) cost
>> about 45min+.
>>>
>>> 3. The 02-xx-xx phase used 3 parallels, and the json-scan (JSON Reader)
>> cost the most of the time.
>>>
>>> 4. Each minor fragment reads nearly 0.12 billion records. Killer !
>>>
>>> As a result, three JSON readers read a total of 338,398,798 records.
>>>
>>> And then, your JSON files are in GZ compression format, with a total of
>> 297, meaning the Drill need a lot of CPUs to decompress.
>>>
>>> Simply understand that your hardware resources are bottlenecks and can't
>> use faster to query large-scale records, and recommend scale-out nodes and
>> using distributed clusters.
>>>
>>> - luoc
>>>
>>>
>>>
>>>
>>>
>>>> 2022年7月9日 上午1:01，Prabhakar Bhosale <bhosale.p.v@gmail.com <mailto:
>> bhosale.p.v@gmail.com>> 写道：
>>>>
>>>> the
>>>
>>
>>
>
Re: Query performance
Posted by Prabhakar Bhosale <bh...@gmail.com>.
Hi Luoc,
Thanks for your reply. I have tried the same data in parquet format and
results are really surprising. The query on that huge data which took 48
mins on json files , returned me the results in 11 sec in paquet format.
I am trying a few more scenarios. Thanks
regards
Prabhakar
On Mon, Jul 11, 2022 at 3:47 PM luoc <lu...@apache.org> wrote:
> Hello Prabhakar,
>
> For the first question, I recommend you try the performance tool like the
> "nmon" to check the cost of the machine's CPU.
>
> I think you can get the results as how many nodes, once you scale the
> number of drillbits to 2 or 3, because you can look at the velocity changes.
>
> You can try to reduce the file size to 512 MB, but we can't make sure
> there's a big impact, but we can reduce the cost of the JVM’ GC.
>
> Parquet is good for data analysis but not for data queries, because the
> Parquet is the columnar storage.
>
> If you like using the "select * from table1", then the Parquet is not a
> good idea.
>
> If you like using the `select max(f1), min(f2) from table1` query text,
> then the Parquet is a great solution.
>
> Next, you may have to choose between CTAS or direct query costs.
>
>
> - luoc
>
> > 2022年7月10日 下午1:22，Prabhakar Bhosale <bh...@gmail.com> 写道：
> >
> > Dear Luoc,
> > Thanks for the insights. This is just a week's data. Production will
> have 15 times more data. So inline with it, I have following queries
> >
> > 1. Is there any template or calculator which will help me size the
> production server (CPU, memory and IO) based on size of data?
> > 2. For such a huge size of data, what are the best practices to be
> followed to store and retrieve the data?
> > 2. What should be the optimal size of the file? Currently the
> uncompressed size of the file is 2GB. So how do we balance between the
> number of files and file size?
> > 3. Do you think the parquet format will perform better than JSON?
> > 4. Is there any way in Drill to detect the "File Create" event and then
> convert JSON to parquet using CTAS?
> >
> > Thanks And Regards
> > Prabhakar
> >
> > On Sat, Jul 9, 2022 at 8:41 PM luoc <luoc@apache.org <mailto:
> luoc@apache.org>> wrote:
> >
> > Hello Prabhakar,
> >
> > I will present my check process and hope to give you advice :
> >
> > 1. I imported the file in your attachment using the `View` button on the
> right side of the `Profile` page.
> >
> > 2. The fragment profile record that: the major fragment (02-xx-xx) cost
> about 45min+.
> >
> > 3. The 02-xx-xx phase used 3 parallels, and the json-scan (JSON Reader)
> cost the most of the time.
> >
> > 4. Each minor fragment reads nearly 0.12 billion records. Killer !
> >
> > As a result, three JSON readers read a total of 338,398,798 records.
> >
> > And then, your JSON files are in GZ compression format, with a total of
> 297, meaning the Drill need a lot of CPUs to decompress.
> >
> > Simply understand that your hardware resources are bottlenecks and can't
> use faster to query large-scale records, and recommend scale-out nodes and
> using distributed clusters.
> >
> > - luoc
> >
> >
> >
> >
> >
> >> 2022年7月9日 上午1:01，Prabhakar Bhosale <bhosale.p.v@gmail.com <mailto:
> bhosale.p.v@gmail.com>> 写道：
> >>
> >> the
> >
>
>
Re: Query performance
Posted by luoc <lu...@apache.org>.
Hello Prabhakar,
For the first question, I recommend you try the performance tool like the "nmon" to check the cost of the machine's CPU.
I think you can get the results as how many nodes, once you scale the number of drillbits to 2 or 3, because you can look at the velocity changes.
You can try to reduce the file size to 512 MB, but we can't make sure there's a big impact, but we can reduce the cost of the JVM’ GC.
Parquet is good for data analysis but not for data queries, because the Parquet is the columnar storage.
If you like using the "select * from table1", then the Parquet is not a good idea.
If you like using the `select max(f1), min(f2) from table1` query text, then the Parquet is a great solution.
Next, you may have to choose between CTAS or direct query costs.
- luoc
> 2022年7月10日 下午1:22，Prabhakar Bhosale <bh...@gmail.com> 写道：
>
> Dear Luoc,
> Thanks for the insights. This is just a week's data. Production will have 15 times more data. So inline with it, I have following queries
>
> 1. Is there any template or calculator which will help me size the production server (CPU, memory and IO) based on size of data?
> 2. For such a huge size of data, what are the best practices to be followed to store and retrieve the data?
> 2. What should be the optimal size of the file? Currently the uncompressed size of the file is 2GB. So how do we balance between the number of files and file size?
> 3. Do you think the parquet format will perform better than JSON?
> 4. Is there any way in Drill to detect the "File Create" event and then convert JSON to parquet using CTAS?
>
> Thanks And Regards
> Prabhakar
>
> On Sat, Jul 9, 2022 at 8:41 PM luoc <luoc@apache.org <ma...@apache.org>> wrote:
>
> Hello Prabhakar,
>
> I will present my check process and hope to give you advice :
>
> 1. I imported the file in your attachment using the `View` button on the right side of the `Profile` page.
>
> 2. The fragment profile record that: the major fragment (02-xx-xx) cost about 45min+.
>
> 3. The 02-xx-xx phase used 3 parallels, and the json-scan (JSON Reader) cost the most of the time.
>
> 4. Each minor fragment reads nearly 0.12 billion records. Killer !
>
> As a result, three JSON readers read a total of 338,398,798 records.
>
> And then, your JSON files are in GZ compression format, with a total of 297, meaning the Drill need a lot of CPUs to decompress.
>
> Simply understand that your hardware resources are bottlenecks and can't use faster to query large-scale records, and recommend scale-out nodes and using distributed clusters.
>
> - luoc
>
>
>
>
>
>> 2022年7月9日 上午1:01，Prabhakar Bhosale <bhosale.p.v@gmail.com <ma...@gmail.com>> 写道：
>>
>> the
>
Re: Query performance
Posted by Prabhakar Bhosale <bh...@gmail.com>.
Dear Luoc,
Thanks for the insights. This is just a week's data. Production will have
15 times more data. So inline with it, I have following queries
1. Is there any template or calculator which will help me size the
production server (CPU, memory and IO) based on size of data?
2. For such a huge size of data, what are the best practices to be followed
to store and retrieve the data?
2. What should be the optimal size of the file? Currently the
uncompressed size of the file is 2GB. So how do we balance between the
number of files and file size?
3. Do you think the parquet format will perform better than JSON?
4. Is there any way in Drill to detect the "File Create" event and then
convert JSON to parquet using CTAS?
Thanks And Regards
Prabhakar
On Sat, Jul 9, 2022 at 8:41 PM luoc <lu...@apache.org> wrote:
>
> Hello Prabhakar,
>
> I will present my check process and hope to give you advice :
>
> 1. I imported the file in your attachment using the `View` button on the
> right side of the `Profile` page.
>
> 2. The fragment profile record that: the major fragment (02-xx-xx) cost
> about 45min+.
>
> 3. The 02-xx-xx phase used 3 parallels, and the json-scan (JSON Reader)
> cost the most of the time.
>
> 4. Each minor fragment reads nearly 0.12 billion records. Killer !
>
> As a result, three JSON readers read a total of 338,398,798 records.
>
> And then, your JSON files are in GZ compression format, with a total of
> 297, meaning the Drill need a lot of CPUs to decompress.
>
> Simply understand that your hardware resources are bottlenecks and can't
> use faster to query large-scale records, and recommend scale-out nodes and
> using distributed clusters.
>
> - luoc
>
>
>
> 2022年7月9日 上午1:01，Prabhakar Bhosale <bh...@gmail.com> 写道：
>
> the
>
>
>
Re: Query performance
Posted by luoc <lu...@apache.org>.
Hello Prabhakar,
I will present my check process and hope to give you advice :
1. I imported the file in your attachment using the `View` button on the right side of the `Profile` page.
2. The fragment profile record that: the major fragment (02-xx-xx) cost about 45min+.
3. The 02-xx-xx phase used 3 parallels, and the json-scan (JSON Reader) cost the most of the time.
4. Each minor fragment reads nearly 0.12 billion records. Killer !
As a result, three JSON readers read a total of 338,398,798 records.
And then, your JSON files are in GZ compression format, with a total of 297, meaning the Drill need a lot of CPUs to decompress.
Simply understand that your hardware resources are bottlenecks and can't use faster to query large-scale records, and recommend scale-out nodes and using distributed clusters.
- luoc
> 2022年7月9日 上午1:01，Prabhakar Bhosale <bh...@gmail.com> 写道：
>
> the
Re: Query performance
Posted by Prabhakar Bhosale <bh...@gmail.com>.
Hi Luoc,
I could not find View button. But instead I saw
"Json Profile". So I downloaded the same json and attaching here. Hope this
helps
On Fri, Jul 8, 2022 at 5:36 PM luoc <lu...@apache.org> wrote:
> Hello Prabhakar,
>
> Since the 1.20, the web UI provided a View button in the Profile page, It
> can be re-play the execution phase, better for the diagnostic.
>
> So, could you please post this profile json file? please add it as the
> attachment of email.
>
> > On Jul 8, 2022, at 18:12, Prabhakar Bhosale <bh...@gmail.com>
> wrote:
> >
> > Hi Team,
> > We are querying the compressed json files. These files have simple json
> > records (1 record / row, no complex json).
> >
> > Total files - 287
> > Average file size - 17,100 KB
> > Drill version - 1.20.1
> > Drill mode - embedded
> >
> > The query text is
> > SELECT col1, col2, col3 from hugedata.TestTable
> > WHERE ID='01U00033' and SECTOR='01'
> > AND TO_DATE(exam_Date,'yyyy-MM-dd') >= TO_DATE('2018-01-01','yyyy-MM-dd')
> > AND TO_DATE(exam_Date,'yyyy-MM-dd') <= TO_DATE('2018-05-30','yyyy-MM-dd')
> > AND FLG <> 'Y' ORDER BY exam_Date,col1, col2 ASC
> >
> >
> > The query took 35 mins to give output. Below is the profile snapshot.
> > Please advise on how query performance can be improved
> >
> > Fragment Profiles
> > Overview
> > <
> http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#fragment-overview
> >
> > Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast
> > EndMin RuntimeAvg RuntimeMax Runtime% BusyLast UpdateLast ProgressMax
> Peak
> > Memory
> > 00-xx-xx 1 / 1 5.774s 5.774s 35m35s 35m35s 35m29s 35m29s 35m29s 86.59%
> > 2.570s 35m28s 9MB
> > 01-xx-xx 1 / 1 5.808s 5.808s 35m35s 35m35s 35m29s 35m29s 35m29s 0.15%
> 2.570s
> > 34m43s 2MB
> > 02-xx-xx 3 / 3 5.810s 5.818s 35m35s 35m35s 35m29s 35m29s 35m29s 99.71%
> > 2.570s 2.571s 7MB
> >
> >
> > Operator ProfilesShow/Hide Estimated Rows
> > Overview
> > <
> http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#operator-overview
> >
> > WARNING: Query result was automatically limited to 1,000 rows
> > Operator IDTypeAvg Setup TimeMax Setup TimeAvg Process TimeMax Process
> TimeMin
> > Wait TimeAvg Wait TimeMax Wait Time% Fragment Time% Query TimeRowsAvg
> Peak
> > MemoryMax Peak Memory
> > 00-xx-00 SCREEN 0.000s 0.000s 0.039s 0.039s 0.001s 0.001s 0.001s 54.33%
> > 0.00% 0 - -
> > 00-xx-01 PROJECT 0.003s 0.003s 0.001s 0.001s 0.000s 0.000s 0.000s 0.91%
> > 0.00% 0 1MB 1MB
> > 00-xx-02 SELECTION_VECTOR_REMOVER 0.000s 0.000s 0.001s 0.001s 0.000s
> 0.000s
> > 0.000s 1.74% 0.00% 0 1MB 1MB
> > 00-xx-03 LIMIT 0.000s 0.000s 0.001s 0.001s 0.000s 0.000s 0.000s 1.92%
> 0.00%
> > 0 1MB 1MB
> > 00-xx-04 LIMIT 0.001s 0.001s 0.000s 0.000s 0.000s 0.000s 0.000s 0.37%
> 0.00%
> > 0 1MB 1MB
> > 00-xx-05 SELECTION_VECTOR_REMOVER 0.008s 0.008s 0.002s 0.002s 0.000s
> 0.000s
> > 0.000s 2.43% 0.00% 0 1MB 1MB
> > 00-xx-06 TOP_N_SORT 0.000s 0.000s 0.024s 0.024s 0.000s 0.000s 0.000s
> 33.84%
> > 0.00% 0 1MB 1MB
> > 00-xx-07 PROJECT 0.004s 0.004s 0.002s 0.002s 0.000s 0.000s 0.000s 2.37%
> > 0.00% 0 1MB 1MB
> > 00-xx-08 UNORDERED_RECEIVER 0.000s 0.000s 0.001s 0.001s 0.009s 0.009s
> 0.009s
> > 2.10% 0.00% 0 - -
> > 01-xx-00 HASH_PARTITION_SENDER 0.299s 0.299s 0.009s 0.009s 0.103s 0.103s
> > 0.103s 12.93% 0.00% 2 202KB 202KB
> > 01-xx-01 UNORDERED_RECEIVER 0.000s 0.000s 0.060s 0.060s 44.424s 44.424s
> > 44.424s 87.07% 0.00% 2 4KB 4KB
> > 02-xx-00 JSON_SUB_SCAN 0.000s 0.000s 17m37s schedule35m15s 0.071s 3.052s
> > schedule6.257s 99.70% 99.70% 264,412,354 1MB 2MB
> > 02-xx-01 PROJECT 0.070s 0.081s 0.009s 0.013s 0.000s 0.000s 0.000s 0.00%
> > 0.00% 2 1MB 1MB
> > 02-xx-02 PROJECT 0.040s 0.044s 0.020s 0.042s 0.000s 0.000s 0.000s 0.00%
> > 0.00% 2 1MB 1MB
> > 02-xx-03 SELECTION_VECTOR_REMOVER 0.010s 0.015s 1.151s 1.166s 0.000s
> 0.000s
> > 0.000s 0.05% 0.05% 2 1MB 1MB
> > 02-xx-04 FILTER 0.937s 0.956s 5.084s 5.190s 0.000s 0.000s 0.000s 0.24%
> 0.24%
> > 264,412,352 2MB 2MB
> > Regards
> > Prabhakar
> >
>
Re: Query performance
Posted by luoc <lu...@apache.org>.
Hello Prabhakar,
Since the 1.20, the web UI provided a View button in the Profile page, It can be re-play the execution phase, better for the diagnostic.
So, could you please post this profile json file? please add it as the attachment of email.
> On Jul 8, 2022, at 18:12, Prabhakar Bhosale <bh...@gmail.com> wrote:
>
> Hi Team,
> We are querying the compressed json files. These files have simple json
> records (1 record / row, no complex json).
>
> Total files - 287
> Average file size - 17,100 KB
> Drill version - 1.20.1
> Drill mode - embedded
>
> The query text is
> SELECT col1, col2, col3 from hugedata.TestTable
> WHERE ID='01U00033' and SECTOR='01'
> AND TO_DATE(exam_Date,'yyyy-MM-dd') >= TO_DATE('2018-01-01','yyyy-MM-dd')
> AND TO_DATE(exam_Date,'yyyy-MM-dd') <= TO_DATE('2018-05-30','yyyy-MM-dd')
> AND FLG <> 'Y' ORDER BY exam_Date,col1, col2 ASC
>
>
> The query took 35 mins to give output. Below is the profile snapshot.
> Please advise on how query performance can be improved
>
> Fragment Profiles
> Overview
> <http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#fragment-overview>
> Major FragmentMinor Fragments ReportingFirst StartLast StartFirst EndLast
> EndMin RuntimeAvg RuntimeMax Runtime% BusyLast UpdateLast ProgressMax Peak
> Memory
> 00-xx-xx 1 / 1 5.774s 5.774s 35m35s 35m35s 35m29s 35m29s 35m29s 86.59%
> 2.570s 35m28s 9MB
> 01-xx-xx 1 / 1 5.808s 5.808s 35m35s 35m35s 35m29s 35m29s 35m29s 0.15% 2.570s
> 34m43s 2MB
> 02-xx-xx 3 / 3 5.810s 5.818s 35m35s 35m35s 35m29s 35m29s 35m29s 99.71%
> 2.570s 2.571s 7MB
>
>
> Operator ProfilesShow/Hide Estimated Rows
> Overview
> <http://bl4ul296:8047/profiles/1d38154a-8544-03fd-9bf5-296aab01e1ec#operator-overview>
> WARNING: Query result was automatically limited to 1,000 rows
> Operator IDTypeAvg Setup TimeMax Setup TimeAvg Process TimeMax Process TimeMin
> Wait TimeAvg Wait TimeMax Wait Time% Fragment Time% Query TimeRowsAvg Peak
> MemoryMax Peak Memory
> 00-xx-00 SCREEN 0.000s 0.000s 0.039s 0.039s 0.001s 0.001s 0.001s 54.33%
> 0.00% 0 - -
> 00-xx-01 PROJECT 0.003s 0.003s 0.001s 0.001s 0.000s 0.000s 0.000s 0.91%
> 0.00% 0 1MB 1MB
> 00-xx-02 SELECTION_VECTOR_REMOVER 0.000s 0.000s 0.001s 0.001s 0.000s 0.000s
> 0.000s 1.74% 0.00% 0 1MB 1MB
> 00-xx-03 LIMIT 0.000s 0.000s 0.001s 0.001s 0.000s 0.000s 0.000s 1.92% 0.00%
> 0 1MB 1MB
> 00-xx-04 LIMIT 0.001s 0.001s 0.000s 0.000s 0.000s 0.000s 0.000s 0.37% 0.00%
> 0 1MB 1MB
> 00-xx-05 SELECTION_VECTOR_REMOVER 0.008s 0.008s 0.002s 0.002s 0.000s 0.000s
> 0.000s 2.43% 0.00% 0 1MB 1MB
> 00-xx-06 TOP_N_SORT 0.000s 0.000s 0.024s 0.024s 0.000s 0.000s 0.000s 33.84%
> 0.00% 0 1MB 1MB
> 00-xx-07 PROJECT 0.004s 0.004s 0.002s 0.002s 0.000s 0.000s 0.000s 2.37%
> 0.00% 0 1MB 1MB
> 00-xx-08 UNORDERED_RECEIVER 0.000s 0.000s 0.001s 0.001s 0.009s 0.009s 0.009s
> 2.10% 0.00% 0 - -
> 01-xx-00 HASH_PARTITION_SENDER 0.299s 0.299s 0.009s 0.009s 0.103s 0.103s
> 0.103s 12.93% 0.00% 2 202KB 202KB
> 01-xx-01 UNORDERED_RECEIVER 0.000s 0.000s 0.060s 0.060s 44.424s 44.424s
> 44.424s 87.07% 0.00% 2 4KB 4KB
> 02-xx-00 JSON_SUB_SCAN 0.000s 0.000s 17m37s schedule35m15s 0.071s 3.052s
> schedule6.257s 99.70% 99.70% 264,412,354 1MB 2MB
> 02-xx-01 PROJECT 0.070s 0.081s 0.009s 0.013s 0.000s 0.000s 0.000s 0.00%
> 0.00% 2 1MB 1MB
> 02-xx-02 PROJECT 0.040s 0.044s 0.020s 0.042s 0.000s 0.000s 0.000s 0.00%
> 0.00% 2 1MB 1MB
> 02-xx-03 SELECTION_VECTOR_REMOVER 0.010s 0.015s 1.151s 1.166s 0.000s 0.000s
> 0.000s 0.05% 0.05% 2 1MB 1MB
> 02-xx-04 FILTER 0.937s 0.956s 5.084s 5.190s 0.000s 0.000s 0.000s 0.24% 0.24%
> 264,412,352 2MB 2MB
> Regards
> Prabhakar
>