You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by John Omernik <jo...@omernik.com> on 2016/05/31 19:49:01 UTC
Profiles Gone in Web UI: The great profile heist
I am scratching my head at this one... I made some minor changes to my
drill-env.sh to enable gclogging, and was using the profiles in the webui
just fine. Due to some previously mentioned issues, I've had to restart
drill bits due to GC issues etc.
Now, while my profiles directory still exists, and my drill-override.conf
has not been changed, no profiles now show up in the webui, even after
drillbit restarts, and running more queries... The profiles are still being
created (I can see them being added to the same profiles directory) just
nothing shows up in the Web UI...
What could be happening here?
*scratching my head
Re: Profiles Gone in Web UI: The great profile heist
Posted by John Omernik <jo...@omernik.com>.
It's in mapr fs via nfs.
On Tuesday, May 31, 2016, Abdel Hakim Deneche <ad...@maprtech.com> wrote:
> are you storing the profiles in a local folder or in nfs ?
>
> On Tue, May 31, 2016 at 12:49 PM, John Omernik <john@omernik.com
> <javascript:;>> wrote:
>
> > I am scratching my head at this one... I made some minor changes to my
> > drill-env.sh to enable gclogging, and was using the profiles in the webui
> > just fine. Due to some previously mentioned issues, I've had to restart
> > drill bits due to GC issues etc.
> >
> > Now, while my profiles directory still exists, and my drill-override.conf
> > has not been changed, no profiles now show up in the webui, even after
> > drillbit restarts, and running more queries... The profiles are still
> being
> > created (I can see them being added to the same profiles directory) just
> > nothing shows up in the Web UI...
> >
> > What could be happening here?
> >
> > *scratching my head
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
--
Sent from my iThing
Re: Profiles Gone in Web UI: The great profile heist
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
are you storing the profiles in a local folder or in nfs ?
On Tue, May 31, 2016 at 12:49 PM, John Omernik <jo...@omernik.com> wrote:
> I am scratching my head at this one... I made some minor changes to my
> drill-env.sh to enable gclogging, and was using the profiles in the webui
> just fine. Due to some previously mentioned issues, I've had to restart
> drill bits due to GC issues etc.
>
> Now, while my profiles directory still exists, and my drill-override.conf
> has not been changed, no profiles now show up in the webui, even after
> drillbit restarts, and running more queries... The profiles are still being
> created (I can see them being added to the same profiles directory) just
> nothing shows up in the Web UI...
>
> What could be happening here?
>
> *scratching my head
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Profiles Gone in Web UI: The great profile heist
Posted by John Omernik <jo...@omernik.com>.
Thank you, I +1ed the JIRA, and added my user list comments. This should be
easily reproducible and fixable.
On Wed, Jun 1, 2016 at 11:17 AM, Krystal Nguyen <kn...@maprtech.com>
wrote:
> Looks like a jira was opened a while back (status is unresolved) regarding
> similar issue you are seeing:
> https://issues.apache.org/jira/browse/DRILL-2593
>
> On Wed, Jun 1, 2016 at 5:52 AM, John Omernik <jo...@omernik.com> wrote:
>
> > Abdel and Jacques -
> >
> > It is absolutely a bad profile, however, Jacques, it was not a empty file
> > profile, instead, it was a profile where the write didn't complete
> (likely
> > due to my bad cluster state post).
> >
> > I was able to reproduce it synthetically like this, say you had a profile
> > that ended similar to this:
> >
> > ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> > 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
> >
> > (The ... represents the part I removed for brevity in this email, the
> rest
> > of the profile is below)
> >
> > We can reproduce the UI profiles issue by removing ,"user":"adm"}
> >
> > so it ends like this:
> >
> > ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> > 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}]
> >
> >
> > Let me know if you know of a JIRA, otherwise I'll open one, this is a
> pain
> > for users, because you have to go through an validate the json on every
> > profile to find the one that is breaking your UI.
> >
> > John
> >
> >
> >
> >
> >
> > Full "Working" Profile:
> >
> > {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> > "start":1464730828036,"end":1464730828131,"query":"alter session set
> > `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> > zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> > dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> > fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> > state":3,"minorFragmentId":0,"operatorProfile":[{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> > peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> > peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> > 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> > 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> > maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> > ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
> >
> >
> > Full "Broken" Profile:
> >
> > {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> > "start":1464730828036,"end":1464730828131,"query":"alter session set
> > `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> > zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> > dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> > fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> > state":3,"minorFragmentId":0,"operatorProfile":[{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> > peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> > inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> > operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> > peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> > 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> > 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> > maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> > ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> > 1464730828126,"lastProgress":1464730828126}]}]
> >
> >
> > On Wed, Jun 1, 2016 at 6:22 AM, John Omernik <jo...@omernik.com> wrote:
> >
> > > I will walk through and look for the corrupt profile. This could be
> the
> > > result of my "cluster in a bad state problem" (actually, it IS the
> result
> > > of that, I have Ben doing testing to recreate the issue). Is there a
> JIRA
> > > for this ? I agree the UI should handle this.
> > >
> > >
> > > On Tuesday, May 31, 2016, Jacques Nadeau <ja...@dremio.com> wrote:
> > >
> > >> Odds are one of your profiles is corrupt. Last I checked, there was a
> > bug
> > >> that a corrupt profile would cause no profiles to show up in list.
> Check
> > >> if
> > >> any of your profiles are zero bytes and delete them.
> > >>
> > >> The UI should really handle this.
> > >>
> > >> --
> > >> Jacques Nadeau
> > >> CTO and Co-Founder, Dremio
> > >>
> > >> On Tue, May 31, 2016 at 12:49 PM, John Omernik <jo...@omernik.com>
> > wrote:
> > >>
> > >> > I am scratching my head at this one... I made some minor changes to
> my
> > >> > drill-env.sh to enable gclogging, and was using the profiles in the
> > >> webui
> > >> > just fine. Due to some previously mentioned issues, I've had to
> > restart
> > >> > drill bits due to GC issues etc.
> > >> >
> > >> > Now, while my profiles directory still exists, and my
> > >> drill-override.conf
> > >> > has not been changed, no profiles now show up in the webui, even
> after
> > >> > drillbit restarts, and running more queries... The profiles are
> still
> > >> being
> > >> > created (I can see them being added to the same profiles directory)
> > just
> > >> > nothing shows up in the Web UI...
> > >> >
> > >> > What could be happening here?
> > >> >
> > >> > *scratching my head
> > >> >
> > >>
> > >
> > >
> > > --
> > > Sent from my iThing
> > >
> >
>
Re: Profiles Gone in Web UI: The great profile heist
Posted by Krystal Nguyen <kn...@maprtech.com>.
Looks like a jira was opened a while back (status is unresolved) regarding
similar issue you are seeing:
https://issues.apache.org/jira/browse/DRILL-2593
On Wed, Jun 1, 2016 at 5:52 AM, John Omernik <jo...@omernik.com> wrote:
> Abdel and Jacques -
>
> It is absolutely a bad profile, however, Jacques, it was not a empty file
> profile, instead, it was a profile where the write didn't complete (likely
> due to my bad cluster state post).
>
> I was able to reproduce it synthetically like this, say you had a profile
> that ended similar to this:
>
> ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
>
> (The ... represents the part I removed for brevity in this email, the rest
> of the profile is below)
>
> We can reproduce the UI profiles issue by removing ,"user":"adm"}
>
> so it ends like this:
>
> ..."endpoint":{"address":"zeta01.mydomain.net","userPort":
> 20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> 1464730828126,"lastProgress":1464730828126}]}]
>
>
> Let me know if you know of a JIRA, otherwise I'll open one, this is a pain
> for users, because you have to go through an validate the json on every
> profile to find the one that is breaking your UI.
>
> John
>
>
>
>
>
> Full "Working" Profile:
>
> {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> "start":1464730828036,"end":1464730828131,"query":"alter session set
> `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> state":3,"minorFragmentId":0,"operatorProfile":[{"
> inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> 1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
>
>
> Full "Broken" Profile:
>
> {"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
> "start":1464730828036,"end":1464730828131,"query":"alter session set
> `store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
> zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
> dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
> fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
> state":3,"minorFragmentId":0,"operatorProfile":[{"
> inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
> peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
> inputProfile":[{"records":1,"batches":1,"schemas":1}],"
> operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
> peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
> 0,"longValue":59}],"waitNanos":1936593}],"startTime":
> 1464730828114,"endTime":1464730828123,"memoryUsed":0,"
> maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
> ","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
> 1464730828126,"lastProgress":1464730828126}]}]
>
>
> On Wed, Jun 1, 2016 at 6:22 AM, John Omernik <jo...@omernik.com> wrote:
>
> > I will walk through and look for the corrupt profile. This could be the
> > result of my "cluster in a bad state problem" (actually, it IS the result
> > of that, I have Ben doing testing to recreate the issue). Is there a JIRA
> > for this ? I agree the UI should handle this.
> >
> >
> > On Tuesday, May 31, 2016, Jacques Nadeau <ja...@dremio.com> wrote:
> >
> >> Odds are one of your profiles is corrupt. Last I checked, there was a
> bug
> >> that a corrupt profile would cause no profiles to show up in list. Check
> >> if
> >> any of your profiles are zero bytes and delete them.
> >>
> >> The UI should really handle this.
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Tue, May 31, 2016 at 12:49 PM, John Omernik <jo...@omernik.com>
> wrote:
> >>
> >> > I am scratching my head at this one... I made some minor changes to my
> >> > drill-env.sh to enable gclogging, and was using the profiles in the
> >> webui
> >> > just fine. Due to some previously mentioned issues, I've had to
> restart
> >> > drill bits due to GC issues etc.
> >> >
> >> > Now, while my profiles directory still exists, and my
> >> drill-override.conf
> >> > has not been changed, no profiles now show up in the webui, even after
> >> > drillbit restarts, and running more queries... The profiles are still
> >> being
> >> > created (I can see them being added to the same profiles directory)
> just
> >> > nothing shows up in the Web UI...
> >> >
> >> > What could be happening here?
> >> >
> >> > *scratching my head
> >> >
> >>
> >
> >
> > --
> > Sent from my iThing
> >
>
Re: Profiles Gone in Web UI: The great profile heist
Posted by John Omernik <jo...@omernik.com>.
Abdel and Jacques -
It is absolutely a bad profile, however, Jacques, it was not a empty file
profile, instead, it was a profile where the write didn't complete (likely
due to my bad cluster state post).
I was able to reproduce it synthetically like this, say you had a profile
that ended similar to this:
..."endpoint":{"address":"zeta01.mydomain.net","userPort":
20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
(The ... represents the part I removed for brevity in this email, the rest
of the profile is below)
We can reproduce the UI profiles issue by removing ,"user":"adm"}
so it ends like this:
..."endpoint":{"address":"zeta01.mydomain.net","userPort":
20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
1464730828126,"lastProgress":1464730828126}]}]
Let me know if you know of a JIRA, otherwise I'll open one, this is a pain
for users, because you have to go through an validate the json on every
profile to find the one that is breaking your UI.
John
Full "Working" Profile:
{"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
"start":1464730828036,"end":1464730828131,"query":"alter session set
`store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
state":3,"minorFragmentId":0,"operatorProfile":[{"
inputProfile":[{"records":1,"batches":1,"schemas":1}],"
operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
inputProfile":[{"records":1,"batches":1,"schemas":1}],"
operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
0,"longValue":59}],"waitNanos":1936593}],"startTime":
1464730828114,"endTime":1464730828123,"memoryUsed":0,"
maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
1464730828126,"lastProgress":1464730828126}]}],"user":"adm"}
Full "Broken" Profile:
{"id":{"part1":2932401029014293687,"part2":6412631480783062510},"type":1,
"start":1464730828036,"end":1464730828131,"query":"alter session set
`store.parquet.enable_dictionary_encoding` = true","foreman":{"address":"
zeta01.mydomain.net","userPort":20001,"controlPort":20002,"
dataPort":20003},"state":2,"totalFragments":1,"finishedFragments":0,"
fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"
state":3,"minorFragmentId":0,"operatorProfile":[{"
inputProfile":[{"records":1,"batches":1,"schemas":1}],"
operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":1020862,"
peakLocalMemoryAllocated":4530432,"waitNanos":0},{"
inputProfile":[{"records":1,"batches":1,"schemas":1}],"
operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":154675,"
peakLocalMemoryAllocated":4530176,"metric":[{"metricId":
0,"longValue":59}],"waitNanos":1936593}],"startTime":
1464730828114,"endTime":1464730828123,"memoryUsed":0,"
maxMemoryUsed":9060608,"endpoint":{"address":"zeta01.mydomain.net
","userPort":20001,"controlPort":20002,"dataPort":20003},"lastUpdate":
1464730828126,"lastProgress":1464730828126}]}]
On Wed, Jun 1, 2016 at 6:22 AM, John Omernik <jo...@omernik.com> wrote:
> I will walk through and look for the corrupt profile. This could be the
> result of my "cluster in a bad state problem" (actually, it IS the result
> of that, I have Ben doing testing to recreate the issue). Is there a JIRA
> for this ? I agree the UI should handle this.
>
>
> On Tuesday, May 31, 2016, Jacques Nadeau <ja...@dremio.com> wrote:
>
>> Odds are one of your profiles is corrupt. Last I checked, there was a bug
>> that a corrupt profile would cause no profiles to show up in list. Check
>> if
>> any of your profiles are zero bytes and delete them.
>>
>> The UI should really handle this.
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Tue, May 31, 2016 at 12:49 PM, John Omernik <jo...@omernik.com> wrote:
>>
>> > I am scratching my head at this one... I made some minor changes to my
>> > drill-env.sh to enable gclogging, and was using the profiles in the
>> webui
>> > just fine. Due to some previously mentioned issues, I've had to restart
>> > drill bits due to GC issues etc.
>> >
>> > Now, while my profiles directory still exists, and my
>> drill-override.conf
>> > has not been changed, no profiles now show up in the webui, even after
>> > drillbit restarts, and running more queries... The profiles are still
>> being
>> > created (I can see them being added to the same profiles directory) just
>> > nothing shows up in the Web UI...
>> >
>> > What could be happening here?
>> >
>> > *scratching my head
>> >
>>
>
>
> --
> Sent from my iThing
>
Re: Profiles Gone in Web UI: The great profile heist
Posted by John Omernik <jo...@omernik.com>.
I will walk through and look for the corrupt profile. This could be the
result of my "cluster in a bad state problem" (actually, it IS the result
of that, I have Ben doing testing to recreate the issue). Is there a JIRA
for this ? I agree the UI should handle this.
On Tuesday, May 31, 2016, Jacques Nadeau <ja...@dremio.com> wrote:
> Odds are one of your profiles is corrupt. Last I checked, there was a bug
> that a corrupt profile would cause no profiles to show up in list. Check if
> any of your profiles are zero bytes and delete them.
>
> The UI should really handle this.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, May 31, 2016 at 12:49 PM, John Omernik <john@omernik.com
> <javascript:;>> wrote:
>
> > I am scratching my head at this one... I made some minor changes to my
> > drill-env.sh to enable gclogging, and was using the profiles in the webui
> > just fine. Due to some previously mentioned issues, I've had to restart
> > drill bits due to GC issues etc.
> >
> > Now, while my profiles directory still exists, and my drill-override.conf
> > has not been changed, no profiles now show up in the webui, even after
> > drillbit restarts, and running more queries... The profiles are still
> being
> > created (I can see them being added to the same profiles directory) just
> > nothing shows up in the Web UI...
> >
> > What could be happening here?
> >
> > *scratching my head
> >
>
--
Sent from my iThing
Re: Profiles Gone in Web UI: The great profile heist
Posted by Jacques Nadeau <ja...@dremio.com>.
Odds are one of your profiles is corrupt. Last I checked, there was a bug
that a corrupt profile would cause no profiles to show up in list. Check if
any of your profiles are zero bytes and delete them.
The UI should really handle this.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, May 31, 2016 at 12:49 PM, John Omernik <jo...@omernik.com> wrote:
> I am scratching my head at this one... I made some minor changes to my
> drill-env.sh to enable gclogging, and was using the profiles in the webui
> just fine. Due to some previously mentioned issues, I've had to restart
> drill bits due to GC issues etc.
>
> Now, while my profiles directory still exists, and my drill-override.conf
> has not been changed, no profiles now show up in the webui, even after
> drillbit restarts, and running more queries... The profiles are still being
> created (I can see them being added to the same profiles directory) just
> nothing shows up in the Web UI...
>
> What could be happening here?
>
> *scratching my head
>