You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by sebb <se...@gmail.com> on 2023/01/04 12:03:48 UTC
Detecting errors in Buildbot
The www-site builds failed recently, however this was not obvious
because the failures don't appear to have been reported anywhere
outside the dashboard.
As I recall, the previous version of BuildBot would send an email for
broken (and mended) builds. This was very handy for detecting and
fixing errors.
Are there any plans to provide something similar for the current Buildbot?
Sebb
Re: Detecting errors in Buildbot
Posted by sebb <se...@gmail.com>.
On Thu, 5 Jan 2023 at 17:41, fluxo <cm...@apache.org> wrote:
>
>
> > On Jan 5, 2023, at 12:03 AM, sebb <se...@gmail.com> wrote:
> >
> > On Thu, 5 Jan 2023 at 01:10, fluxo <cml@apache.org <ma...@apache.org>> wrote:
> >>
> >>> On Jan 4, 2023, at 3:15 PM, sebb <se...@gmail.com> wrote:
> >>>
> >>> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus <cml@apache.org <ma...@apache.org>> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
> >>>>>
> >>>>> The www-site builds failed recently, however this was not obvious
> >>>>> because the failures don't appear to have been reported anywhere
> >>>>> outside the dashboard.
> >>>>>
> >>>>> As I recall, the previous version of BuildBot would send an email for
> >>>>> broken (and mended) builds. This was very handy for detecting and
> >>>>> fixing errors.
> >>>>>
> >>>>> Are there any plans to provide something similar for the current Buildbot?
> >>>>>
> >>>>
> >>>> The functionality is available. Here is an example configuration for setting up an email notifier.
> >>>
> >>> Thanks, AFAICT something similar is already set up here:
> >>>
> >>> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> >>>
> >>> However it does not seem to be working.
> >>
> >>
> >> I've opened INFRA-24056 to track this. The error is attached below. As a stopgap, we can add an extraRecipients field, because it's not correctly detecting the contributors, and is thus unable to generate a list of "interested users."
> >
> > I see.
> >
> >> What list makes most sense to see failures related to the pelican websites builds?
> >
> > I'm not sure if there is a single list that would be correct for all
> > those builds.
>
> We're creating a site-builds@infra list for this purpose, as currently the workaround requires all pelican build messages to go to a single address. People will be able to opt-in (subscribe) for build output, but it will be noisy, yes.
I see.
It looks like the list is private; not sure that is necessary.
Also it looks like subscription is moderated, which is surely unnecessary?
>
>
>
> > However it would probably be OK to use the public list site-dev@a.o <ma...@a.o>
> > which was used in the past for bb builds. See for example:
> >
> > https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p
> >
> > If this proves too noisy it might be necessary to redirect some build
> > mails elsewhere.
> >
> > There also needs to be a way to ensure that script failures such as
> > the one below are brought to the attention of builds and/or infra.
> >
> >> 2022-12-31 19:22:37+0000 [-] INVALID EMAIL: asfinfra
> >> 2022-12-31 19:22:37+0000 [-] sending mail (653 bytes) to []
> >> 2022-12-31 19:22:37+0000 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7fa8f003ce80>
> >> 2022-12-31 19:22:37+0000 [ESMTPSender,client] Got exception when handling reporter events
> >> Traceback (most recent call last):
> >> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 1658, in _inlineCallbacks
> >> cast(Failure, result).throwExceptionIntoGenerator, gen
> >> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 63, in run
> >> return f(*args, **kwargs)
> >> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerato
> >> r
> >> return g.throw(self.type, self.value, self.tb)
> >> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 95, in _got_event
> >> log.err(e, 'Got exception when handling reporter events')
> >> --- <exception caught here> ---
> >> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 93, in _got_event
> >> yield self.sendMessage(reports)
> >> twisted.mail._except.SMTPDeliveryError: No recipients accepted
> >>>>> MAIL FROM:<bu...@apache.org>
> >> <<< 250 2.1.0 Ok
> >>
> >>
> >>>
> >>> Sebb
> >>>> template=u'''\
> >>>> Build status: {{ summary }}
> >>>> Worker used: {{ workername }}
> >>>> URL: {{ build_url }}
> >>>> Blamelist: {{ blamelist | join(', ') }}
> >>>> Build Text: {{ build['state_string'] }}
> >>>> Status Detected: {{ status_detected }}
> >>>> {{ sourcestamps }}
> >>>>
> >>>> Steps:
> >>>> {% for step in build['steps'] %}
> >>>> {{ step['name'] }}: {{ step['results'] }}
> >>>> {% endfor %}
> >>>>
> >>>> -- ASF Buildbot
> >>>> '''
> >>>> simple_generator = reporters.BuildStatusGenerator(
> >>>> mode=('all'),
> >>>> builders=["infrastructure-test",],
> >>>> message_formatter=reporters.MessageFormatter(
> >>>> template=template,
> >>>> template_type='plain',
> >>>> want_properties=True,
> >>>> want_steps=True,
> >>>> ),
> >>>> )
> >>>> itn = reporters.MailNotifier(
> >>>> fromaddr="buildbot@apache.org",
> >>>> sendToInterestedUsers=True,
> >>>> generators=[simple_generator],
> >>>> extraRecipients=["gnomes@infra.apache.org"],
> >>>> )
> >>>> bb.add_service(itn)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> Sebb
>
Re: Detecting errors in Buildbot
Posted by fluxo <cm...@apache.org>.
> On Jan 5, 2023, at 12:03 AM, sebb <se...@gmail.com> wrote:
>
> On Thu, 5 Jan 2023 at 01:10, fluxo <cml@apache.org <ma...@apache.org>> wrote:
>>
>>> On Jan 4, 2023, at 3:15 PM, sebb <se...@gmail.com> wrote:
>>>
>>> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus <cml@apache.org <ma...@apache.org>> wrote:
>>>>
>>>>
>>>>
>>>>> On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
>>>>>
>>>>> The www-site builds failed recently, however this was not obvious
>>>>> because the failures don't appear to have been reported anywhere
>>>>> outside the dashboard.
>>>>>
>>>>> As I recall, the previous version of BuildBot would send an email for
>>>>> broken (and mended) builds. This was very handy for detecting and
>>>>> fixing errors.
>>>>>
>>>>> Are there any plans to provide something similar for the current Buildbot?
>>>>>
>>>>
>>>> The functionality is available. Here is an example configuration for setting up an email notifier.
>>>
>>> Thanks, AFAICT something similar is already set up here:
>>>
>>> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
>>>
>>> However it does not seem to be working.
>>
>>
>> I've opened INFRA-24056 to track this. The error is attached below. As a stopgap, we can add an extraRecipients field, because it's not correctly detecting the contributors, and is thus unable to generate a list of "interested users."
>
> I see.
>
>> What list makes most sense to see failures related to the pelican websites builds?
>
> I'm not sure if there is a single list that would be correct for all
> those builds.
We're creating a site-builds@infra list for this purpose, as currently the workaround requires all pelican build messages to go to a single address. People will be able to opt-in (subscribe) for build output, but it will be noisy, yes.
> However it would probably be OK to use the public list site-dev@a.o <ma...@a.o>
> which was used in the past for bb builds. See for example:
>
> https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p
>
> If this proves too noisy it might be necessary to redirect some build
> mails elsewhere.
>
> There also needs to be a way to ensure that script failures such as
> the one below are brought to the attention of builds and/or infra.
>
>> 2022-12-31 19:22:37+0000 [-] INVALID EMAIL: asfinfra
>> 2022-12-31 19:22:37+0000 [-] sending mail (653 bytes) to []
>> 2022-12-31 19:22:37+0000 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7fa8f003ce80>
>> 2022-12-31 19:22:37+0000 [ESMTPSender,client] Got exception when handling reporter events
>> Traceback (most recent call last):
>> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 1658, in _inlineCallbacks
>> cast(Failure, result).throwExceptionIntoGenerator, gen
>> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 63, in run
>> return f(*args, **kwargs)
>> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerato
>> r
>> return g.throw(self.type, self.value, self.tb)
>> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 95, in _got_event
>> log.err(e, 'Got exception when handling reporter events')
>> --- <exception caught here> ---
>> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 93, in _got_event
>> yield self.sendMessage(reports)
>> twisted.mail._except.SMTPDeliveryError: No recipients accepted
>>>>> MAIL FROM:<bu...@apache.org>
>> <<< 250 2.1.0 Ok
>>
>>
>>>
>>> Sebb
>>>> template=u'''\
>>>> Build status: {{ summary }}
>>>> Worker used: {{ workername }}
>>>> URL: {{ build_url }}
>>>> Blamelist: {{ blamelist | join(', ') }}
>>>> Build Text: {{ build['state_string'] }}
>>>> Status Detected: {{ status_detected }}
>>>> {{ sourcestamps }}
>>>>
>>>> Steps:
>>>> {% for step in build['steps'] %}
>>>> {{ step['name'] }}: {{ step['results'] }}
>>>> {% endfor %}
>>>>
>>>> -- ASF Buildbot
>>>> '''
>>>> simple_generator = reporters.BuildStatusGenerator(
>>>> mode=('all'),
>>>> builders=["infrastructure-test",],
>>>> message_formatter=reporters.MessageFormatter(
>>>> template=template,
>>>> template_type='plain',
>>>> want_properties=True,
>>>> want_steps=True,
>>>> ),
>>>> )
>>>> itn = reporters.MailNotifier(
>>>> fromaddr="buildbot@apache.org",
>>>> sendToInterestedUsers=True,
>>>> generators=[simple_generator],
>>>> extraRecipients=["gnomes@infra.apache.org"],
>>>> )
>>>> bb.add_service(itn)
>>>>
>>>>
>>>>
>>>>
>>>>> Sebb
Re: Detecting errors in Buildbot
Posted by sebb <se...@gmail.com>.
On Thu, 5 Jan 2023 at 01:10, fluxo <cm...@apache.org> wrote:
>
> > On Jan 4, 2023, at 3:15 PM, sebb <se...@gmail.com> wrote:
> >
> > On Wed, 4 Jan 2023 at 18:18, Chris Lambertus <cml@apache.org <ma...@apache.org>> wrote:
> >>
> >>
> >>
> >>> On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
> >>>
> >>> The www-site builds failed recently, however this was not obvious
> >>> because the failures don't appear to have been reported anywhere
> >>> outside the dashboard.
> >>>
> >>> As I recall, the previous version of BuildBot would send an email for
> >>> broken (and mended) builds. This was very handy for detecting and
> >>> fixing errors.
> >>>
> >>> Are there any plans to provide something similar for the current Buildbot?
> >>>
> >>
> >> The functionality is available. Here is an example configuration for setting up an email notifier.
> >
> > Thanks, AFAICT something similar is already set up here:
> >
> > https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
> >
> > However it does not seem to be working.
>
>
> I've opened INFRA-24056 to track this. The error is attached below. As a stopgap, we can add an extraRecipients field, because it's not correctly detecting the contributors, and is thus unable to generate a list of "interested users."
I see.
> What list makes most sense to see failures related to the pelican websites builds?
I'm not sure if there is a single list that would be correct for all
those builds.
However it would probably be OK to use the public list site-dev@a.o
which was used in the past for bb builds. See for example:
https://lists.apache.org/thread/rddkfm7wn3k7vk5rfzdt6sx0fn7zvh1p
If this proves too noisy it might be necessary to redirect some build
mails elsewhere.
There also needs to be a way to ensure that script failures such as
the one below are brought to the attention of builds and/or infra.
> 2022-12-31 19:22:37+0000 [-] INVALID EMAIL: asfinfra
> 2022-12-31 19:22:37+0000 [-] sending mail (653 bytes) to []
> 2022-12-31 19:22:37+0000 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7fa8f003ce80>
> 2022-12-31 19:22:37+0000 [ESMTPSender,client] Got exception when handling reporter events
> Traceback (most recent call last):
> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 1658, in _inlineCallbacks
> cast(Failure, result).throwExceptionIntoGenerator, gen
> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 63, in run
> return f(*args, **kwargs)
> File "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerato
> r
> return g.throw(self.type, self.value, self.tb)
> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 95, in _got_event
> log.err(e, 'Got exception when handling reporter events')
> --- <exception caught here> ---
> File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 93, in _got_event
> yield self.sendMessage(reports)
> twisted.mail._except.SMTPDeliveryError: No recipients accepted
> >>> MAIL FROM:<bu...@apache.org>
> <<< 250 2.1.0 Ok
>
>
> >
> > Sebb
> >> template=u'''\
> >> Build status: {{ summary }}
> >> Worker used: {{ workername }}
> >> URL: {{ build_url }}
> >> Blamelist: {{ blamelist | join(', ') }}
> >> Build Text: {{ build['state_string'] }}
> >> Status Detected: {{ status_detected }}
> >> {{ sourcestamps }}
> >>
> >> Steps:
> >> {% for step in build['steps'] %}
> >> {{ step['name'] }}: {{ step['results'] }}
> >> {% endfor %}
> >>
> >> -- ASF Buildbot
> >> '''
> >> simple_generator = reporters.BuildStatusGenerator(
> >> mode=('all'),
> >> builders=["infrastructure-test",],
> >> message_formatter=reporters.MessageFormatter(
> >> template=template,
> >> template_type='plain',
> >> want_properties=True,
> >> want_steps=True,
> >> ),
> >> )
> >> itn = reporters.MailNotifier(
> >> fromaddr="buildbot@apache.org",
> >> sendToInterestedUsers=True,
> >> generators=[simple_generator],
> >> extraRecipients=["gnomes@infra.apache.org"],
> >> )
> >> bb.add_service(itn)
> >>
> >>
> >>
> >>
> >>> Sebb
>
Re: Detecting errors in Buildbot
Posted by fluxo <cm...@apache.org>.
> On Jan 4, 2023, at 3:15 PM, sebb <se...@gmail.com> wrote:
>
> On Wed, 4 Jan 2023 at 18:18, Chris Lambertus <cml@apache.org <ma...@apache.org>> wrote:
>>
>>
>>
>>> On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
>>>
>>> The www-site builds failed recently, however this was not obvious
>>> because the failures don't appear to have been reported anywhere
>>> outside the dashboard.
>>>
>>> As I recall, the previous version of BuildBot would send an email for
>>> broken (and mended) builds. This was very handy for detecting and
>>> fixing errors.
>>>
>>> Are there any plans to provide something similar for the current Buildbot?
>>>
>>
>> The functionality is available. Here is an example configuration for setting up an email notifier.
>
> Thanks, AFAICT something similar is already set up here:
>
> https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
>
> However it does not seem to be working.
I've opened INFRA-24056 to track this. The error is attached below. As a stopgap, we can add an extraRecipients field, because it's not correctly detecting the contributors, and is thus unable to generate a list of "interested users." What list makes most sense to see failures related to the pelican websites builds?
2022-12-31 19:22:37+0000 [-] INVALID EMAIL: asfinfra
2022-12-31 19:22:37+0000 [-] sending mail (653 bytes) to []
2022-12-31 19:22:37+0000 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7fa8f003ce80>
2022-12-31 19:22:37+0000 [ESMTPSender,client] Got exception when handling reporter events
Traceback (most recent call last):
File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 1658, in _inlineCallbacks
cast(Failure, result).throwExceptionIntoGenerator, gen
File "/x1/buildmaster/lib/python3.6/site-packages/twisted/internet/defer.py", line 63, in run
return f(*args, **kwargs)
File "/x1/buildmaster/lib/python3.6/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerato
r
return g.throw(self.type, self.value, self.tb)
File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 95, in _got_event
log.err(e, 'Got exception when handling reporter events')
--- <exception caught here> ---
File "/x1/buildmaster/lib/python3.6/site-packages/buildbot/reporters/base.py", line 93, in _got_event
yield self.sendMessage(reports)
twisted.mail._except.SMTPDeliveryError: No recipients accepted
>>> MAIL FROM:<bu...@apache.org>
<<< 250 2.1.0 Ok
>
> Sebb
>> template=u'''\
>> Build status: {{ summary }}
>> Worker used: {{ workername }}
>> URL: {{ build_url }}
>> Blamelist: {{ blamelist | join(', ') }}
>> Build Text: {{ build['state_string'] }}
>> Status Detected: {{ status_detected }}
>> {{ sourcestamps }}
>>
>> Steps:
>> {% for step in build['steps'] %}
>> {{ step['name'] }}: {{ step['results'] }}
>> {% endfor %}
>>
>> -- ASF Buildbot
>> '''
>> simple_generator = reporters.BuildStatusGenerator(
>> mode=('all'),
>> builders=["infrastructure-test",],
>> message_formatter=reporters.MessageFormatter(
>> template=template,
>> template_type='plain',
>> want_properties=True,
>> want_steps=True,
>> ),
>> )
>> itn = reporters.MailNotifier(
>> fromaddr="buildbot@apache.org",
>> sendToInterestedUsers=True,
>> generators=[simple_generator],
>> extraRecipients=["gnomes@infra.apache.org"],
>> )
>> bb.add_service(itn)
>>
>>
>>
>>
>>> Sebb
Re: Detecting errors in Buildbot
Posted by sebb <se...@gmail.com>.
On Wed, 4 Jan 2023 at 18:18, Chris Lambertus <cm...@apache.org> wrote:
>
>
>
> > On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
> >
> > The www-site builds failed recently, however this was not obvious
> > because the failures don't appear to have been reported anywhere
> > outside the dashboard.
> >
> > As I recall, the previous version of BuildBot would send an email for
> > broken (and mended) builds. This was very handy for detecting and
> > fixing errors.
> >
> > Are there any plans to provide something similar for the current Buildbot?
> >
>
> The functionality is available. Here is an example configuration for setting up an email notifier.
Thanks, AFAICT something similar is already set up here:
https://github.com/apache/infrastructure-p6/blob/bd3e16c73a14b306942829baa516ca5101cbcdd3/modules/buildbot_asf/files/pelican_websites.py#L56
However it does not seem to be working.
Sebb
> template=u'''\
> Build status: {{ summary }}
> Worker used: {{ workername }}
> URL: {{ build_url }}
> Blamelist: {{ blamelist | join(', ') }}
> Build Text: {{ build['state_string'] }}
> Status Detected: {{ status_detected }}
> {{ sourcestamps }}
>
> Steps:
> {% for step in build['steps'] %}
> {{ step['name'] }}: {{ step['results'] }}
> {% endfor %}
>
> -- ASF Buildbot
> '''
> simple_generator = reporters.BuildStatusGenerator(
> mode=('all'),
> builders=["infrastructure-test",],
> message_formatter=reporters.MessageFormatter(
> template=template,
> template_type='plain',
> want_properties=True,
> want_steps=True,
> ),
> )
> itn = reporters.MailNotifier(
> fromaddr="buildbot@apache.org",
> sendToInterestedUsers=True,
> generators=[simple_generator],
> extraRecipients=["gnomes@infra.apache.org"],
> )
> bb.add_service(itn)
>
>
>
>
> > Sebb
>
Re: Detecting errors in Buildbot
Posted by Chris Lambertus <cm...@apache.org>.
> On Jan 4, 2023, at 4:03 AM, sebb <se...@gmail.com> wrote:
>
> The www-site builds failed recently, however this was not obvious
> because the failures don't appear to have been reported anywhere
> outside the dashboard.
>
> As I recall, the previous version of BuildBot would send an email for
> broken (and mended) builds. This was very handy for detecting and
> fixing errors.
>
> Are there any plans to provide something similar for the current Buildbot?
>
The functionality is available. Here is an example configuration for setting up an email notifier.
template=u'''\
Build status: {{ summary }}
Worker used: {{ workername }}
URL: {{ build_url }}
Blamelist: {{ blamelist | join(', ') }}
Build Text: {{ build['state_string'] }}
Status Detected: {{ status_detected }}
{{ sourcestamps }}
Steps:
{% for step in build['steps'] %}
{{ step['name'] }}: {{ step['results'] }}
{% endfor %}
-- ASF Buildbot
'''
simple_generator = reporters.BuildStatusGenerator(
mode=('all'),
builders=["infrastructure-test",],
message_formatter=reporters.MessageFormatter(
template=template,
template_type='plain',
want_properties=True,
want_steps=True,
),
)
itn = reporters.MailNotifier(
fromaddr="buildbot@apache.org",
sendToInterestedUsers=True,
generators=[simple_generator],
extraRecipients=["gnomes@infra.apache.org"],
)
bb.add_service(itn)
> Sebb