Resource usage exceeded: topics-per-project

[Mikhail Gryzykhin <mi...@google.com>]

Hi everyone,

Our python pipelines failed with limit exceeded error
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>:

ResourceExhausted: 429 Your project has exceeded a limit:
(type="topics-per-project", current=10000, maximum=10000).


Does anyone know if there were new tests that use topics added recently?

I tried to see list of topics, but UI fails
<https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
to load. Will see if I can use APIs to investigate.

If anyone has good insight, please, pick up BEAM-6610
<https://issues.apache.org/jira/browse/BEAM-6610>.

--Mikhail

Have feedback <http://go/migryz-feedback>?

Re: Resource usage exceeded: topics-per-project

[Andrew Pilloud <ap...@google.com>]

Oops, I'm mixing up terms here. Topics != Subscriptions. We shouldn't be
leaking topics.

Andrew

On Wed, Feb 6, 2019 at 9:04 AM Kenneth Knowles <kl...@google.com> wrote:

> To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL
> doesn't compensate for that.
>
> On Wed, Feb 6, 2019 at 8:45 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Thank you for quick response Andrew.
>>
>> I'll cleanup these. I'll keep the bug open and assign it to @Kenneth
>> Knowles <kl...@google.com> who's working on SQK for follow up: we need a
>> way to automatically cleanup topics.
>>
>> Current suggestions:
>> 1. Make SQL cleanup created topics
>> 2. Cleanup topics created by SQL in tests
>> 3. Increase quota so that topics have enough time to be cleaned up
>> automatically.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Wed, Feb 6, 2019 at 8:05 AM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> SQL doesn't cleanup pubsub subscriptions. Feel free to delete those.
>>>
>>> Andrew
>>>
>>> On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> +Kenneth Knowles <kl...@google.com> you're working on SQL recently, so
>>>> might provide some info.
>>>>
>>>> I see a lot of topics of format
>>>> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
>>>> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
>>>> Seems we do not cleanup properly.
>>>>
>>>> Is it safe to cleanup topics with this name?
>>>>
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>>
>>>> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com>
>>>> wrote:
>>>>
>>>>> Minor UPD:
>>>>> As expected it fails most of our test jobs, since we use Pub/Subs in
>>>>> many tests.
>>>>>
>>>>> --Mikhail
>>>>>
>>>>> Have feedback <http://go/migryz-feedback>?
>>>>>
>>>>>
>>>>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Our python pipelines failed with limit exceeded error
>>>>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>>>>>> :
>>>>>>
>>>>>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>>>>>
>>>>>>
>>>>>> Does anyone know if there were new tests that use topics added
>>>>>> recently?
>>>>>>
>>>>>> I tried to see list of topics, but UI fails
>>>>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>>>>>> to load. Will see if I can use APIs to investigate.
>>>>>>
>>>>>> If anyone has good insight, please, pick up BEAM-6610
>>>>>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>>>>>
>>>>>> --Mikhail
>>>>>>
>>>>>> Have feedback <http://go/migryz-feedback>?
>>>>>>
>>>>>

Re: Resource usage exceeded: topics-per-project

[Kenneth Knowles <kl...@google.com>]

Ah, I jumped to the wrong vocabulary. There is no auto-created topic. That
is part of TestPubsub and TestPubsubSignal. It should be cleaned up. The
only use of those is in SQL ITs. But PubsubIO _should_ use it.

Kenn

On Wed, Feb 6, 2019 at 9:04 AM Kenneth Knowles <kl...@google.com> wrote:

> To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL
> doesn't compensate for that.
>
> On Wed, Feb 6, 2019 at 8:45 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Thank you for quick response Andrew.
>>
>> I'll cleanup these. I'll keep the bug open and assign it to @Kenneth
>> Knowles <kl...@google.com> who's working on SQK for follow up: we need a
>> way to automatically cleanup topics.
>>
>> Current suggestions:
>> 1. Make SQL cleanup created topics
>> 2. Cleanup topics created by SQL in tests
>> 3. Increase quota so that topics have enough time to be cleaned up
>> automatically.
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Wed, Feb 6, 2019 at 8:05 AM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> SQL doesn't cleanup pubsub subscriptions. Feel free to delete those.
>>>
>>> Andrew
>>>
>>> On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> +Kenneth Knowles <kl...@google.com> you're working on SQL recently, so
>>>> might provide some info.
>>>>
>>>> I see a lot of topics of format
>>>> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
>>>> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
>>>> Seems we do not cleanup properly.
>>>>
>>>> Is it safe to cleanup topics with this name?
>>>>
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>>
>>>> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com>
>>>> wrote:
>>>>
>>>>> Minor UPD:
>>>>> As expected it fails most of our test jobs, since we use Pub/Subs in
>>>>> many tests.
>>>>>
>>>>> --Mikhail
>>>>>
>>>>> Have feedback <http://go/migryz-feedback>?
>>>>>
>>>>>
>>>>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Our python pipelines failed with limit exceeded error
>>>>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>>>>>> :
>>>>>>
>>>>>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>>>>>
>>>>>>
>>>>>> Does anyone know if there were new tests that use topics added
>>>>>> recently?
>>>>>>
>>>>>> I tried to see list of topics, but UI fails
>>>>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>>>>>> to load. Will see if I can use APIs to investigate.
>>>>>>
>>>>>> If anyone has good insight, please, pick up BEAM-6610
>>>>>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>>>>>
>>>>>> --Mikhail
>>>>>>
>>>>>> Have feedback <http://go/migryz-feedback>?
>>>>>>
>>>>>

Re: Resource usage exceeded: topics-per-project

[Kenneth Knowles <kl...@google.com>]

To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL
doesn't compensate for that.

On Wed, Feb 6, 2019 at 8:45 AM Mikhail Gryzykhin <mi...@google.com> wrote:

> Thank you for quick response Andrew.
>
> I'll cleanup these. I'll keep the bug open and assign it to @Kenneth
> Knowles <kl...@google.com> who's working on SQK for follow up: we need a
> way to automatically cleanup topics.
>
> Current suggestions:
> 1. Make SQL cleanup created topics
> 2. Cleanup topics created by SQL in tests
> 3. Increase quota so that topics have enough time to be cleaned up
> automatically.
>
> Regards,
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
>
> On Wed, Feb 6, 2019 at 8:05 AM Andrew Pilloud <ap...@google.com> wrote:
>
>> SQL doesn't cleanup pubsub subscriptions. Feel free to delete those.
>>
>> Andrew
>>
>> On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mi...@google.com> wrote:
>>
>>> +Kenneth Knowles <kl...@google.com> you're working on SQL recently, so
>>> might provide some info.
>>>
>>> I see a lot of topics of format
>>> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
>>> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
>>> Seems we do not cleanup properly.
>>>
>>> Is it safe to cleanup topics with this name?
>>>
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>>>
>>> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> Minor UPD:
>>>> As expected it fails most of our test jobs, since we use Pub/Subs in
>>>> many tests.
>>>>
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>>
>>>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Our python pipelines failed with limit exceeded error
>>>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>>>>> :
>>>>>
>>>>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>>>>
>>>>>
>>>>> Does anyone know if there were new tests that use topics added
>>>>> recently?
>>>>>
>>>>> I tried to see list of topics, but UI fails
>>>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>>>>> to load. Will see if I can use APIs to investigate.
>>>>>
>>>>> If anyone has good insight, please, pick up BEAM-6610
>>>>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>>>>
>>>>> --Mikhail
>>>>>
>>>>> Have feedback <http://go/migryz-feedback>?
>>>>>
>>>>

Re: Resource usage exceeded: topics-per-project

[Mikhail Gryzykhin <mi...@google.com>]

Thank you for quick response Andrew.

I'll cleanup these. I'll keep the bug open and assign it to @Kenneth Knowles
<kl...@google.com> who's working on SQK for follow up: we need a way to
automatically cleanup topics.

Current suggestions:
1. Make SQL cleanup created topics
2. Cleanup topics created by SQL in tests
3. Increase quota so that topics have enough time to be cleaned up
automatically.

Regards,
--Mikhail

Have feedback <http://go/migryz-feedback>?


On Wed, Feb 6, 2019 at 8:05 AM Andrew Pilloud <ap...@google.com> wrote:

> SQL doesn't cleanup pubsub subscriptions. Feel free to delete those.
>
> Andrew
>
> On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mi...@google.com> wrote:
>
>> +Kenneth Knowles <kl...@google.com> you're working on SQL recently, so
>> might provide some info.
>>
>> I see a lot of topics of format
>> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
>> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
>> Seems we do not cleanup properly.
>>
>> Is it safe to cleanup topics with this name?
>>
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Minor UPD:
>>> As expected it fails most of our test jobs, since we use Pub/Subs in
>>> many tests.
>>>
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>>>
>>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Our python pipelines failed with limit exceeded error
>>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>>>> :
>>>>
>>>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>>>
>>>>
>>>> Does anyone know if there were new tests that use topics added recently?
>>>>
>>>> I tried to see list of topics, but UI fails
>>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>>>> to load. Will see if I can use APIs to investigate.
>>>>
>>>> If anyone has good insight, please, pick up BEAM-6610
>>>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>>>
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>

Re: Resource usage exceeded: topics-per-project

[Andrew Pilloud <ap...@google.com>]

SQL doesn't cleanup pubsub subscriptions. Feel free to delete those.

Andrew

On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mi...@google.com> wrote:

> +Kenneth Knowles <kl...@google.com> you're working on SQL recently, so
> might provide some info.
>
> I see a lot of topics of format
> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
> Seems we do not cleanup properly.
>
> Is it safe to cleanup topics with this name?
>
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
>
> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Minor UPD:
>> As expected it fails most of our test jobs, since we use Pub/Subs in many
>> tests.
>>
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Our python pipelines failed with limit exceeded error
>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>>> :
>>>
>>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>>
>>>
>>> Does anyone know if there were new tests that use topics added recently?
>>>
>>> I tried to see list of topics, but UI fails
>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>>> to load. Will see if I can use APIs to investigate.
>>>
>>> If anyone has good insight, please, pick up BEAM-6610
>>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>>
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>>

Re: Resource usage exceeded: topics-per-project

[Mikhail Gryzykhin <mi...@google.com>]

+Kenneth Knowles <kl...@google.com> you're working on SQL recently, so might
provide some info.

I see a lot of topics of format
rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721
<https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>.
Seems we do not cleanup properly.

Is it safe to cleanup topics with this name?

--Mikhail

Have feedback <http://go/migryz-feedback>?


On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mi...@google.com> wrote:

> Minor UPD:
> As expected it fails most of our test jobs, since we use Pub/Subs in many
> tests.
>
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
>
> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Hi everyone,
>>
>> Our python pipelines failed with limit exceeded error
>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
>> :
>>
>> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>>
>>
>> Does anyone know if there were new tests that use topics added recently?
>>
>> I tried to see list of topics, but UI fails
>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
>> to load. Will see if I can use APIs to investigate.
>>
>> If anyone has good insight, please, pick up BEAM-6610
>> <https://issues.apache.org/jira/browse/BEAM-6610>.
>>
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>

Re: Resource usage exceeded: topics-per-project

[Mikhail Gryzykhin <mi...@google.com>]

Minor UPD:
As expected it fails most of our test jobs, since we use Pub/Subs in many
tests.

--Mikhail

Have feedback <http://go/migryz-feedback>?


On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mi...@google.com> wrote:

> Hi everyone,
>
> Our python pipelines failed with limit exceeded error
> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console>
> :
>
> ResourceExhausted: 429 Your project has exceeded a limit: (type="topics-per-project", current=10000, maximum=10000).
>
>
> Does anyone know if there were new tests that use topics added recently?
>
> I tried to see list of topics, but UI fails
> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing>
> to load. Will see if I can use APIs to investigate.
>
> If anyone has good insight, please, pick up BEAM-6610
> <https://issues.apache.org/jira/browse/BEAM-6610>.
>
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>