You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Lina Mårtensson via user <us...@beam.apache.org> on 2025/05/15 17:43:02 UTC
BigQuery full of abandoned temp datasets
Dear beam-users,
We write a lot to BigQuery but our dataset space is now inundated with many
many datasets named `beam_temp_data...` and `bq_read_all...`
Are we doing something wrong or is this a bug?
It seems like these should be temp datasets that are automatically removed
after some time, and/or named starting with _ to make them private.
See attached screenshot.
Thanks,
-Lina
[image: Screenshot 2025-05-15 at 10.41.10 AM.png]
Re: BigQuery full of abandoned temp datasets
Posted by XQ Hu via user <us...@beam.apache.org>.
For the BQ Python SDK, temp_dataset can change that location.
On Mon, Jun 2, 2025 at 2:50 PM Lina Mårtensson <li...@camus.energy> wrote:
> Thanks!
>
> What do you think about changing the naming of them? Starting with _ so
> that they're private/not visible by default?
>
> On Wed, May 28, 2025 at 6:30 AM XQ Hu <xq...@google.com> wrote:
>
>> I think it is probably much better and safer for the users to create some
>> cron jobs to clean up these temp datasets.
>>
>> On Wed, May 28, 2025 at 12:06 AM Lina Mårtensson <li...@camus.energy>
>> wrote:
>>
>>> It turns out to be when we interrupt jobs. That makes sense, can't clean
>>> up the temp table if the job has died.
>>>
>>> But - if they were temp datasets that were automatically removed after
>>> some time, and/or named starting with _ to make them private, the BigQuery
>>> dataset & table space wouldn't have to be polluted by all these dead tables
>>> forever.
>>> It would be great to see this change.
>>>
>>> Thanks!
>>>
>>> On Mon, May 19, 2025 at 9:50 AM XQ Hu <xq...@google.com> wrote:
>>>
>>>> https://github.com/apache/beam/issues/20544 is this issue related to
>>>> your case? Can you provide more details about how to reproduce this?
>>>>
>>>>
>>>> On Thu, May 15, 2025 at 1:44 PM Lina Mårtensson via user <
>>>> user@beam.apache.org> wrote:
>>>>
>>>>> Dear beam-users,
>>>>>
>>>>> We write a lot to BigQuery but our dataset space is now inundated with
>>>>> many many datasets named `beam_temp_data...` and `bq_read_all...`
>>>>> Are we doing something wrong or is this a bug?
>>>>> It seems like these should be temp datasets that are automatically
>>>>> removed after some time, and/or named starting with _ to make them private.
>>>>>
>>>>> See attached screenshot.
>>>>>
>>>>> Thanks,
>>>>> -Lina
>>>>>
>>>>> [image: Screenshot 2025-05-15 at 10.41.10 AM.png]
>>>>>
>>>>
Re: BigQuery full of abandoned temp datasets
Posted by Lina Mårtensson via user <us...@beam.apache.org>.
Thanks!
What do you think about changing the naming of them? Starting with _ so
that they're private/not visible by default?
On Wed, May 28, 2025 at 6:30 AM XQ Hu <xq...@google.com> wrote:
> I think it is probably much better and safer for the users to create some
> cron jobs to clean up these temp datasets.
>
> On Wed, May 28, 2025 at 12:06 AM Lina Mårtensson <li...@camus.energy>
> wrote:
>
>> It turns out to be when we interrupt jobs. That makes sense, can't clean
>> up the temp table if the job has died.
>>
>> But - if they were temp datasets that were automatically removed after
>> some time, and/or named starting with _ to make them private, the BigQuery
>> dataset & table space wouldn't have to be polluted by all these dead tables
>> forever.
>> It would be great to see this change.
>>
>> Thanks!
>>
>> On Mon, May 19, 2025 at 9:50 AM XQ Hu <xq...@google.com> wrote:
>>
>>> https://github.com/apache/beam/issues/20544 is this issue related to
>>> your case? Can you provide more details about how to reproduce this?
>>>
>>>
>>> On Thu, May 15, 2025 at 1:44 PM Lina Mårtensson via user <
>>> user@beam.apache.org> wrote:
>>>
>>>> Dear beam-users,
>>>>
>>>> We write a lot to BigQuery but our dataset space is now inundated with
>>>> many many datasets named `beam_temp_data...` and `bq_read_all...`
>>>> Are we doing something wrong or is this a bug?
>>>> It seems like these should be temp datasets that are automatically
>>>> removed after some time, and/or named starting with _ to make them private.
>>>>
>>>> See attached screenshot.
>>>>
>>>> Thanks,
>>>> -Lina
>>>>
>>>> [image: Screenshot 2025-05-15 at 10.41.10 AM.png]
>>>>
>>>
Re: BigQuery full of abandoned temp datasets
Posted by XQ Hu via user <us...@beam.apache.org>.
I think it is probably much better and safer for the users to create some
cron jobs to clean up these temp datasets.
On Wed, May 28, 2025 at 12:06 AM Lina Mårtensson <li...@camus.energy> wrote:
> It turns out to be when we interrupt jobs. That makes sense, can't clean
> up the temp table if the job has died.
>
> But - if they were temp datasets that were automatically removed after
> some time, and/or named starting with _ to make them private, the BigQuery
> dataset & table space wouldn't have to be polluted by all these dead tables
> forever.
> It would be great to see this change.
>
> Thanks!
>
> On Mon, May 19, 2025 at 9:50 AM XQ Hu <xq...@google.com> wrote:
>
>> https://github.com/apache/beam/issues/20544 is this issue related to
>> your case? Can you provide more details about how to reproduce this?
>>
>>
>> On Thu, May 15, 2025 at 1:44 PM Lina Mårtensson via user <
>> user@beam.apache.org> wrote:
>>
>>> Dear beam-users,
>>>
>>> We write a lot to BigQuery but our dataset space is now inundated with
>>> many many datasets named `beam_temp_data...` and `bq_read_all...`
>>> Are we doing something wrong or is this a bug?
>>> It seems like these should be temp datasets that are automatically
>>> removed after some time, and/or named starting with _ to make them private.
>>>
>>> See attached screenshot.
>>>
>>> Thanks,
>>> -Lina
>>>
>>> [image: Screenshot 2025-05-15 at 10.41.10 AM.png]
>>>
>>
Re: BigQuery full of abandoned temp datasets
Posted by Lina Mårtensson via user <us...@beam.apache.org>.
It turns out to be when we interrupt jobs. That makes sense, can't clean up
the temp table if the job has died.
But - if they were temp datasets that were automatically removed after some
time, and/or named starting with _ to make them private, the BigQuery
dataset & table space wouldn't have to be polluted by all these dead tables
forever.
It would be great to see this change.
Thanks!
On Mon, May 19, 2025 at 9:50 AM XQ Hu <xq...@google.com> wrote:
> https://github.com/apache/beam/issues/20544 is this issue related to your
> case? Can you provide more details about how to reproduce this?
>
>
> On Thu, May 15, 2025 at 1:44 PM Lina Mårtensson via user <
> user@beam.apache.org> wrote:
>
>> Dear beam-users,
>>
>> We write a lot to BigQuery but our dataset space is now inundated with
>> many many datasets named `beam_temp_data...` and `bq_read_all...`
>> Are we doing something wrong or is this a bug?
>> It seems like these should be temp datasets that are automatically
>> removed after some time, and/or named starting with _ to make them private.
>>
>> See attached screenshot.
>>
>> Thanks,
>> -Lina
>>
>> [image: Screenshot 2025-05-15 at 10.41.10 AM.png]
>>
>
Re: BigQuery full of abandoned temp datasets
Posted by XQ Hu via user <us...@beam.apache.org>.
https://github.com/apache/beam/issues/20544 is this issue related to your
case? Can you provide more details about how to reproduce this?
On Thu, May 15, 2025 at 1:44 PM Lina Mårtensson via user <
user@beam.apache.org> wrote:
> Dear beam-users,
>
> We write a lot to BigQuery but our dataset space is now inundated with
> many many datasets named `beam_temp_data...` and `bq_read_all...`
> Are we doing something wrong or is this a bug?
> It seems like these should be temp datasets that are automatically removed
> after some time, and/or named starting with _ to make them private.
>
> See attached screenshot.
>
> Thanks,
> -Lina
>
> [image: Screenshot 2025-05-15 at 10.41.10 AM.png]
>