Re: [I] WriteToBigQuery in BundleBasedDirectRunner fails when method is FILE_LOADS [beam]

["Wellbek (via GitHub)" <gi...@apache.org>]

Wellbek commented on issue #21061:
URL: https://github.com/apache/beam/issues/21061#issuecomment-4460771819

   Hey! As my intro to contributing to Apache Beam, I took a look at this older issue to see whether it still relevant.
   
   From the traceback, my understanding is that the failure happened when `WriteToBigQuery` was used with:
   
   * `method="FILE_LOADS"`
   * a dynamic `table` callable
   * and the `BundleBasedDirectRunner`
   
   which then failed with:
   
   ```python
   AttributeError: 'str' object has no attribute 'projectId'
   ```
   
   From the issue description, problem stems from `wait_for_bq_job` expecting a proper `JobReference` object, while a plain string was being passed somewhere in the `WaitForBQJobs` path.
   
   To reproduce this, I first created a small GCP setup:
   * created a new GCP project
   * created a GCS bucket
   * created a BigQuery dataset
   
   I then wrote a minimal reproducer (analogous to your step outlined) using:
   
   * `BundleBasedDirectRunner`
   * dynamic destinations
   * `method="FILE_LOADS"`
   
   The reproducer writes rows into multiple dynamic BigQuery tables to ensure the copy/load path is triggered.
   
   I tested this in a local setup with `python:3.14.3` using a recent Beam SDK cloned at May 15 2026 (Beam 2.73.0 release). The pipeline completed successfully and the rows were written correctly to BQ. I could not reproduce the original `AttributeError`.
   
   Just to make sure, I tried some more test cases, specifically
   
   * larger numbers of dynamic tables (50)
   * more rows
   * smaller `batch_size`
   * larger payloads (1KB text)
   * mixed data types (`None`, floats, booleans, unicode text, long strings)
   
   All of these also completed successfully without any error.
   
   During testing, I also added temporary type checks before calls to `wait_for_bq_job` through the cloned python sdk to verify what was being passed in. In all cases, the value was a proper `JobReference` object and never a plain string.
   
   One thing that also stood out while looking through the code history is that the mentioned `WaitForBQJobs` seems to have been removed in Beam 2.42.0 -> 2.43.0 (see https://github.com/apache/beam/compare/v2.42.0...v2.43.0). 
   **I therefore suspect the issue may already have been fixed as part of later refactors.**
   
   At this point I'm unable to reproduce the bug, so I wanted to check whether this issue can likely be closed, or whether there is still a known failing scenario I may have missed.
   
   Since this is also my first time digging into the Beam codebase, I'd really appreciate any guidance on, 1) whether there are more scenarios I should verify against (e.g., using older beam versions), 2) whether there's a more specific reproducer I should try
   
   Happy to run more tests or investigate further if helpful. Cheers
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org