[GitHub] [beam] TSienki opened a new issue, #24726: [Bug]: Cannot read partitions metadata table from bigquery

[GitBox <gi...@apache.org>]

TSienki opened a new issue, #24726:
URL: https://github.com/apache/beam/issues/24726

   ### What happened?
   
   Hello, 
   I wanted to read partitions metadata from bigquery table `project_id.dataset_id.INFORMATION_SCHEMA.PARTITIONS` using `ReadFromBigQuery`. Unfortunately, this function raises an error:
   ```
   RuntimeError: apitools.base.py.exceptions.HttpNotFoundError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/[project_id]/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Mon, 19 Dec 2022 17:02:59 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '404', 'content-length': '388', '-content-encoding': 'gzip'}>, content <{
     "error": {
       "code": 404,
       "message": "Not found: Dataset [project_id]:`[table_name] was not found in location EU",
       "errors": [
         {
           "message": "Not found: Dataset [project_id]:`[table_name] was not found in location EU",
           "domain": "global",
           "reason": "notFound"
         }
       ],
       "status": "NOT_FOUND"
     }
   }
   > [while running '[11]: FindPreviousPartitionDate/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction']
   ```
   
   The part of code that causes the error: 
   ```python 
   "FindPreviousPartitionDate" >> beam.io.ReadFromBigQuery(
       query="SELECT * FROM `[project_id].[dataset_id].INFORMATION_SCHEMA.PARTITIONS`",
       use_standard_sql=True,
       flatten_results=False
   )
   ```
   
   I replaced my actual project id and dataset id with tokens `[project_id]`, `[dataset_id]`. 
   I've tested it with beam versions `2.36.0` and `2.43.0` using direct and dataflow runners. Also tried running it with different argument values like `method`, or `use_standard_sql`, but it doesn't help. 
   
   Do you know if it is possible to read from this table using `ReadFromBigQuery`
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [X] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [X] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Cannot read partitions metadata table from bigquery [beam]

["ding-qin (via GitHub)" <gi...@apache.org>]

ding-qin commented on issue #24726:
URL: https://github.com/apache/beam/issues/24726#issuecomment-1894980729

   I used apache_beam 2.53.0 and this bug still exists.
   It's a common use case that metada of a BigQuery table is required in the query.
   Not only INFORMATION_SCHEMA but also _partitiondate is required in the many cases. For example,
   `select _partitiondate, * from xxxx`
   The same query can be executed successfully using bigquery client library. The reason I want to do it using apache beam is because I want to convert to complex SQL joins to PTransform so that it can be executed in parallel to improve the performance.
   However, I have to work around it for now.
   Hopefully it can be fixed as soon as possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org