You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jxb667 (via GitHub)" <gi...@apache.org> on 2025/08/13 23:45:32 UTC

[I] BigQueryToGCSOperator add a new_line argument. [airflow]

jxb667 opened a new issue, #54488:
URL: https://github.com/apache/airflow/issues/54488

   ### Description
   
   Currently all CSV output files are produced with only \n line endings, it would be useful if you could also specify one of the other conventions. For example the MS: \n\r.
   
   ### Use case/motivation
   
   Currently we are reloading the file replacing the line feeds and re-writing. This is not particularly efficient, and adding an argument (much like the field_delimeter argument) to specify is not an uncommon feature for csv writing functions/processes.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BigQueryToGCSOperator add a new_line argument. [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #54488:
URL: https://github.com/apache/airflow/issues/54488#issuecomment-3186206586

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BigQueryToGCSOperator add a new_line argument. [airflow]

Posted by "dejii (via GitHub)" <gi...@apache.org>.
dejii commented on issue #54488:
URL: https://github.com/apache/airflow/issues/54488#issuecomment-3186256672

   Looking at [Job#JobConfigurationExtract](https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationExtract), there’s no parameter to control line terminators. Since `BigQueryToGCSOperator` relies directly on this API, the feature can be supported natively. 
   
   If efficiency is the goal, you might consider streaming data directly from the table using the [BigQuery Storage Read API](https://cloud.google.com/bigquery/docs/reference/storage), formatting it as needed, and then uploading the result to GCS. The upload step can be batched to further improve performance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BigQueryToGCSOperator add a new_line argument. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54488:
URL: https://github.com/apache/airflow/issues/54488#issuecomment-3190810953

   > Looking at [Job#JobConfigurationExtract](https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationExtract), there’s no parameter to control line terminators. Since BigQueryToGCSOperator relies directly on this API, the feature can't be supported natively.
   
   Yep. closing as won't do - but feel free to open an issue to Google API team @jxb667 - maybe then they add it - and then we can add it too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BigQueryToGCSOperator add a new_line argument. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #54488: BigQueryToGCSOperator add a new_line argument.
URL: https://github.com/apache/airflow/issues/54488


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org