[GitHub] [arrow] vpatil-uptycs opened a new issue, #35900: pyarrow.csv.CSVWriter not working with Hadoopfilesystem

["vpatil-uptycs (via GitHub)" <gi...@apache.org>]

vpatil-uptycs opened a new issue, #35900:
URL: https://github.com/apache/arrow/issues/35900

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ```TypeError: Unable to read from object of type: <class 'pyarrow._hdfs.HadoopFileSystem'>```
   
   Code:
   
   ```
   def write_csv_file(engine, query, location):
       base_dir_path = os.path.dirname(location)
       filesystem, path = fs.from_uri(location)
       filesystem.create_dir(base_dir_path)
       with engine.connect() as conn:
           conn.execution_options(stream_results=True)
           cursor_result: CursorResult = conn.execute(query)
           csv_schema = get_csv_schema(cursor_result=cursor_result)    
           total_rows = 0
           with pa.csv.CSVWriter(filesystem, schema=csv_schema) as writer:
               while True:
                   rows: list[Row] = cursor_result.fetchmany(10000)
                   row_count = len(rows)
                   if row_count > 0:
                       df = pd.DataFrame(rows)
                       table = pa.Table.from_pandas(df)
                       writer.write_table(table)
                   else:
                       break
                   total_rows += row_count
           cursor_result.close()
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] vpatil-uptycs closed issue #35900: pyarrow.csv.CSVWriter not working with Hadoopfilesystem

["vpatil-uptycs (via GitHub)" <gi...@apache.org>]

vpatil-uptycs closed issue #35900: pyarrow.csv.CSVWriter not working with Hadoopfilesystem
URL: https://github.com/apache/arrow/issues/35900


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] vpatil-uptycs commented on issue #35900: pyarrow.csv.CSVWriter not working with Hadoopfilesystem

["vpatil-uptycs (via GitHub)" <gi...@apache.org>]

vpatil-uptycs commented on issue #35900:
URL: https://github.com/apache/arrow/issues/35900#issuecomment-1575554760

   I was able to do it....
   
   
   ```
   def write_csv_file(engine, query, location):
       base_dir_path = os.path.dirname(location)
       filesystem, path = FileSystem.from_uri(location)
       filesystem.create_dir(base_dir_path)
       with engine.connect() as conn:
           conn.execution_options(stream_results=True)
           cursor_result: CursorResult = conn.execute(query)
           csv_schema = get_csv_schema(cursor_result=cursor_result)    
           total_rows = 0
           with filesystem.open_output_stream(location, compression='gzip') as file_stream:
               csv_writer = pa.csv.CSVWriter(file_stream, csv_schema)
               while True:
                   rows: list[Row] = cursor_result.fetchmany(10000)
                   row_count = len(rows)
                   if row_count > 0:
                       df = pd.DataFrame(rows)
                       table = pa.Table.from_pandas(df, schema=csv_schema)
                       csv_writer.write_table(table)
                   else:
                       break
                   total_rows += row_count
               csv_writer.close()
           cursor_result.close()
   ```
   
   Hence, closing the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org