Re: [I] BEWARE, DATA LOSS: CreateDataFrameFromArrow reads every batch from Parquet but only the first batch from ND‑JSON unless JSON is loaded with WithChunk(-1) [spark-connect-go]

["grundprinzip (via GitHub)" <gi...@apache.org>]

grundprinzip commented on issue #155:
URL: https://github.com/apache/spark-connect-go/issues/155#issuecomment-3206497083

   I was able to repro this in PySpark as well, yes it seems to be the case that there is an issue with the way the batches are read....
   
   ```
   from pyspark.sql.connect.plan import LocalRelation
   import pyspark.sql.types as t
   from pyspark.sql.connect.dataframe import DataFrame as CDF
   
   import pyarrow as pa
   import pyarrow.ipc as ipc
   
   # Create table with 100 integer rows and a single column named "ID"
   table = pa.Table.from_arrays([pa.array(range(100), type=pa.int64())], names=["ID"])
   
   # Serialize as Arrow IPC into a byte array with exactly 10 record batches
   sink = pa.BufferOutputStream()
   with ipc.new_stream(sink, table.schema) as writer:
       for b in table.to_batches(max_chunksize=10):
           print("batch....")
           writer.write_batch(b)
   
   ipc_bytes = sink.getvalue().to_pybytes()
   
   
   schema = t.StructType([
       t.StructField("ID", t.LongType(), True)
   ])
   
   lr = LocalRelation(table, schema.json())
   lr_df = CDF(lr, spark)
   agg_df = lr_df.groupBy().count()
   
   agg_df.show()
   
   # Extract the plan and modify the local relation
   plan = agg_df._plan.to_proto(spark._client)
   new_p = agg_df._plan.to_proto(spark._client)
   new_p.root.aggregate.input.local_relation.data = ipc_bytes
   
   
   #print(plan)
   #print(new_p)
   
   
   spark._client.to_table(plan, {})
   
   spark._client.to_table(new_p, {})
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org