You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "AlwaysZhi (via GitHub)" <gi...@apache.org> on 2023/09/11 04:00:19 UTC
[GitHub] [arrow] AlwaysZhi opened a new issue, #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
AlwaysZhi opened a new issue, #37647:
URL: https://github.com/apache/arrow/issues/37647
### Describe the usage question you have. Please include as many useful details as possible.
When I attempt to use the array.RecordFromJSON() method to read a relatively large JSON file (approximately 200MB in size), Ithe method does not return for a significant amount of time. It seems to be either blocked indefinitely or possibly stuck in an ongoing data processing loop. I am currently unable to determine whether this is a bug or how to successfully utilize the method to achieve my objective.
Here is my code:
func readJsonFile() {
schema := arrow.NewSchema([]arrow.Field{
{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
}, nil)
jsonFile, err := os.Open("/home/data/mydata.json")
if err != nil {
fmt.Println("open json file:", err)
return
}
defer jsonFile.Close()
mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
rec, _, err := array.RecordFromJSON(mem, schema, jsonFile)
fmt.Printf("record number is： %d", rec.NumRows())
fmt.Println("field name 0 -", rec.Schema().Field(0).Name)
fmt.Println("field name 1 -", rec.Schema().Field(1).Name)
fmt.Println("field name 2 -", rec.Schema().Field(2).Name)
}
My json file format looks like this:
[{"instruction":"a", "input":"b", "output","o1"}, {"instruction":"c", "input":"d", "output","o2"}, ...]
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37647:
URL: https://github.com/apache/arrow/issues/37647#issuecomment-1718324514
Would you be able to provide the actual data file that causes the problem? I created a 200MB JSON file with random values using the format schema you provided and then ran the code you provided to read it. It took just around 7 seconds or so to run and I wasn't able to reproduce indefinite blocking that you reported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37647:
URL: https://github.com/apache/arrow/issues/37647#issuecomment-1723600212
@AlwaysZhi was using the `WithMultipleDocs` option enough to solve the issue? or is there more of a question here?
Typically you need the `WithMultipleDocs` option if the format of the json file is:
```json
{"a": 1, "b": 2......}
{"a": 2, "b": 3......}
{.......}
```
Rather than having them all wrapped inside of `[ ]` as a single JSON document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37647:
URL: https://github.com/apache/arrow/issues/37647#issuecomment-1714583036
I'll take a look at this sometime this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlwaysZhi commented on issue #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
Posted by "AlwaysZhi (via GitHub)" <gi...@apache.org>.
AlwaysZhi commented on issue #37647:
URL: https://github.com/apache/arrow/issues/37647#issuecomment-1720500455
Now i convert the json format and use **WithMultipleDocs** option, it works well.
```go
func TestReadJson(t *testing.T) {
file, err := os.Open("/home/data/180000.json")
if err != nil {
fmt.Println("open json:", err)
}
defer file.Close()
content, err := io.ReadAll(file)
if err != nil {
fmt.Println("read file:", err)
}
var jsonData []map[string]interface{}
err = json.Unmarshal(content, &jsonData)
if err != nil {
fmt.Println("Unmarshal failed:", err)
}
schema := arrow.NewSchema([]arrow.Field{
{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
}, nil)
mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
var recs []arrow.Record
for _, obj := range jsonData {
bytes, err := json.Marshal(obj)
if err != nil {
fmt.Println("Marshal failed:", err)
}
rec, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(string(bytes)), array.WithMultipleDocs())
recs = append(recs, rec)
}
fmt.Println("value -", recs[0].Column(0))
fmt.Println("value -", recs[0].Column(1))
fmt.Println("value -", recs[0].Column(2))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AlwaysZhi commented on issue #37647: [Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file
Posted by "AlwaysZhi (via GitHub)" <gi...@apache.org>.
AlwaysZhi commented on issue #37647:
URL: https://github.com/apache/arrow/issues/37647#issuecomment-1720457329
ok， here is my data file
[data.zip](https://github.com/apache/arrow/files/12615378/data.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org