You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mikelui (via GitHub)" <gi...@apache.org> on 2023/08/25 02:50:38 UTC

[GitHub] [arrow] mikelui opened a new pull request, #37376: GH-32439: [Python] Change order of Append during PyStructConverter

mikelui opened a new pull request, #37376:
URL: https://github.com/apache/arrow/pull/37376

   
   
   ### Rationale for this change
   
   See: #32439 
   
   ### What changes are included in this PR?
   
   During conversion from Python to Arrow, when a struct's child hits a capacity error and chunking is triggered, this can leave the Finish'd chunk in an invalid state since the struct's length does not match the length of its children.
   
   This change simply tries to Append the children first, and only if successful will Append the struct. This is safe because the order of Append'ing between the struct and its child is not specified. It is only specified that they must be consistent with each other.
   
   This is per: 
   
   https://github.com/apache/arrow/blob/86b7a84c9317fa08222eb63f6930bbb54c2e6d0b/cpp/src/arrow/array/builder_nested.h#L507-L508
   
   ### Are these changes tested?
   
   A unit test is added that would previously have an invalid data error:
   
   ```
   >       tab = pa.Table.from_pandas(df)
   
   pyarrow/tests/test_pandas.py:4970: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   pyarrow/table.pxi:3788: in pyarrow.lib.Table.from_pandas
       return cls.from_arrays(arrays, schema=schema)
   pyarrow/table.pxi:3890: in pyarrow.lib.Table.from_arrays
       result.validate()
   pyarrow/table.pxi:3170: in pyarrow.lib.Table.validate
       check_status(self.table.Validate())
   
   # ...
   
   FAILED pyarrow/tests/test_pandas.py::test_nested_chunking_valid - pyarrow.lib.ArrowInvalid: Column 0: In chunk 0: Invalid: List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array (2 < 3)
   ```
   
   NOTE: This unit test uses about 7GB of memory (max RSS) on my macbook pro. This might make CI challenging; I'm open to suggestions to limit it.
   
   ### Are there any user-facing changes?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-32439: [Python] Fix off by one bug when chunking nested structs [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #37376:
URL: https://github.com/apache/arrow/pull/37376


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-32439: [Python] Fix off by one bug when chunking nested structs [arrow]

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1755696776

   cc @westonpace  @wjones127  @mapleFU  @pitrou can someone take a look here? 🄹


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mikelui commented on pull request #37376: GH-32439: [Python] Change order of Append during PyStructConverter

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1692720381

   Failures are due to high memory from the unit test šŸ˜®ā€šŸ’Ø


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mikelui commented on pull request #37376: GH-32439: [Python] Change order of Append during PyStructConverter

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1694552004

   I removed the additional test that was blocking tests, due to intensive memory requirements.
   
   For posterity, I left the test in as the initial commit (with the removal being a successive commit). Folks can confirm the behavior fix independently.
   
   I don't think we should block fixing a critical bug on the memory intensive test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-32439: [Python] Fix off by one bug when chunking nested structs [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1760567267

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 8cdce281634a29b8f79fcc4725f7fad643fa6f7a.
   
   There were no benchmark performance regressions. šŸŽ‰
   
   The [full Conbench report](https://github.com/apache/arrow/runs/17661194656) has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] c0g commented on pull request #37376: GH-32439: [Python] Fix off by one bug when chunking nested structs

Posted by "c0g (via GitHub)" <gi...@apache.org>.
c0g commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1712905974

   I’m hitting this error too, any chance this PR could be merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-32439: [Python] Fix off by one bug when chunking nested structs [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1746340170

   Sorry for slow response from us @mikelui , thank you for keeping it going!
   I plan to look at the PR today.
   
   PS: The mark for the tests with large memory is the correct way to go šŸ‘ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-32439: [Python] Fix off by one bug when chunking nested structs [arrow]

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1745779283

   @AlenkaF @wjones127 There are some backlog PRs is there any chance of getting this reviewed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #37376: GH-32439: [Python] Change order of Append during PyStructConverter

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1692681643

   :warning: GitHub issue #32439 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mikelui commented on pull request #37376: GH-32439: [Python] Change order of Append during PyStructConverter

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1693406450

   Lowered the size of the binary array to be within mem limits (barely scraping by 🄹)
   
   720,000,000 * 3 = 2,160,000,000, which triggers chunking over 2,147,483,647


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mikelui commented on pull request #37376: GH-32439: [Python] Fix off by one bug when chunking nested structs

Posted by "mikelui (via GitHub)" <gi...@apache.org>.
mikelui commented on PR #37376:
URL: https://github.com/apache/arrow/pull/37376#issuecomment-1713768842

   @westonpace since you seemed to be PoC on the GH issue, can you take a look or direct us to the right person to review this? It seems there's a bit of a backlog in reviewing [Python] PR's, anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org