[I] parquet does not retain haven::tagged_na() [arrow]

["skolenik (via GitHub)" <gi...@apache.org>]

skolenik opened a new issue, #49149:
URL: https://github.com/apache/arrow/issues/49149

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   It appears that `arrow::write_parquet()` does not preserve `haven::tagged_na()` missing values, and converts them to regular missing values instead (loss of data).
   
   ```
   library(arrow)
   library(haven)
   library(dplyr)
   library(labelled)
   
   this_pq <- tempfile(fileext = "parquet")
   
   mydf <- data.frame(x = c(1,NA, haven::tagged_na("a")))
   mydf
   
   # save as parquet and reopen
   arrow::write_dataset(dataset = mydf, path = this_pq)
   this_ds <- open_dataset(sources = this_pq)
   mydf2 <- collect(this_ds)
   
   # expected result
   is_tagged_na(mydf$x, "a")
   
   # actual
   is_tagged_na(mydf2$x, "a")
   is_regular_na(mydf2$x)
   
   packageVersion("arrow")
   ```
   
   Output:
   
   ```
   > # expected result 
   > is_tagged_na(mydf$x, "a")
   [1] FALSE FALSE  TRUE 
   > # actual 
   > is_tagged_na(mydf2$x, "a") 
   [1] FALSE FALSE FALSE 
   > is_regular_na(mydf2$x) 
   > [1] FALSE  TRUE  TRUE
   > packageVersion("arrow")
   [1] ‘23.0.0’
   ```
    
   https://haven.tidyverse.org/reference/tagged_na.html
   
   Background: In SAS, the special missing values `.a`, `.b`, ..., `.z` are implemented as near negative infinity. In Stata, the special missing values `.a`, `.b`, ..., `.z` are implemented as near positive infinity. So they are literally reserving a few top values in a given format to be interpreted as special missing values rather than numbers (so for the int8 format, Stata goes from -127 to 100, with the value of 101 being interpreted as `.a`, ... 126 as `.z` and 127 as `NA`, see https://www.stata.com/help.cgi?datatypes and https://www.stata.com/help.cgi?missing.) What the implementation is in `haven`, I don't really know (the labels are implemented as `attributes()` and are more or less retained, see a somewhat extended reprex below). The main value-added of the whole concept is that you can distinguish the reasons for missing values with labels such as `haven::labelled(your_numeric_vector, labels = c("Don't know" = tagged_na("d"), "Refused" = tagged_na("r"), "Valid skip" = tagge
 d_na("s"), "Not in universe" = tagged_na("u") ) )`. 
   
   Labels are OK-ish:
   
   ```
   mydf <- data.frame(x = labelled(c(1,NA, haven::tagged_na("a")), labels = c("Blah" = 1, "aaa" = tagged_na("a"))))
   arrow::write_dataset(dataset = mydf, path = this_pq, format="parquet")
   this_ds <- open_dataset(sources = this_pq)
   mydf2 <- collect(this_ds)
   get_value_labels(mydf$x) |> labelled::print_tagged_na()
   get_value_labels(mydf2$x) |> labelled::print_tagged_na()
   ```
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [R] parquet does not retain haven::tagged_na() [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #49149:
URL: https://github.com/apache/arrow/issues/49149#issuecomment-3923020181

   I asked Claude to investigate this. Here's what it found:
   
   **How `haven::tagged_na()` works:**
   Tagged NAs encode missing value tags (a-z) directly in the IEEE 754 NaN payload bits. Regular R NA has bytes `a2 07 00 00 00 00 f0 7f`, while `tagged_na('a')` has `a2 07 00 00 61 00 f0 7f` (0x61 = 'a' in byte 5). R's `ISNA()` returns TRUE for both.
   
   **R → Arrow conversion:**
   Arrow's `is_NA<double>()` calls R's `ISNA()` macro (`r/src/r_to_arrow.cpp:148`), so tagged NAs are marked as null in the validity bitmap. However, the actual bytes ARE preserved in the data buffer.
   
   **Arrow → R conversion (where the loss happens):**
   `Converter_Double::Ingest_some_nulls()` (`r/src/array_to_vector.cpp:219-236`) iterates through values and checks the validity bitmap. For null positions, it writes `NA_REAL` regardless of what's in the buffer:
   
   ```cpp
   auto null_one = [&](R_xlen_t i) {
     p_data[i] = NA_REAL;  // overwrites buffer bytes with standard NA
     return Status::OK();
   };
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] parquet does not retain haven::tagged_na() [arrow]

["v4xsh (via GitHub)" <gi...@apache.org>]

v4xsh commented on issue #49149:
URL: https://github.com/apache/arrow/issues/49149#issuecomment-3855320260

   Thanks for the detailed report and the clear reprex, this is very helpful.
   
   I’d like to work on this issue. From the example, it looks like `haven::tagged_na()` values are not preserved across the Arrow → Parquet → Arrow round-trip and are being converted into regular missing values, resulting in a loss of the tag semantics.
   
   I’ll investigate where this information is dropped in the R → Arrow conversion or during Parquet serialization and follow up with findings.
   
   Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org