You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "MislavSag (via GitHub)" <gi...@apache.org> on 2025/03/25 20:09:02 UTC
[I] Failed to parse string: '' as a scalar of type double [arrow]
MislavSag opened a new issue, #45935:
URL: https://github.com/apache/arrow/issues/45935
### Describe the bug, including details regarding any error messages, version, and platform.
I am trying to import data from parquet dataset using R arrow
```
open_dataset("/home/sn/data/equity/us/predictors_daily/forecasts", format = "parquet") %>%
mutate(ets_last_66_Lo80 = try_cast(ets_last_66_Lo80, float64())) %>%
filter(date == dates_seq[100]) %>%
select(88) %>%
collect()
```
I got following error
```
Error in `compute.arrow_dplyr_query()`:
! Invalid: Failed to parse string: '' as a scalar of type double
Run `rlang::last_trace()` to see where the error occurred.
```
I tried to remove "" in followin ways:
- using if_else - not uspported by arrow
- using case_when - not uspported by arrow
- using != - not uspported by arrow
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Failed to parse string: '' as a scalar of type double [arrow]
Posted by "MislavSag (via GitHub)" <gi...@apache.org>.
MislavSag commented on issue #45935:
URL: https://github.com/apache/arrow/issues/45935#issuecomment-2768833451
Hello @amoeba ,
Here is the schema
Schema
date: date32[day]
autoarima_1_66_PointForecast: double
autoarima_1_66_Lo80: double
autoarima_1_66_Hi80: double
autoarima_1_66_Lo95: double
autoarima_1_66_Hi95: double
autoarima_last_66_PointForecast: double
autoarima_last_66_Lo80: double
autoarima_last_66_Hi80: double
autoarima_last_66_Lo95: double
autoarima_last_66_Hi95: double
autoarima_mean_66_PointForecast: double
autoarima_mean_66_Lo80: double
autoarima_mean_66_Hi80: double
autoarima_mean_66_Lo95: double
autoarima_mean_66_Hi95: double
autoarima_sd_66_PointForecast: double
autoarima_sd_66_Lo80: double
autoarima_sd_66_Hi80: double
autoarima_sd_66_Lo95: double
autoarima_sd_66_Hi95: double
autoarima_1_252_PointForecast: double
autoarima_1_252_Lo80: double
autoarima_1_252_Hi80: double
autoarima_1_252_Lo95: double
autoarima_1_252_Hi95: double
autoarima_last_252_PointForecast: double
autoarima_last_252_Lo80: double
autoarima_last_252_Hi80: double
autoarima_last_252_Lo95: double
autoarima_last_252_Hi95: double
autoarima_mean_252_PointForecast: double
autoarima_mean_252_Lo80: double
autoarima_mean_252_Hi80: double
autoarima_mean_252_Lo95: double
autoarima_mean_252_Hi95: double
autoarima_sd_252_PointForecast: double
autoarima_sd_252_Lo80: double
autoarima_sd_252_Hi80: double
autoarima_sd_252_Lo95: double
autoarima_sd_252_Hi95: double
nnetar_1_66_PointForecast: double
nnetar_1_66_Lo80: double
nnetar_1_66_Hi80: double
nnetar_1_66_Lo95: double
nnetar_1_66_Hi95: double
nnetar_last_66_PointForecast: double
nnetar_last_66_Lo80: double
nnetar_last_66_Hi80: double
nnetar_last_66_Lo95: double
nnetar_last_66_Hi95: double
nnetar_mean_66_PointForecast: double
nnetar_mean_66_Lo80: double
nnetar_mean_66_Hi80: double
nnetar_mean_66_Lo95: double
nnetar_mean_66_Hi95: double
nnetar_sd_66_PointForecast: double
nnetar_sd_66_Lo80: double
nnetar_sd_66_Hi80: double
nnetar_sd_66_Lo95: double
nnetar_sd_66_Hi95: double
nnetar_1_252_PointForecast: double
nnetar_1_252_Lo80: double
nnetar_1_252_Hi80: double
nnetar_1_252_Lo95: double
nnetar_1_252_Hi95: double
nnetar_last_252_PointForecast: double
nnetar_last_252_Lo80: double
nnetar_last_252_Hi80: double
nnetar_last_252_Lo95: double
nnetar_last_252_Hi95: double
nnetar_mean_252_PointForecast: double
nnetar_mean_252_Lo80: double
nnetar_mean_252_Hi80: double
nnetar_mean_252_Lo95: double
nnetar_mean_252_Hi95: double
nnetar_sd_252_PointForecast: double
nnetar_sd_252_Lo80: double
nnetar_sd_252_Hi80: double
nnetar_sd_252_Lo95: double
nnetar_sd_252_Hi95: double
ets_1_66_PointForecast: double
ets_1_66_Lo80: double
ets_1_66_Hi80: double
ets_1_66_Lo95: double
ets_1_66_Hi95: double
ets_last_66_PointForecast: double
ets_last_66_Lo80: double
ets_last_66_Hi80: double
ets_last_66_Lo95: double
ets_last_66_Hi95: double
ets_mean_66_PointForecast: double
ets_mean_66_Lo80: double
ets_mean_66_Hi80: double
ets_mean_66_Lo95: double
ets_mean_66_Hi95: double
ets_sd_66_PointForecast: double
ets_sd_66_Lo80: double
ets_sd_66_Hi80: double
ets_sd_66_Lo95: double
ets_sd_66_Hi95: double
ets_1_252_PointForecast: double
ets_1_252_Lo80: double
ets_1_252_Hi80: double
ets_1_252_Lo95: double
ets_1_252_Hi95: double
ets_last_252_PointForecast: double
ets_last_252_Lo80: double
ets_last_252_Hi80: double
ets_last_252_Lo95: double
ets_last_252_Hi95: double
ets_mean_252_PointForecast: double
ets_mean_252_Lo80: double
ets_mean_252_Hi80: double
ets_mean_252_Lo95: double
ets_mean_252_Hi95: double
ets_sd_252_PointForecast: double
ets_sd_252_Lo80: double
ets_sd_252_Hi80: double
ets_sd_252_Lo95: double
ets_sd_252_Hi95: double
symbol: string
The problem start with this column
```
> schema(x)[88]
Schema
ets_last_66_Lo80: double
```
I have parquet database grouped by symbol column. For most of symbols my arrow/dplyr pipeline works, but for some it returns above error.
could it be that accidentaly this column is saved as string for some symbols so it "doesn't know" how to import it as double?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Failed to parse string: '' as a scalar of type double [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45935:
URL: https://github.com/apache/arrow/issues/45935#issuecomment-2762218779
Hi @MislavSag, we might need a bit more info to help you here. First, what does the schema look like when you just run,
```r
open_dataset("/home/sn/data/equity/us/predictors_daily/forecasts", format = "parquet")
```
I'm assuming `ets_last_66_Lo80` is `string` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Failed to parse string: '' as a scalar of type double [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #45935: [R] Failed to parse string: '' as a scalar of type double
URL: https://github.com/apache/arrow/issues/45935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Failed to parse string: '' as a scalar of type double [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #45935:
URL: https://github.com/apache/arrow/issues/45935#issuecomment-3925310471
Closing this - looks like there's a nice workaround suggested, but feel free to reopen if any other issues!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [R] Failed to parse string: '' as a scalar of type double [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45935:
URL: https://github.com/apache/arrow/issues/45935#issuecomment-2770897719
> could it be that accidentaly this column is saved as string for some symbols so it "doesn't know" how to import it as double?
That's a pretty reasonable guess. `open_dataset` determines the schema from the first file it discovers so if the files don't all have the same schemas, you can run into behavior like this. A quick way to check would be to use the `unify_schemas` argument to `open_dataset` (See `?open_dataset). I'm guessing that schema will be different when you set unify_schemas to TRUE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org