You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/08/01 07:16:00 UTC
[jira] [Resolved] (SPARK-28567) pyspark-sql returns error "Paths
and fileStatuses are expected to be 1. . Got paths: 1 fileStatuses: 26"
[ https://issues.apache.org/jira/browse/SPARK-28567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-28567.
----------------------------------
Resolution: Incomplete
> pyspark-sql returns error "Paths and fileStatuses are expected to be 1. . Got paths: 1 fileStatuses: 26"
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-28567
> URL: https://issues.apache.org/jira/browse/SPARK-28567
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.3.2
> Environment: Sql below..
> query = "WITH hdr as ( \
> select vin_nbr, element_cd, max(element_timstm) as element_timstmp from dev_edm.gk_veh_element \
> where vin_nbr like '2C%' and element_timstm >= cast('{0}' as timestamp) - interval '1' day \
> and element_timstm < cast('{1}' as timestamp) \
> and ((yr_nbr=substring('{2}', 1, 4) and mth_nbr=substring('{3}', 6, 2) and day_nbr=substring('{4}', 9, 2)) \
> or (yr_nbr=substring('{5}', 1, 4) and mth_nbr=substring('{6}', 6, 2) and day_nbr=substring('{7}', 9, 2))) \
> and ( \
> (ELEMENT_CD = 'LEFT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true' ) or \
> (ELEMENT_CD = 'RIGHT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true') or \
> (ELEMENT_CD = 'LEFT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true' ) or \
> (ELEMENT_CD = 'RIGHT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) = 'true') or \
> (ELEMENT_CD in ('RIGHT_FRONT_TIRE_PRESSURE', 'RIGHT_REAR_TIRE_PRESSURE', 'LEFT_FRONT_TIRE_PRESSURE', 'LEFT_REAR_TIRE_PRESSURE') ) or \
> (ELEMENT_CD = 'DIESEL_EXHAUST_FLUID' and (lower(ELEMENT_VAL) = 'on' ) ) or \
> (ELEMENT_CD = 'BRAKE_FLUID_LOW' and (lower(ELEMENT_VAL) = 'true' ) ) or \
> (ELEMENT_CD = 'BATTERY_CRITICALLY_LOW' and (lower(ELEMENT_VAL) = 'true' ) ) or \
> (ELEMENT_CD = 'ODO_READ' ) or \
> (ELEMENT_CD = 'EOL_READ' and ELEMENT_VAL < 30 ) ) \
> group by vin_nbr, element_cd \
> order by vin_nbr, element_cd) \
> SELECT distinct hdr.vin_nbr, hdr.element_cd, \
> case when dtl.element_cd like '%PRESSURE' then cast(cast(round(dtl.element_val * 1000 / 6894.76) as int) as string) else dtl.element_val end element_val, \
> hdr.element_timstmp, 'BDC LOAD' load_process_nm, 'CFL_SVC' created_user_id, current_timestamp created_timstm, 'CFL_SVC' updated_user_id, current_timestamp updated_timstm \
> from dev_edm.gk_veh_element_np dtl, hdr \
> where dtl.vin_nbr = hdr.vin_nbr \
> and dtl.element_cd = hdr.element_cd \
> and dtl.element_timstm = hdr.element_timstmp \
> and ((dtl.yr_nbr=substring('{8}', 1, 4) and dtl.mth_nbr=substring('{9}', 6, 2) and dtl.day_nbr=substring('{10}', 9, 2)) \
> or (dtl.yr_nbr=substring('{11}', 1, 4) and dtl.mth_nbr=substring('{12}', 6, 2) and dtl.day_nbr=substring('{11}', 9, 2))) \
> and case \
> when dtl.element_cd like '%PRESSURE_STATUS_LOW_FLAG' then lower(dtl.element_val) = 'true' \
> when dtl.element_cd = 'DIESEL_EXHAUST_FLUID' then lower(dtl.element_val) = 'on' \
> when dtl.element_cd = 'BRAKE_FLUID_LOW' then lower(dtl.element_val) = 'true' \
> when dtl.element_cd = 'BATTERY_CRITICALLY_LOW' then lower(dtl.element_val) = 'true' \
> else 1=1 end".format(crntdate, crntdate, prevdate, prevdate, prevdate, crntdate, crntdate, crntdate, prevdate, prevdate, prevdate, crntdate, crntdate, crntdate );
> df = hive.executeQuery(query)
> df.show(10)
> Reporter: Ganesh Kumar
> Priority: Major
>
> df.show(10) errors out.
> pyspark-sql returns an error " Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalStateException: Requested to generate single split. Paths and fileStatuses are expected to be 1. Got paths: 1 fileStatuses: 26"
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org