You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reece Robinson (Jira)" <ji...@apache.org> on 2023/10/03 23:02:00 UTC
[jira] [Created] (SPARK-45403) Spark SQL returns table column names as literal data values for Hive tables
Reece Robinson created SPARK-45403:
--------------------------------------
Summary: Spark SQL returns table column names as literal data values for Hive tables
Key: SPARK-45403
URL: https://issues.apache.org/jira/browse/SPARK-45403
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.0
Environment: I am using Spark 3.4.0 however this has been an issue for years.
Reporter: Reece Robinson
When using Spark SQL and Hive JDBC driver to access a Hive table the resulting row data is replaced with the literal column name in the resulting dataframe result.
When I run this:
jdbcDF = spark.read \
.format("jdbc") \
.options(driver="org.apache.hive.jdbc.HiveDriver",
url="jdbc:hive2://10.20.174.171:10009",
user="10009",
password="123",
query="select * from demo.hospitals limit 10"
) \
.load()
I get:
+------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+ |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| +------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+ |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| |provider_num|npi|name|address|city|state|zip|fips_county|lat|lon|phone|provider_type_code|category|emergency|upin|pin|region_code|bed_count|clia_lab_number|HIP_PK| +------------+---+----+-------+----+-----+---+-----------+---+---+-----+------------------+--------+---------+----+---+-----------+---------+---------------+------+
I should see:
+--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+ | person_pk| race_value|sex_code|poverty_value|veteran_value|ppr_pro| patient_pk| di_dk| pov_pk|vet_pk|veteran|total_paid|num_drugs|immunized| +--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+ |001252a7-a1e7-428...|01 - American Ind...| F| 37.0| null| 2|65007233-424e-4c2...|9d66f5b7-ab10-47f...|1f3d76c8-d039-483...| |unknown| null| null| true| |002673d4-579a-4d1...|01 - American Ind...| M| 64.0| null| 2|a3c89a7f-d57d-4be...|2f6ffa09-e5b3-419...|7dbfc730-64bc-4a9...| |unknown| null| null| true| |00267822-8192-44f...|01 - American Ind...| F| 0.0| null| 2|cd318b72-35d4-422...|44646492-60ef-44e...|d5f462ef-cd4c-497...| |unknown| null| null| true| |0028fece-59ec-4db...|01 - American Ind...| F| 0.0| null| 2|ee9e09aa-67bc-47e...|3be068de-7fe3-44d...|63a04010-c381-4aa...| |unknown| null| null| true| |003470e7-b548-444...|06 - American Ind...| M| 171.0| null| 2|7ed5b0f9-02b3-459...|1b778c9f-71ab-45a...|84ecc23a-6c39-44d...| |unknown| null| null| false| |0044a493-e226-409...|01 - American Ind...| F| 0.0| null| 2|c821f5b2-d0af-428...|26144dac-81f0-44e...|f7355eeb-89a3-4f0...| |unknown| null| null| true| |004d44d0-fdf7-403...|01 - American Ind...| F| 37.0| null| 2|cb6c8e5c-71ab-409...|88eaf3c4-5f00-4e9...|78679644-f4e7-450...| |unknown| null| null| true| |0059c1bf-5263-42a...|03 - Black or Afr...| M| 0.0| null| 2|da9247d1-96fb-44d...|6831544a-faf9-426...|3534f3a8-a367-41e...| |unknown| null| null| true| |007b82b6-ae2e-49e...|01 - American Ind...| M| 43.0| null| 2|3e6fcc8c-c484-465...|90e2a03f-f0a4-48f...|5c9c71e1-901b-481...| |unknown| null| null| true| |00917cf5-d879-43f...|01 - American Ind...| F| 0.0| null| 2|b4fa8b2f-7452-4f5...|779eaa9e-7961-4f0...|2d19ad6b-de35-4bc...| |unknown| null| null| true| +--------------------+--------------------+--------+-------------+-------------+-------+--------------------+--------------------+--------------------+------+-------+----------+---------+---------+
(This is de-identified data)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org