You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nyoungstudios (via GitHub)" <gi...@apache.org> on 2023/03/21 23:13:26 UTC
[GitHub] [beam] nyoungstudios opened a new issue, #25917: [Bug]: Example files for avro bitcoin do not exist in GCS
nyoungstudios opened a new issue, #25917:
URL: https://github.com/apache/beam/issues/25917
### What happened?
The input files as defined here do not exist: https://github.com/apache/beam/blob/ca0787642a6b3804a742326147281c99ae8d08d2/sdks/python/apache_beam/examples/avro_bitcoin.py#L116
Listing the files in that bucket returns a 404, bucket not found exception
```bash
gsutil -m ls -lh "gs://beam-avro-test/bitcoin/txns/*"
# BucketNotFoundException: 404 gs://beam-avro-test bucket does not exist.
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1547164640
I changed the schema a little bit:
```
SCHEMA = {
'fields': [
{'name': 'hvfhs_license_num', 'type': ['null', 'string']},
{'name': 'dispatching_base_num', 'type': ['null', 'string']},
{'name': 'originating_base_num', 'type': ['null', 'string']},
{'name': 'request_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'on_scene_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'pickup_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'dropoff_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'PULocationID', 'type': ['null', 'long']},
{'name': 'DOLocationID', 'type': ['null', 'long']},
{'name': 'trip_miles', 'type': ['null', 'double']},
{'name': 'trip_time', 'type': ['null', 'long']},
{'name': 'base_passenger_fare', 'type': ['null', 'double']},
{'name': 'tolls', 'type': ['null', 'double']},
{'name': 'bcf', 'type': ['null', 'double']},
{'name': 'sales_tax', 'type': ['null', 'double']},
{'name': 'congestion_surcharge', 'type': ['null', 'double']},
{'name': 'airport_fee', 'type': ['null', 'long', 'double']},
{'name': 'tips', 'type': ['null', 'double']},
{'name': 'driver_pay', 'type': ['null', 'double']},
{'name': 'shared_request_flag', 'type': ['null', 'string']},
{'name': 'shared_match_flag', 'type': ['null', 'string']},
{'name': 'access_a_ride_flag', 'type': ['null', 'string']},
{'name': 'wav_request_flag', 'type': ['null', 'string']},
{'name': 'wav_match_flag', 'type': ['null', 'string']},
],
'name': 'nyc_fhv_trips',
'type': 'record',
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #25917: [Bug]: Example files for avro bitcoin do not exist in GCS
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1486123838
Thanks for trying out the example. Looks like it is broken and doesn't have an integration test exercising it. I asked for details on https://github.com/apache/beam/pull/5496. In the mean time I think we can repurpose this bug to fix this example or provide a better one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1555491833
Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1546782073
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1505695622
+1 to @liferoad's comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1547852111
Done with uploading. gs://apache-beam-samples/nyc_trip/avro contain the converted avro files. Please let me know if these files work for you. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1550426194
Let me fix this. I will update these files later. Thanks for catching this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1546782591
@liferoad @tvalentyn
I have linked a pr to add a new Avro example and remove the old one. I found some data on New York City trip data (like Uber/Lyft/etc. ride data). Here is a script I wrote to download the parquet files.
```bash
#!/bin/bash
# Downloads parquet data from nyc.gov
# Parquet data contains the trip data for NYC ride sharing services (HVFHV)
# The source website is: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
set -eu
START_YEAR=2019
START_MONTH=9
END_YEAR=2023
END_MONTH=2
OUTPUT_FOLDER="${1:-}"
if [[ -z "$OUTPUT_FOLDER" ]]; then
echo "Must provide output folder as command line argument"
exit 1
fi
mkdir -p "$OUTPUT_FOLDER"
for year in $(seq $START_YEAR $END_YEAR);
do
for month in $(seq 1 12);
do
case $year in
$START_YEAR)
if [[ $START_YEAR -eq $END_YEAR ]]; then
if [[ $month -ge $START_MONTH && $month -le $END_MONTH ]]; then
flag=1
else
flag=0
fi
else
if [[ $month -ge $START_MONTH ]]; then
flag=1
else
flag=0
fi
fi
;;
$END_YEAR)
if [[ $month -le $END_MONTH ]]; then
flag=1
else
flag=0
fi
;;
*)
flag=1
;;
esac
if [[ "$flag" -eq 1 ]]; then
name="fhvhv_tripdata_$year-$(printf "%02g" $month).parquet"
url="https://d37ci6vzurychx.cloudfront.net/trip-data/$name"
curl "$url" -o "$OUTPUT_FOLDER/$name"
fi
done
done
```
And here is the Beam code to convert the parquet files to Avro files:
```python
import argparse
import logging
import apache_beam as beam
from apache_beam.io import ReadFromParquet, WriteToAvro
from apache_beam.options.pipeline_options import PipelineOptions, SetupOptions
SCHEMA = {
'fields': [
{'name': 'hvfhs_license_num', 'type': ['null', 'string']},
{'name': 'dispatching_base_num', 'type': ['null', 'string']},
{'name': 'originating_base_num', 'type': ['null', 'string']},
{'name': 'request_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'on_scene_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'pickup_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'dropoff_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'PULocationID', 'type': ['null', 'long']},
{'name': 'DOLocationID', 'type': ['null', 'long']},
{'name': 'trip_miles', 'type': ['null', 'double']},
{'name': 'trip_time', 'type': ['null', 'long']},
{'name': 'base_passenger_fare', 'type': ['null', 'double']},
{'name': 'tolls', 'type': ['null', 'double']},
{'name': 'bcf', 'type': ['null', 'double']},
{'name': 'sales_tax', 'type': ['null', 'double']},
{'name': 'congestion_surcharge', 'type': ['null', 'double']},
{'name': 'airport_fee', 'type': ['null', 'long']},
{'name': 'tips', 'type': ['null', 'double']},
{'name': 'driver_pay', 'type': ['null', 'double']},
{'name': 'shared_request_flag', 'type': ['null', 'string']},
{'name': 'shared_match_flag', 'type': ['null', 'string']},
{'name': 'access_a_ride_flag', 'type': ['null', 'string']},
{'name': 'wav_request_flag', 'type': ['null', 'string']},
{'name': 'wav_match_flag', 'type': ['null', 'long']},
],
'name': 'nyc_fhv_trips',
'type': 'record',
}
class ConvertTimestampsDoFn(beam.DoFn):
def process(self, record: dict):
for timestamp_key in ('request_datetime', 'on_scene_datetime', 'pickup_datetime', 'dropoff_datetime'):
if record[timestamp_key]:
# convert datetime object to milliseconds
record[timestamp_key] = int(record[timestamp_key].timestamp() * 1000)
yield record
def run(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument(
'--input',
dest='input',
help='Input file to process.',
required=True,
)
parser.add_argument(
'--output',
dest='output',
help='Output file to write results to.',
required=True,
)
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = True
with beam.Pipeline(options=pipeline_options) as p:
(
p
| ReadFromParquet(known_args.input)
| beam.ParDo(ConvertTimestampsDoFn())
| WriteToAvro(known_args.output, SCHEMA, file_name_suffix='.avro')
)
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()
```
If you could run these scripts and upload the converted Avro files to one of those public facing Google Cloud Storage buckets, that would be great. And I can subsequently update the GCS URI in my pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1553964318
Thank you, the schema looks great now! And I ran the pipeline on Dataflow on all of the files and it has the expected outcome. The only minor suggestion that I have is to rename the files from `.avro-00000-of-00001.avro` suffix to simply just the extension `.avro`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1504111028
@liferoad thanks for the ping - sorry, I haven't gotten around to it yet. I think I still plan on working on it, but if you have more time and you want to work on it, it is fine by me if you take the issue. However, we don't have a dataset ready to write the example with.
@tvalentyn do you have any recommendations on publicly available datasets that we should use or it we should transform an existing sample data to avros. I searched the other sample bucket that you mentioned, and there are not any avros there.
```bash
gsutil -m ls -lh "gs://apache-beam-samples/**/*.avro"
# CommandException: One or more URLs matched no objects.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1547167137
I am uploading both parquet and avro formats under gs://apache-beam-samples/nyc_trip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
URL: https://github.com/apache/beam/issues/25917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1555065456
For the reference, here is the code I finally used to convert one parquet to avro:
```
import argparse
import logging
import apache_beam as beam
from apache_beam.io import ReadFromParquet, WriteToAvro
from apache_beam.options.pipeline_options import PipelineOptions, SetupOptions
SCHEMA = {
'fields': [
{'name': 'hvfhs_license_num', 'type': ['null', 'string']},
{'name': 'dispatching_base_num', 'type': ['null', 'string']},
{'name': 'originating_base_num', 'type': ['null', 'string']},
{'name': 'request_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'on_scene_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'pickup_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'dropoff_datetime', 'logicalType': 'timestamp-millis', 'type': ['null', 'long']},
{'name': 'PULocationID', 'type': ['null', 'long']},
{'name': 'DOLocationID', 'type': ['null', 'long']},
{'name': 'trip_miles', 'type': ['null', 'double']},
{'name': 'trip_time', 'type': ['null', 'long']},
{'name': 'base_passenger_fare', 'type': ['null', 'double']},
{'name': 'tolls', 'type': ['null', 'double']},
{'name': 'bcf', 'type': ['null', 'double']},
{'name': 'sales_tax', 'type': ['null', 'double']},
{'name': 'congestion_surcharge', 'type': ['null', 'double']},
{'name': 'airport_fee', 'type': ['null', 'double']},
{'name': 'tips', 'type': ['null', 'double']},
{'name': 'driver_pay', 'type': ['null', 'double']},
{'name': 'shared_request_flag', 'type': ['null', 'string']},
{'name': 'shared_match_flag', 'type': ['null', 'string']},
{'name': 'access_a_ride_flag', 'type': ['null', 'string']},
{'name': 'wav_request_flag', 'type': ['null', 'string']},
{'name': 'wav_match_flag', 'type': ['null', 'string']},
],
'name': 'nyc_fhv_trips',
'type': 'record',
}
class ConvertTimestampsDoFn(beam.DoFn):
def process(self, record: dict):
for timestamp_key in ('request_datetime', 'on_scene_datetime', 'pickup_datetime', 'dropoff_datetime'):
if record[timestamp_key]:
# convert datetime object to milliseconds
record[timestamp_key] = int(record[timestamp_key].timestamp() * 1000)
if record["airport_fee"]:
record["airport_fee"] = float(record["airport_fee"])
yield record
def run(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument(
'--input',
dest='input',
help='Input file to process.',
required=True,
)
parser.add_argument(
'--output',
dest='output',
help='Output file to write results to.',
required=True,
)
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = True
with beam.Pipeline(options=pipeline_options) as p:
(
p
| ReadFromParquet(known_args.input)
| beam.ParDo(ConvertTimestampsDoFn())
| WriteToAvro(known_args.output, SCHEMA, file_name_suffix='.avro')
)
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1487869206
I'll be happy to work on updating the example. For an example dataset, should I transform one of the existing datasets in the `gs://dataflow-samples/` bucket to avro? Or should I find a new dataset on Kaggle or Wikipedia or something else? And I assume I need write access to that GCS bucket. Or maybe you can upload it to that bucket? And I can just write the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1489230586
Thank you very much for offering help! I can help upload the dataset to a publicly-accessible bucket. We also have gs://apache-beam-samples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1551590700
Can you give it another try? hopefully, this time the files are correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1555917898
No problem, let us know if you need any help. Have fun!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1504063805
@nyoungstudios Do you still plan to work on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1555064033
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nyoungstudios commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "nyoungstudios (via GitHub)" <gi...@apache.org>.
nyoungstudios commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1548887069
@liferoad Thanks for uploading the files, I was checking some of the files and noticed that the type of the "airport_fee" field type doesn't look to be the union of `['null', 'long', 'double']`.
Rather, it looks to be of type record with two parts, the first field of type long and the second of type double.
Not sure the best way to quickly print the top of the file in Beam, but I ran these commands in the `spark-shell --packages org.apache.spark:spark-avro_2.12:3.1.1` on Google Dataproc.
```scala
# running these commands on a converted avro file
scala> val dfa = spark.read.format("avro").load("gs://apache-beam-samples/nyc_trip/avro/fhvhv_tripdata_2022-12.avro-00000-of-00001.avro")
scala> dfa.printSchema
root
|-- hvfhs_license_num: string (nullable = true)
|-- dispatching_base_num: string (nullable = true)
|-- originating_base_num: string (nullable = true)
|-- request_datetime: long (nullable = true)
|-- on_scene_datetime: long (nullable = true)
|-- pickup_datetime: long (nullable = true)
|-- dropoff_datetime: long (nullable = true)
|-- PULocationID: long (nullable = true)
|-- DOLocationID: long (nullable = true)
|-- trip_miles: double (nullable = true)
|-- trip_time: long (nullable = true)
|-- base_passenger_fare: double (nullable = true)
|-- tolls: double (nullable = true)
|-- bcf: double (nullable = true)
|-- sales_tax: double (nullable = true)
|-- congestion_surcharge: double (nullable = true)
|-- airport_fee: struct (nullable = true)
| |-- member0: long (nullable = true)
| |-- member1: double (nullable = true)
|-- tips: double (nullable = true)
|-- driver_pay: double (nullable = true)
|-- shared_request_flag: string (nullable = true)
|-- shared_match_flag: string (nullable = true)
|-- access_a_ride_flag: string (nullable = true)
|-- wav_request_flag: string (nullable = true)
|-- wav_match_flag: string (nullable = true)
# running these commands on the parquet file
scala> val df = spark.read.parquet("gs://apache-beam-samples/nyc_trip/parquet/fhvhv_tripdata_2022-12.parquet")
scala> df.printSchema
root
|-- hvfhs_license_num: string (nullable = true)
|-- dispatching_base_num: string (nullable = true)
|-- originating_base_num: string (nullable = true)
|-- request_datetime: timestamp (nullable = true)
|-- on_scene_datetime: timestamp (nullable = true)
|-- pickup_datetime: timestamp (nullable = true)
|-- dropoff_datetime: timestamp (nullable = true)
|-- PULocationID: long (nullable = true)
|-- DOLocationID: long (nullable = true)
|-- trip_miles: double (nullable = true)
|-- trip_time: long (nullable = true)
|-- base_passenger_fare: double (nullable = true)
|-- tolls: double (nullable = true)
|-- bcf: double (nullable = true)
|-- sales_tax: double (nullable = true)
|-- congestion_surcharge: double (nullable = true)
|-- airport_fee: double (nullable = true)
|-- tips: double (nullable = true)
|-- driver_pay: double (nullable = true)
|-- shared_request_flag: string (nullable = true)
|-- shared_match_flag: string (nullable = true)
|-- access_a_ride_flag: string (nullable = true)
|-- wav_request_flag: string (nullable = true)
|-- wav_match_flag: string (nullable = true)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on issue #25917: [Bug]: Fix or rewrite a broken Avro Example.
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #25917:
URL: https://github.com/apache/beam/issues/25917#issuecomment-1505192380
You can pick up any dataset you like. We can help you put them under `gs://apache-beam-samples`. I am going to leave this issue to you for now. Feel free to tag us if you need any help. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org