You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Ken Sedgwick <ks...@bonsai.com> on 2015/11/06 02:12:37 UTC
Writing Parquet files from C++ or Go?
Greetings,
We'd like to write Parquet files from a C++ or Go program. Has this been
done before? Are there any resources to know about?
Many thanks in advance!
Ken
--
Ken Sedgwick
Bonsai Software, Inc.
http://www.bonsai.com/ken/
(510) 269-7334
ken@bonsai.com
Public Key: http://www.bonsai.com/ken/ken.asc
GPG Fingerprint: 851E 3B07 E586 0843 9434 5CC7 4033 3B9B 3F3F 9640
ConnectException on SHOW SCHEMAS
Posted by Chris Mathews <ma...@uk2.net>.
This took me a while to discover what the issue was, so to save others time I include the following details.
If you have an HDFS storage plugin configured and execute a SHOW SCHEMAS query then you would expect to get something like like this:
sqlline -u jdbc:drill:sk=zen-drill-vm
0: jdbc:drill:sk=zen-drill-vm> show schemas;
+---------------------+
| SCHEMA_NAME |
+---------------------+
| INFORMATION_SCHEMA |
| dfs.default |
| dfs.root |
| dfs.tmp |
| hdfs.default |
| hdfs.pm |
| hdfs.root |
| hdfs.zen |
| sys |
+---------------------+
9 rows selected (0.328 seconds)
0: jdbc:drill:sk=zen-drill-vm>
If the HDFS services are not running then you will get this:
0: jdbc:drill:sk=zen-drill-vm> show schemas;
Error: SYSTEM ERROR: ConnectException: Connection refused
[Error Id: 724d9d26-8798-4165-b266-fb139a23145c on smech-vm-hdt01:31010] (state=,code=0)
0: jdbc:drill:sk=zen-drill-vm>
As you can see there is no hint as to what the problem is apart from the hostname and port number 31010, which after a while a realised was the port used by drill to access the HDFS service. I initially thought it was a problem with the web interfaces or the sqlline utility trying to connect to drill, since the drill node is the same as the HDFS node; maybe the solution to the problem would have been more obvious to me if I had drill and HDFS installed on different nodes.
Perhaps this error could be trapped, and either ignore the HDFS schemas or maybe display a warning message of some kind. This behaviour may also occur for other storage plugins such as HIVE but I do not know.
Should I raised this as a Jira for this ?
Cheers — Chris
Fwd: parquet-tools helper script for hadoop access
Posted by Chris Mathews <ma...@uk2.net>.
sorry just realised, should’ve included drill users for info not dev.
> Begin forwarded message:
>
> From: Chris Mathews <ma...@uk2.net>
> Date: 12 November 2015 at 10:43:06 GMT
> To: dev@parquet.apache.org
> Cc: dev@drill.apache.org
> Subject: parquet-tools helper script for hadoop access
>
> I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.
>
> Hopefully others will also find it useful.
>
> Cheers — Chris
>
> —— cut ——
>
> #!/usr/bin/env bash
> #
> # Author:
> # Chris Mathews
> # CTO and Co-Founder, SysMech
> # www.sysmech.co.uk
> #
>
> # Determine the path to this script's directory
> APPPATH=$( cd "$(dirname "$0")" ; pwd -P )
>
> # NOTE: pre-requsite
> # 1. HADOOP_HOME must be defined
> # 2. create link to current parquet-tools-<version>.jar library
> # eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
> #
> PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"
>
> if [ -z "${HADOOP_HOME}" ]
> then
> echo ""
> echo "warning: HADOOP_HOME not define!"
> echo ""
>
> elif (! [ -f ${PARQUET_TOOLS} ] )
> then
> echo ""
> echo "warning: file ${PARQUET_TOOLS} not found!"
> echo ""
> echo "info: create a link to the current parquet-tools library."
> echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
> echo ""
>
> else
> # Run the application
> ${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
> fi
>
> —— cut ——
>
>
parquet-tools helper script for hadoop access
Posted by Chris Mathews <ma...@uk2.net>.
I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.
Hopefully others will also find it useful.
Cheers — Chris
—— cut ——
#!/usr/bin/env bash
#
# Author:
# Chris Mathews
# CTO and Co-Founder, SysMech
# www.sysmech.co.uk
#
# Determine the path to this script's directory
APPPATH=$( cd "$(dirname "$0")" ; pwd -P )
# NOTE: pre-requsite
# 1. HADOOP_HOME must be defined
# 2. create link to current parquet-tools-<version>.jar library
# eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
#
PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"
if [ -z "${HADOOP_HOME}" ]
then
echo ""
echo "warning: HADOOP_HOME not define!"
echo ""
elif (! [ -f ${PARQUET_TOOLS} ] )
then
echo ""
echo "warning: file ${PARQUET_TOOLS} not found!"
echo ""
echo "info: create a link to the current parquet-tools library."
echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
echo ""
else
# Run the application
${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
fi
—— cut ——
parquet-tools helper script for hadoop access
Posted by Chris Mathews <ma...@uk2.net>.
I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.
Hopefully others will also find it useful.
Cheers — Chris
—— cut ——
#!/usr/bin/env bash
#
# Author:
# Chris Mathews
# CTO and Co-Founder, SysMech
# www.sysmech.co.uk
#
# Determine the path to this script's directory
APPPATH=$( cd "$(dirname "$0")" ; pwd -P )
# NOTE: pre-requsite
# 1. HADOOP_HOME must be defined
# 2. create link to current parquet-tools-<version>.jar library
# eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
#
PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"
if [ -z "${HADOOP_HOME}" ]
then
echo ""
echo "warning: HADOOP_HOME not define!"
echo ""
elif (! [ -f ${PARQUET_TOOLS} ] )
then
echo ""
echo "warning: file ${PARQUET_TOOLS} not found!"
echo ""
echo "info: create a link to the current parquet-tools library."
echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
echo ""
else
# Run the application
${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
fi
—— cut ——
Re: Writing Parquet files from C++ or Go?
Posted by Ryan Blue <bl...@cloudera.com>.
Ken,
There is a parquet-cpp project [1], but I don't think it has write
support. There is also the Impala code that you might want to check out
[2]. That's ASLv2 licensed so you could either use it directly or use it
as a guide to add a writer to the parquet-cpp project.
rb
[1]: https://github.com/apache/parquet-cpp
[2]:
https://github.com/cloudera/Impala/blob/cdh5-trunk/be/src/exec/hdfs-parquet-table-writer.cc
On 11/05/2015 05:12 PM, Ken Sedgwick wrote:
> Greetings,
>
> We'd like to write Parquet files from a C++ or Go program. Has this been
> done before? Are there any resources to know about?
>
> Many thanks in advance!
>
> Ken
>
--
Ryan Blue
Software Engineer
Cloudera, Inc.