You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Ken Sedgwick <ks...@bonsai.com> on 2015/11/06 02:12:37 UTC

Writing Parquet files from C++ or Go?

Greetings,

We'd like to write Parquet files from a C++ or Go program.  Has this been
done before?  Are there any resources to know about?

Many thanks in advance!

Ken

-- 
Ken Sedgwick
Bonsai Software, Inc.
http://www.bonsai.com/ken/
(510) 269-7334
ken@bonsai.com
Public Key: http://www.bonsai.com/ken/ken.asc
GPG Fingerprint: 851E 3B07 E586 0843 9434  5CC7 4033 3B9B 3F3F 9640

ConnectException on SHOW SCHEMAS

Posted by Chris Mathews <ma...@uk2.net>.
This took me a while to discover what the issue was, so to save others time I include the following details.

If you have an HDFS storage plugin configured and execute a SHOW SCHEMAS query then you would expect to get something like like this:

sqlline -u jdbc:drill:sk=zen-drill-vm

0: jdbc:drill:sk=zen-drill-vm> show schemas;
+---------------------+
|     SCHEMA_NAME     |
+---------------------+
| INFORMATION_SCHEMA  |
| dfs.default         |
| dfs.root            |
| dfs.tmp             |
| hdfs.default        |
| hdfs.pm             |
| hdfs.root           |
| hdfs.zen            |
| sys                 |
+---------------------+
9 rows selected (0.328 seconds)
0: jdbc:drill:sk=zen-drill-vm>

If the HDFS services are not running then you will get this:

0: jdbc:drill:sk=zen-drill-vm> show schemas;
Error: SYSTEM ERROR: ConnectException: Connection refused

[Error Id: 724d9d26-8798-4165-b266-fb139a23145c on smech-vm-hdt01:31010] (state=,code=0)
0: jdbc:drill:sk=zen-drill-vm> 

As you can see there is no hint as to what the problem is apart from the hostname and port number 31010, which after a while a realised was the port used by drill to access the HDFS service. I initially thought it was a problem with the web interfaces or the sqlline utility trying to connect to drill, since the drill node is the same as the HDFS node; maybe the solution to the problem would have been more obvious to me if I had drill and HDFS installed on different nodes.

Perhaps this error could be trapped, and either ignore the HDFS schemas or maybe display a warning message of some kind.  This behaviour may also occur for other storage plugins such as HIVE but I do not know.

Should I raised this as a Jira for this ?

Cheers — Chris




Fwd: parquet-tools helper script for hadoop access

Posted by Chris Mathews <ma...@uk2.net>.
sorry just realised, should’ve included drill users for info not dev.

> Begin forwarded message:
> 
> From: Chris Mathews <ma...@uk2.net>
> Date: 12 November 2015 at 10:43:06 GMT
> To: dev@parquet.apache.org
> Cc: dev@drill.apache.org
> Subject: parquet-tools helper script for hadoop access
> 
> I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is  along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.
> 
> Hopefully others will also find it useful.
> 
> Cheers — Chris
> 
> —— cut ——
> 
> #!/usr/bin/env bash
> #
> # Author:
> #     Chris Mathews
> #     CTO and Co-Founder, SysMech
> #     www.sysmech.co.uk
> #
> 
> # Determine the path to this script's directory
> APPPATH=$( cd "$(dirname "$0")" ; pwd -P )
> 
> # NOTE: pre-requsite
> #  1.   HADOOP_HOME must be defined
> #  2.   create link to current parquet-tools-<version>.jar library
> #       eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
> #
> PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"
> 
> if [ -z "${HADOOP_HOME}" ]
> then
>    echo ""
>    echo "warning: HADOOP_HOME not define!"
>    echo ""
> 
> elif (! [ -f ${PARQUET_TOOLS} ] )
> then
>    echo ""
>    echo "warning: file ${PARQUET_TOOLS} not found!"
>    echo ""
>    echo "info: create a link to the current parquet-tools library."
>    echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
>    echo ""
> 
> else
>    # Run the application
>    ${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
> fi
> 
> —— cut ——
> 
> 


parquet-tools helper script for hadoop access

Posted by Chris Mathews <ma...@uk2.net>.
I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is  along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.

Hopefully others will also find it useful.

Cheers — Chris

—— cut ——

#!/usr/bin/env bash
#
# Author:
#     Chris Mathews
#     CTO and Co-Founder, SysMech
#     www.sysmech.co.uk
#

# Determine the path to this script's directory
APPPATH=$( cd "$(dirname "$0")" ; pwd -P )

# NOTE: pre-requsite
#  1.   HADOOP_HOME must be defined
#  2.   create link to current parquet-tools-<version>.jar library
#       eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
#
PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"

if [ -z "${HADOOP_HOME}" ]
then
    echo ""
    echo "warning: HADOOP_HOME not define!"
    echo ""

elif (! [ -f ${PARQUET_TOOLS} ] )
then
    echo ""
    echo "warning: file ${PARQUET_TOOLS} not found!"
    echo ""
    echo "info: create a link to the current parquet-tools library."
    echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
    echo ""

else
    # Run the application
    ${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
fi

—— cut ——



parquet-tools helper script for hadoop access

Posted by Chris Mathews <ma...@uk2.net>.
I’ve written an additional parquet-tools helper script to access files in the hadoop environment. It is  along the lines of the existing parquet-schema etc. scripts provided as part of the parquet-tools distribution, and is designed to live in the same location as these scripts.

Hopefully others will also find it useful.

Cheers — Chris

—— cut ——

#!/usr/bin/env bash
#
# Author:
#     Chris Mathews
#     CTO and Co-Founder, SysMech
#     www.sysmech.co.uk
#

# Determine the path to this script's directory
APPPATH=$( cd "$(dirname "$0")" ; pwd -P )

# NOTE: pre-requsite
#  1.   HADOOP_HOME must be defined
#  2.   create link to current parquet-tools-<version>.jar library
#       eg: ln -s parquet-tools-1.8.1.jar parquet-tools.jar
#
PARQUET_TOOLS="${APPPATH}/lib/parquet-tools.jar"

if [ -z "${HADOOP_HOME}" ]
then
    echo ""
    echo "warning: HADOOP_HOME not define!"
    echo ""

elif (! [ -f ${PARQUET_TOOLS} ] )
then
    echo ""
    echo "warning: file ${PARQUET_TOOLS} not found!"
    echo ""
    echo "info: create a link to the current parquet-tools library."
    echo "info: e.g.: ln -s ${APPPATH}/lib/parquet-tools-1.8.1.jar ${APPPATH}/lib/parquet-tools.jar"
    echo ""

else
    # Run the application
    ${HADOOP_HOME}/bin/hadoop jar ${PARQUET_TOOLS} "$@"
fi

—— cut ——



Re: Writing Parquet files from C++ or Go?

Posted by Ryan Blue <bl...@cloudera.com>.
Ken,

There is a parquet-cpp project [1], but I don't think it has write 
support. There is also the Impala code that you might want to check out 
[2]. That's ASLv2 licensed so you could either use it directly or use it 
as a guide to add a writer to the parquet-cpp project.

rb

[1]: https://github.com/apache/parquet-cpp
[2]: 
https://github.com/cloudera/Impala/blob/cdh5-trunk/be/src/exec/hdfs-parquet-table-writer.cc

On 11/05/2015 05:12 PM, Ken Sedgwick wrote:
> Greetings,
>
> We'd like to write Parquet files from a C++ or Go program.  Has this been
> done before?  Are there any resources to know about?
>
> Many thanks in advance!
>
> Ken
>


-- 
Ryan Blue
Software Engineer
Cloudera, Inc.