You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2019/06/12 00:29:55 UTC
[arrow-site] branch asf-site updated: Remove very stale docs/latest
(#6)
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1911826 Remove very stale docs/latest (#6)
1911826 is described below
commit 1911826bfab3ff6cf524994d3ccd6c2a453371e5
Author: Neal Richardson <ne...@gmail.com>
AuthorDate: Tue Jun 11 17:29:51 2019 -0700
Remove very stale docs/latest (#6)
---
docs/latest/.buildinfo | 4 -
docs/latest/.doctrees/cpp/api.doctree | Bin 5500 -> 0 bytes
docs/latest/.doctrees/cpp/api/array.doctree | Bin 228714 -> 0 bytes
docs/latest/.doctrees/cpp/api/builder.doctree | Bin 304183 -> 0 bytes
docs/latest/.doctrees/cpp/api/datatype.doctree | Bin 514453 -> 0 bytes
docs/latest/.doctrees/cpp/api/memory.doctree | Bin 319930 -> 0 bytes
docs/latest/.doctrees/cpp/api/support.doctree | Bin 84789 -> 0 bytes
docs/latest/.doctrees/cpp/api/table.doctree | Bin 289493 -> 0 bytes
docs/latest/.doctrees/cpp/arrays.doctree | Bin 31694 -> 0 bytes
docs/latest/.doctrees/cpp/conventions.doctree | Bin 15269 -> 0 bytes
docs/latest/.doctrees/cpp/datatypes.doctree | Bin 15883 -> 0 bytes
docs/latest/.doctrees/cpp/examples.doctree | Bin 25067 -> 0 bytes
docs/latest/.doctrees/cpp/getting_started.doctree | Bin 5652 -> 0 bytes
docs/latest/.doctrees/cpp/index.doctree | Bin 6031 -> 0 bytes
docs/latest/.doctrees/cpp/memory.doctree | Bin 23767 -> 0 bytes
docs/latest/.doctrees/cpp/overview.doctree | Bin 17259 -> 0 bytes
docs/latest/.doctrees/cpp/tables.doctree | Bin 16665 -> 0 bytes
docs/latest/.doctrees/environment.pickle | Bin 1570787 -> 0 bytes
docs/latest/.doctrees/format/Guidelines.doctree | Bin 8864 -> 0 bytes
docs/latest/.doctrees/format/IPC.doctree | Bin 33400 -> 0 bytes
docs/latest/.doctrees/format/Layout.doctree | Bin 87213 -> 0 bytes
docs/latest/.doctrees/format/Metadata.doctree | Bin 47679 -> 0 bytes
docs/latest/.doctrees/format/README.doctree | Bin 12685 -> 0 bytes
docs/latest/.doctrees/index.doctree | Bin 6597 -> 0 bytes
docs/latest/.doctrees/python/api.doctree | Bin 242246 -> 0 bytes
docs/latest/.doctrees/python/benchmarks.doctree | Bin 11684 -> 0 bytes
docs/latest/.doctrees/python/csv.doctree | Bin 18540 -> 0 bytes
docs/latest/.doctrees/python/data.doctree | Bin 73774 -> 0 bytes
docs/latest/.doctrees/python/development.doctree | Bin 45651 -> 0 bytes
docs/latest/.doctrees/python/extending.doctree | Bin 150306 -> 0 bytes
docs/latest/.doctrees/python/filesystems.doctree | Bin 36618 -> 0 bytes
.../python/generated/pyarrow.Array.doctree | Bin 74349 -> 0 bytes
.../python/generated/pyarrow.ArrayValue.doctree | Bin 4990 -> 0 bytes
.../python/generated/pyarrow.BinaryArray.doctree | Bin 75561 -> 0 bytes
.../python/generated/pyarrow.BinaryValue.doctree | Bin 10047 -> 0 bytes
.../python/generated/pyarrow.BooleanArray.doctree | Bin 75757 -> 0 bytes
.../python/generated/pyarrow.BooleanValue.doctree | Bin 7856 -> 0 bytes
.../python/generated/pyarrow.Buffer.doctree | Bin 23082 -> 0 bytes
.../generated/pyarrow.BufferOutputStream.doctree | Bin 72191 -> 0 bytes
.../python/generated/pyarrow.BufferReader.doctree | Bin 68850 -> 0 bytes
.../python/generated/pyarrow.ChunkedArray.doctree | Bin 49499 -> 0 bytes
.../python/generated/pyarrow.Column.doctree | Bin 52189 -> 0 bytes
.../pyarrow.CompressedInputStream.doctree | Bin 72196 -> 0 bytes
.../pyarrow.CompressedOutputStream.doctree | Bin 72461 -> 0 bytes
.../python/generated/pyarrow.DataType.doctree | Bin 15959 -> 0 bytes
.../python/generated/pyarrow.Date32Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.Date32Value.doctree | Bin 7937 -> 0 bytes
.../python/generated/pyarrow.Date64Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.Date64Value.doctree | Bin 7937 -> 0 bytes
.../generated/pyarrow.Decimal128Array.doctree | Bin 76405 -> 0 bytes
.../python/generated/pyarrow.DecimalValue.doctree | Bin 7871 -> 0 bytes
.../generated/pyarrow.DictionaryArray.doctree | Bin 91908 -> 0 bytes
.../python/generated/pyarrow.DoubleValue.doctree | Bin 7832 -> 0 bytes
.../python/generated/pyarrow.Field.doctree | Bin 26141 -> 0 bytes
.../generated/pyarrow.FixedSizeBinaryArray.doctree | Bin 77325 -> 0 bytes
.../generated/pyarrow.FixedSizeBinaryValue.doctree | Bin 8128 -> 0 bytes
.../pyarrow.FixedSizeBufferWriter.doctree | Bin 76498 -> 0 bytes
.../python/generated/pyarrow.FloatValue.doctree | Bin 7803 -> 0 bytes
.../generated/pyarrow.FloatingPointArray.doctree | Bin 76961 -> 0 bytes
.../generated/pyarrow.HadoopFileSystem.cat.doctree | Bin 4570 -> 0 bytes
.../pyarrow.HadoopFileSystem.chmod.doctree | Bin 5783 -> 0 bytes
.../pyarrow.HadoopFileSystem.chown.doctree | Bin 7350 -> 0 bytes
.../pyarrow.HadoopFileSystem.delete.doctree | Bin 6543 -> 0 bytes
.../generated/pyarrow.HadoopFileSystem.df.doctree | Bin 4558 -> 0 bytes
.../pyarrow.HadoopFileSystem.disk_usage.doctree | Bin 5613 -> 0 bytes
.../pyarrow.HadoopFileSystem.download.doctree | Bin 3957 -> 0 bytes
.../pyarrow.HadoopFileSystem.exists.doctree | Bin 4515 -> 0 bytes
.../pyarrow.HadoopFileSystem.get_capacity.doctree | Bin 4719 -> 0 bytes
...pyarrow.HadoopFileSystem.get_space_used.doctree | Bin 4729 -> 0 bytes
.../pyarrow.HadoopFileSystem.info.doctree | Bin 5540 -> 0 bytes
.../generated/pyarrow.HadoopFileSystem.ls.doctree | Bin 7247 -> 0 bytes
.../pyarrow.HadoopFileSystem.mkdir.doctree | Bin 5820 -> 0 bytes
.../pyarrow.HadoopFileSystem.open.doctree | Bin 6215 -> 0 bytes
.../pyarrow.HadoopFileSystem.rename.doctree | Bin 6131 -> 0 bytes
.../generated/pyarrow.HadoopFileSystem.rm.doctree | Bin 3943 -> 0 bytes
.../pyarrow.HadoopFileSystem.upload.doctree | Bin 4224 -> 0 bytes
.../python/generated/pyarrow.HdfsFile.doctree | Bin 69447 -> 0 bytes
.../python/generated/pyarrow.Int16Array.doctree | Bin 75393 -> 0 bytes
.../python/generated/pyarrow.Int16Value.doctree | Bin 7793 -> 0 bytes
.../python/generated/pyarrow.Int32Array.doctree | Bin 75393 -> 0 bytes
.../python/generated/pyarrow.Int32Value.doctree | Bin 7793 -> 0 bytes
.../python/generated/pyarrow.Int64Array.doctree | Bin 75393 -> 0 bytes
.../python/generated/pyarrow.Int64Value.doctree | Bin 7793 -> 0 bytes
.../python/generated/pyarrow.Int8Array.doctree | Bin 75197 -> 0 bytes
.../python/generated/pyarrow.Int8Value.doctree | Bin 7764 -> 0 bytes
.../python/generated/pyarrow.IntegerArray.doctree | Bin 75785 -> 0 bytes
.../python/generated/pyarrow.ListArray.doctree | Bin 83695 -> 0 bytes
.../python/generated/pyarrow.ListValue.doctree | Bin 9954 -> 0 bytes
.../generated/pyarrow.LocalFileSystem.doctree | Bin 62271 -> 0 bytes
.../generated/pyarrow.MemoryMappedFile.doctree | Bin 73623 -> 0 bytes
.../python/generated/pyarrow.MemoryPool.doctree | Bin 10758 -> 0 bytes
.../python/generated/pyarrow.Message.doctree | Bin 20378 -> 0 bytes
.../python/generated/pyarrow.MessageReader.doctree | Bin 10260 -> 0 bytes
.../.doctrees/python/generated/pyarrow.NA.doctree | Bin 3487 -> 0 bytes
.../python/generated/pyarrow.NativeFile.doctree | Bin 66990 -> 0 bytes
.../python/generated/pyarrow.NullArray.doctree | Bin 75169 -> 0 bytes
.../python/generated/pyarrow.NumericArray.doctree | Bin 75757 -> 0 bytes
.../python/generated/pyarrow.OSFile.doctree | Bin 65841 -> 0 bytes
.../python/generated/pyarrow.PythonFile.doctree | Bin 64697 -> 0 bytes
.../python/generated/pyarrow.RecordBatch.doctree | Bin 68957 -> 0 bytes
.../pyarrow.RecordBatchFileReader.doctree | Bin 27844 -> 0 bytes
.../pyarrow.RecordBatchFileWriter.doctree | Bin 24795 -> 0 bytes
.../pyarrow.RecordBatchStreamReader.doctree | Bin 24410 -> 0 bytes
.../pyarrow.RecordBatchStreamWriter.doctree | Bin 24915 -> 0 bytes
.../generated/pyarrow.ResizableBuffer.doctree | Bin 28900 -> 0 bytes
.../python/generated/pyarrow.Scalar.doctree | Bin 4857 -> 0 bytes
.../python/generated/pyarrow.Schema.doctree | Bin 59692 -> 0 bytes
.../generated/pyarrow.SerializationContext.doctree | Bin 37025 -> 0 bytes
.../generated/pyarrow.SerializedPyObject.doctree | Bin 24266 -> 0 bytes
.../python/generated/pyarrow.StringArray.doctree | Bin 75809 -> 0 bytes
.../python/generated/pyarrow.StringValue.doctree | Bin 7877 -> 0 bytes
.../python/generated/pyarrow.Table.doctree | Bin 104750 -> 0 bytes
.../python/generated/pyarrow.Tensor.doctree | Bin 22258 -> 0 bytes
.../python/generated/pyarrow.Time32Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.Time64Array.doctree | Bin 75589 -> 0 bytes
.../generated/pyarrow.TimestampArray.doctree | Bin 76177 -> 0 bytes
.../generated/pyarrow.TimestampValue.doctree | Bin 10479 -> 0 bytes
.../python/generated/pyarrow.UInt16Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.UInt16Value.doctree | Bin 7822 -> 0 bytes
.../python/generated/pyarrow.UInt32Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.UInt32Value.doctree | Bin 7822 -> 0 bytes
.../python/generated/pyarrow.UInt64Array.doctree | Bin 75589 -> 0 bytes
.../python/generated/pyarrow.UInt64Value.doctree | Bin 7822 -> 0 bytes
.../python/generated/pyarrow.UInt8Array.doctree | Bin 75393 -> 0 bytes
.../python/generated/pyarrow.UInt8Value.doctree | Bin 7793 -> 0 bytes
.../generated/pyarrow.allocate_buffer.doctree | Bin 8103 -> 0 bytes
.../python/generated/pyarrow.binary.doctree | Bin 5900 -> 0 bytes
.../python/generated/pyarrow.bool_.doctree | Bin 3646 -> 0 bytes
.../python/generated/pyarrow.chunked_array.doctree | Bin 6764 -> 0 bytes
.../python/generated/pyarrow.compress.doctree | Bin 10205 -> 0 bytes
.../python/generated/pyarrow.concat_tables.doctree | Bin 6131 -> 0 bytes
.../python/generated/pyarrow.cpu_count.doctree | Bin 5315 -> 0 bytes
.../generated/pyarrow.create_memory_map.doctree | Bin 6294 -> 0 bytes
.../generated/pyarrow.csv.ConvertOptions.doctree | Bin 16250 -> 0 bytes
.../generated/pyarrow.csv.ParseOptions.doctree | Bin 36155 -> 0 bytes
.../generated/pyarrow.csv.ReadOptions.doctree | Bin 14142 -> 0 bytes
.../python/generated/pyarrow.csv.read_csv.doctree | Bin 12378 -> 0 bytes
.../python/generated/pyarrow.date32.doctree | Bin 3712 -> 0 bytes
.../python/generated/pyarrow.date64.doctree | Bin 3728 -> 0 bytes
.../python/generated/pyarrow.decimal128.doctree | Bin 6388 -> 0 bytes
.../python/generated/pyarrow.decompress.doctree | Bin 11628 -> 0 bytes
.../generated/pyarrow.default_memory_pool.doctree | Bin 3644 -> 0 bytes
.../python/generated/pyarrow.deserialize.doctree | Bin 6996 -> 0 bytes
.../pyarrow.deserialize_components.doctree | Bin 7133 -> 0 bytes
.../generated/pyarrow.deserialize_from.doctree | Bin 7622 -> 0 bytes
.../python/generated/pyarrow.dictionary.doctree | Bin 7165 -> 0 bytes
.../generated/pyarrow.feather.read_feather.doctree | Bin 9152 -> 0 bytes
.../pyarrow.feather.write_feather.doctree | Bin 6312 -> 0 bytes
.../python/generated/pyarrow.float16.doctree | Bin 3677 -> 0 bytes
.../python/generated/pyarrow.float32.doctree | Bin 3681 -> 0 bytes
.../python/generated/pyarrow.float64.doctree | Bin 3681 -> 0 bytes
.../generated/pyarrow.foreign_buffer.doctree | Bin 5472 -> 0 bytes
.../generated/pyarrow.from_numpy_dtype.doctree | Bin 3940 -> 0 bytes
.../python/generated/pyarrow.get_include.doctree | Bin 4400 -> 0 bytes
.../python/generated/pyarrow.get_libraries.doctree | Bin 4683 -> 0 bytes
.../generated/pyarrow.get_library_dirs.doctree | Bin 4505 -> 0 bytes
.../pyarrow.get_record_batch_size.doctree | Bin 4130 -> 0 bytes
.../generated/pyarrow.get_tensor_size.doctree | Bin 4018 -> 0 bytes
.../python/generated/pyarrow.hdfs.connect.doctree | Bin 13873 -> 0 bytes
.../python/generated/pyarrow.input_stream.doctree | Bin 8074 -> 0 bytes
.../python/generated/pyarrow.int16.doctree | Bin 3641 -> 0 bytes
.../python/generated/pyarrow.int32.doctree | Bin 3641 -> 0 bytes
.../python/generated/pyarrow.int64.doctree | Bin 3641 -> 0 bytes
.../python/generated/pyarrow.int8.doctree | Bin 3626 -> 0 bytes
.../python/generated/pyarrow.ipc.open_file.doctree | Bin 8488 -> 0 bytes
.../generated/pyarrow.ipc.open_stream.doctree | Bin 8350 -> 0 bytes
.../python/generated/pyarrow.list_.doctree | Bin 5794 -> 0 bytes
.../pyarrow.log_memory_allocations.doctree | Bin 5384 -> 0 bytes
.../python/generated/pyarrow.memory_map.doctree | Bin 6800 -> 0 bytes
.../python/generated/pyarrow.null.doctree | Bin 3612 -> 0 bytes
.../python/generated/pyarrow.open_file.doctree | Bin 4427 -> 0 bytes
.../python/generated/pyarrow.open_stream.doctree | Bin 4465 -> 0 bytes
.../python/generated/pyarrow.output_stream.doctree | Bin 8084 -> 0 bytes
.../pyarrow.parquet.ParquetDataset.doctree | Bin 39273 -> 0 bytes
.../generated/pyarrow.parquet.ParquetFile.doctree | Bin 43504 -> 0 bytes
.../pyarrow.parquet.ParquetWriter.doctree | Bin 27461 -> 0 bytes
.../pyarrow.parquet.read_metadata.doctree | Bin 8190 -> 0 bytes
.../generated/pyarrow.parquet.read_pandas.doctree | Bin 12447 -> 0 bytes
.../generated/pyarrow.parquet.read_schema.doctree | Bin 8160 -> 0 bytes
.../generated/pyarrow.parquet.read_table.doctree | Bin 13812 -> 0 bytes
.../pyarrow.parquet.write_metadata.doctree | Bin 12205 -> 0 bytes
.../generated/pyarrow.parquet.write_table.doctree | Bin 17679 -> 0 bytes
.../pyarrow.parquet.write_to_dataset.doctree | Bin 14681 -> 0 bytes
.../generated/pyarrow.plasma.ObjectID.doctree | Bin 11769 -> 0 bytes
.../generated/pyarrow.plasma.PlasmaBuffer.doctree | Bin 27887 -> 0 bytes
.../generated/pyarrow.plasma.PlasmaClient.doctree | Bin 105858 -> 0 bytes
.../python/generated/pyarrow.py_buffer.doctree | Bin 3898 -> 0 bytes
.../python/generated/pyarrow.read_message.doctree | Bin 6117 -> 0 bytes
.../generated/pyarrow.read_record_batch.doctree | Bin 6580 -> 0 bytes
.../generated/pyarrow.read_serialized.doctree | Bin 6726 -> 0 bytes
.../python/generated/pyarrow.read_tensor.doctree | Bin 5816 -> 0 bytes
.../python/generated/pyarrow.serialize.doctree | Bin 6875 -> 0 bytes
.../python/generated/pyarrow.serialize_to.doctree | Bin 7453 -> 0 bytes
.../python/generated/pyarrow.set_cpu_count.doctree | Bin 3948 -> 0 bytes
.../generated/pyarrow.set_memory_pool.doctree | Bin 3753 -> 0 bytes
.../python/generated/pyarrow.string.doctree | Bin 3660 -> 0 bytes
.../python/generated/pyarrow.struct.doctree | Bin 6778 -> 0 bytes
.../python/generated/pyarrow.time32.doctree | Bin 5177 -> 0 bytes
.../python/generated/pyarrow.time64.doctree | Bin 5194 -> 0 bytes
.../python/generated/pyarrow.timestamp.doctree | Bin 7408 -> 0 bytes
.../pyarrow.total_allocated_bytes.doctree | Bin 4033 -> 0 bytes
.../generated/pyarrow.types.is_binary.doctree | Bin 4501 -> 0 bytes
.../generated/pyarrow.types.is_boolean.doctree | Bin 4486 -> 0 bytes
.../python/generated/pyarrow.types.is_date.doctree | Bin 4435 -> 0 bytes
.../generated/pyarrow.types.is_date32.doctree | Bin 4483 -> 0 bytes
.../generated/pyarrow.types.is_date64.doctree | Bin 4499 -> 0 bytes
.../generated/pyarrow.types.is_decimal.doctree | Bin 4486 -> 0 bytes
.../generated/pyarrow.types.is_dictionary.doctree | Bin 4553 -> 0 bytes
.../pyarrow.types.is_fixed_size_binary.doctree | Bin 4660 -> 0 bytes
.../generated/pyarrow.types.is_float16.doctree | Bin 4522 -> 0 bytes
.../generated/pyarrow.types.is_float32.doctree | Bin 4526 -> 0 bytes
.../generated/pyarrow.types.is_float64.doctree | Bin 4526 -> 0 bytes
.../generated/pyarrow.types.is_floating.doctree | Bin 4531 -> 0 bytes
.../generated/pyarrow.types.is_int16.doctree | Bin 4454 -> 0 bytes
.../generated/pyarrow.types.is_int32.doctree | Bin 4454 -> 0 bytes
.../generated/pyarrow.types.is_int64.doctree | Bin 4454 -> 0 bytes
.../python/generated/pyarrow.types.is_int8.doctree | Bin 4437 -> 0 bytes
.../generated/pyarrow.types.is_integer.doctree | Bin 4490 -> 0 bytes
.../python/generated/pyarrow.types.is_list.doctree | Bin 4435 -> 0 bytes
.../python/generated/pyarrow.types.is_map.doctree | Bin 4434 -> 0 bytes
.../generated/pyarrow.types.is_nested.doctree | Bin 4469 -> 0 bytes
.../python/generated/pyarrow.types.is_null.doctree | Bin 4435 -> 0 bytes
.../pyarrow.types.is_signed_integer.doctree | Bin 4611 -> 0 bytes
.../generated/pyarrow.types.is_string.doctree | Bin 4495 -> 0 bytes
.../generated/pyarrow.types.is_struct.doctree | Bin 4469 -> 0 bytes
.../generated/pyarrow.types.is_temporal.doctree | Bin 4551 -> 0 bytes
.../python/generated/pyarrow.types.is_time.doctree | Bin 4435 -> 0 bytes
.../generated/pyarrow.types.is_time32.doctree | Bin 4469 -> 0 bytes
.../generated/pyarrow.types.is_time64.doctree | Bin 4469 -> 0 bytes
.../generated/pyarrow.types.is_timestamp.doctree | Bin 4520 -> 0 bytes
.../generated/pyarrow.types.is_uint16.doctree | Bin 4471 -> 0 bytes
.../generated/pyarrow.types.is_uint32.doctree | Bin 4471 -> 0 bytes
.../generated/pyarrow.types.is_uint64.doctree | Bin 4471 -> 0 bytes
.../generated/pyarrow.types.is_uint8.doctree | Bin 4454 -> 0 bytes
.../generated/pyarrow.types.is_unicode.doctree | Bin 4418 -> 0 bytes
.../generated/pyarrow.types.is_union.doctree | Bin 4452 -> 0 bytes
.../pyarrow.types.is_unsigned_integer.doctree | Bin 4645 -> 0 bytes
.../python/generated/pyarrow.uint16.doctree | Bin 3660 -> 0 bytes
.../python/generated/pyarrow.uint32.doctree | Bin 3660 -> 0 bytes
.../python/generated/pyarrow.uint64.doctree | Bin 3660 -> 0 bytes
.../python/generated/pyarrow.uint8.doctree | Bin 3643 -> 0 bytes
.../python/generated/pyarrow.utf8.doctree | Bin 3592 -> 0 bytes
.../python/generated/pyarrow.write_tensor.doctree | Bin 6467 -> 0 bytes
.../.doctrees/python/getting_involved.doctree | Bin 9584 -> 0 bytes
docs/latest/.doctrees/python/index.doctree | Bin 7584 -> 0 bytes
docs/latest/.doctrees/python/install.doctree | Bin 9119 -> 0 bytes
docs/latest/.doctrees/python/ipc.doctree | Bin 55983 -> 0 bytes
docs/latest/.doctrees/python/memory.doctree | Bin 51539 -> 0 bytes
docs/latest/.doctrees/python/numpy.doctree | Bin 11592 -> 0 bytes
docs/latest/.doctrees/python/pandas.doctree | Bin 28498 -> 0 bytes
docs/latest/.doctrees/python/parquet.doctree | Bin 62147 -> 0 bytes
docs/latest/.doctrees/python/plasma.doctree | Bin 58088 -> 0 bytes
docs/latest/_modules/index.html | 226 -
docs/latest/_modules/pyarrow.html | 478 -
docs/latest/_modules/pyarrow/feather.html | 455 -
docs/latest/_modules/pyarrow/filesystem.html | 634 --
docs/latest/_modules/pyarrow/hdfs.html | 428 -
docs/latest/_modules/pyarrow/ipc.html | 408 -
docs/latest/_modules/pyarrow/parquet.html | 1543 ---
docs/latest/_modules/pyarrow/types.html | 516 -
docs/latest/_sources/cpp/api.rst.txt | 30 -
docs/latest/_sources/cpp/api/array.rst.txt | 92 -
docs/latest/_sources/cpp/api/builder.rst.txt | 56 -
docs/latest/_sources/cpp/api/datatype.rst.txt | 148 -
docs/latest/_sources/cpp/api/memory.rst.txt | 90 -
docs/latest/_sources/cpp/api/support.rst.txt | 29 -
docs/latest/_sources/cpp/api/table.rst.txt | 52 -
docs/latest/_sources/cpp/arrays.rst.txt | 211 -
docs/latest/_sources/cpp/conventions.rst.txt | 91 -
docs/latest/_sources/cpp/datatypes.rst.txt | 65 -
docs/latest/_sources/cpp/examples.rst.txt | 30 -
docs/latest/_sources/cpp/getting_started.rst.txt | 31 -
docs/latest/_sources/cpp/index.rst.txt | 32 -
docs/latest/_sources/cpp/memory.rst.txt | 127 -
docs/latest/_sources/cpp/overview.rst.txt | 93 -
docs/latest/_sources/cpp/tables.rst.txt | 87 -
docs/latest/_sources/format/Guidelines.rst.txt | 43 -
docs/latest/_sources/format/IPC.rst.txt | 237 -
docs/latest/_sources/format/Layout.rst.txt | 664 --
docs/latest/_sources/format/Metadata.rst.txt | 396 -
docs/latest/_sources/format/README.rst.txt | 53 -
docs/latest/_sources/index.rst.txt | 42 -
docs/latest/_sources/python/api.rst.txt | 399 -
docs/latest/_sources/python/benchmarks.rst.txt | 53 -
docs/latest/_sources/python/csv.rst.txt | 92 -
docs/latest/_sources/python/data.rst.txt | 434 -
docs/latest/_sources/python/development.rst.txt | 409 -
docs/latest/_sources/python/extending.rst.txt | 361 -
docs/latest/_sources/python/filesystems.rst.txt | 93 -
.../python/generated/pyarrow.Array.rst.txt | 43 -
.../python/generated/pyarrow.ArrayValue.rst.txt | 16 -
.../python/generated/pyarrow.BinaryArray.rst.txt | 43 -
.../python/generated/pyarrow.BinaryValue.rst.txt | 23 -
.../python/generated/pyarrow.BooleanArray.rst.txt | 43 -
.../python/generated/pyarrow.BooleanValue.rst.txt | 22 -
.../python/generated/pyarrow.Buffer.rst.txt | 33 -
.../generated/pyarrow.BufferOutputStream.rst.txt | 51 -
.../python/generated/pyarrow.BufferReader.rst.txt | 50 -
.../python/generated/pyarrow.ChunkedArray.rst.txt | 40 -
.../python/generated/pyarrow.Column.rst.txt | 41 -
.../pyarrow.CompressedInputStream.rst.txt | 50 -
.../pyarrow.CompressedOutputStream.rst.txt | 50 -
.../python/generated/pyarrow.DataType.rst.txt | 30 -
.../python/generated/pyarrow.Date32Array.rst.txt | 43 -
.../python/generated/pyarrow.Date32Value.rst.txt | 22 -
.../python/generated/pyarrow.Date64Array.rst.txt | 43 -
.../python/generated/pyarrow.Date64Value.rst.txt | 22 -
.../generated/pyarrow.Decimal128Array.rst.txt | 43 -
.../python/generated/pyarrow.DecimalValue.rst.txt | 22 -
.../generated/pyarrow.DictionaryArray.rst.txt | 46 -
.../python/generated/pyarrow.DoubleValue.rst.txt | 22 -
.../python/generated/pyarrow.Field.rst.txt | 34 -
.../generated/pyarrow.FixedSizeBinaryArray.rst.txt | 43 -
.../generated/pyarrow.FixedSizeBinaryValue.rst.txt | 22 -
.../pyarrow.FixedSizeBufferWriter.rst.txt | 53 -
.../python/generated/pyarrow.FloatValue.rst.txt | 22 -
.../generated/pyarrow.FloatingPointArray.rst.txt | 43 -
.../generated/pyarrow.HadoopFileSystem.cat.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.chmod.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.chown.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.delete.rst.txt | 6 -
.../generated/pyarrow.HadoopFileSystem.df.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.disk_usage.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.download.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.exists.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.get_capacity.rst.txt | 6 -
...pyarrow.HadoopFileSystem.get_space_used.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.info.rst.txt | 6 -
.../generated/pyarrow.HadoopFileSystem.ls.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.mkdir.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.open.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.rename.rst.txt | 6 -
.../generated/pyarrow.HadoopFileSystem.rm.rst.txt | 6 -
.../pyarrow.HadoopFileSystem.upload.rst.txt | 6 -
.../python/generated/pyarrow.HdfsFile.rst.txt | 52 -
.../python/generated/pyarrow.Int16Array.rst.txt | 43 -
.../python/generated/pyarrow.Int16Value.rst.txt | 22 -
.../python/generated/pyarrow.Int32Array.rst.txt | 43 -
.../python/generated/pyarrow.Int32Value.rst.txt | 22 -
.../python/generated/pyarrow.Int64Array.rst.txt | 43 -
.../python/generated/pyarrow.Int64Value.rst.txt | 22 -
.../python/generated/pyarrow.Int8Array.rst.txt | 43 -
.../python/generated/pyarrow.Int8Value.rst.txt | 22 -
.../python/generated/pyarrow.IntegerArray.rst.txt | 43 -
.../python/generated/pyarrow.ListArray.rst.txt | 45 -
.../python/generated/pyarrow.ListValue.rst.txt | 28 -
.../generated/pyarrow.LocalFileSystem.rst.txt | 43 -
.../generated/pyarrow.MemoryMappedFile.rst.txt | 52 -
.../python/generated/pyarrow.MemoryPool.rst.txt | 23 -
.../python/generated/pyarrow.Message.rst.txt | 31 -
.../python/generated/pyarrow.MessageReader.rst.txt | 23 -
.../_sources/python/generated/pyarrow.NA.rst.txt | 6 -
.../python/generated/pyarrow.NativeFile.rst.txt | 50 -
.../python/generated/pyarrow.NullArray.rst.txt | 43 -
.../python/generated/pyarrow.NumericArray.rst.txt | 43 -
.../python/generated/pyarrow.OSFile.rst.txt | 50 -
.../python/generated/pyarrow.PythonFile.rst.txt | 50 -
.../python/generated/pyarrow.RecordBatch.rst.txt | 39 -
.../pyarrow.RecordBatchFileReader.rst.txt | 33 -
.../pyarrow.RecordBatchFileWriter.rst.txt | 26 -
.../pyarrow.RecordBatchStreamReader.rst.txt | 32 -
.../pyarrow.RecordBatchStreamWriter.rst.txt | 26 -
.../generated/pyarrow.ResizableBuffer.rst.txt | 34 -
.../python/generated/pyarrow.Scalar.rst.txt | 16 -
.../python/generated/pyarrow.Schema.rst.txt | 41 -
.../generated/pyarrow.SerializationContext.rst.txt | 28 -
.../generated/pyarrow.SerializedPyObject.rst.txt | 33 -
.../python/generated/pyarrow.StringArray.rst.txt | 43 -
.../python/generated/pyarrow.StringValue.rst.txt | 22 -
.../python/generated/pyarrow.Table.rst.txt | 48 -
.../python/generated/pyarrow.Tensor.rst.txt | 36 -
.../python/generated/pyarrow.Time32Array.rst.txt | 43 -
.../python/generated/pyarrow.Time64Array.rst.txt | 43 -
.../generated/pyarrow.TimestampArray.rst.txt | 43 -
.../generated/pyarrow.TimestampValue.rst.txt | 28 -
.../python/generated/pyarrow.UInt16Array.rst.txt | 43 -
.../python/generated/pyarrow.UInt16Value.rst.txt | 22 -
.../python/generated/pyarrow.UInt32Array.rst.txt | 43 -
.../python/generated/pyarrow.UInt32Value.rst.txt | 22 -
.../python/generated/pyarrow.UInt64Array.rst.txt | 43 -
.../python/generated/pyarrow.UInt64Value.rst.txt | 22 -
.../python/generated/pyarrow.UInt8Array.rst.txt | 43 -
.../python/generated/pyarrow.UInt8Value.rst.txt | 22 -
.../generated/pyarrow.allocate_buffer.rst.txt | 6 -
.../python/generated/pyarrow.binary.rst.txt | 6 -
.../python/generated/pyarrow.bool_.rst.txt | 6 -
.../python/generated/pyarrow.chunked_array.rst.txt | 6 -
.../python/generated/pyarrow.compress.rst.txt | 6 -
.../python/generated/pyarrow.concat_tables.rst.txt | 6 -
.../python/generated/pyarrow.cpu_count.rst.txt | 6 -
.../generated/pyarrow.create_memory_map.rst.txt | 6 -
.../generated/pyarrow.csv.ConvertOptions.rst.txt | 23 -
.../generated/pyarrow.csv.ParseOptions.rst.txt | 28 -
.../generated/pyarrow.csv.ReadOptions.rst.txt | 23 -
.../python/generated/pyarrow.csv.read_csv.rst.txt | 6 -
.../python/generated/pyarrow.date32.rst.txt | 6 -
.../python/generated/pyarrow.date64.rst.txt | 6 -
.../python/generated/pyarrow.decimal128.rst.txt | 6 -
.../python/generated/pyarrow.decompress.rst.txt | 6 -
.../generated/pyarrow.default_memory_pool.rst.txt | 6 -
.../python/generated/pyarrow.deserialize.rst.txt | 6 -
.../pyarrow.deserialize_components.rst.txt | 6 -
.../generated/pyarrow.deserialize_from.rst.txt | 6 -
.../python/generated/pyarrow.dictionary.rst.txt | 6 -
.../generated/pyarrow.feather.read_feather.rst.txt | 6 -
.../pyarrow.feather.write_feather.rst.txt | 6 -
.../python/generated/pyarrow.float16.rst.txt | 6 -
.../python/generated/pyarrow.float32.rst.txt | 6 -
.../python/generated/pyarrow.float64.rst.txt | 6 -
.../generated/pyarrow.foreign_buffer.rst.txt | 6 -
.../generated/pyarrow.from_numpy_dtype.rst.txt | 6 -
.../python/generated/pyarrow.get_include.rst.txt | 6 -
.../python/generated/pyarrow.get_libraries.rst.txt | 6 -
.../generated/pyarrow.get_library_dirs.rst.txt | 6 -
.../pyarrow.get_record_batch_size.rst.txt | 6 -
.../generated/pyarrow.get_tensor_size.rst.txt | 6 -
.../python/generated/pyarrow.hdfs.connect.rst.txt | 6 -
.../python/generated/pyarrow.input_stream.rst.txt | 6 -
.../python/generated/pyarrow.int16.rst.txt | 6 -
.../python/generated/pyarrow.int32.rst.txt | 6 -
.../python/generated/pyarrow.int64.rst.txt | 6 -
.../_sources/python/generated/pyarrow.int8.rst.txt | 6 -
.../python/generated/pyarrow.ipc.open_file.rst.txt | 6 -
.../generated/pyarrow.ipc.open_stream.rst.txt | 6 -
.../python/generated/pyarrow.list_.rst.txt | 6 -
.../pyarrow.log_memory_allocations.rst.txt | 6 -
.../python/generated/pyarrow.memory_map.rst.txt | 6 -
.../_sources/python/generated/pyarrow.null.rst.txt | 6 -
.../python/generated/pyarrow.open_file.rst.txt | 6 -
.../python/generated/pyarrow.open_stream.rst.txt | 6 -
.../python/generated/pyarrow.output_stream.rst.txt | 6 -
.../pyarrow.parquet.ParquetDataset.rst.txt | 25 -
.../generated/pyarrow.parquet.ParquetFile.rst.txt | 33 -
.../pyarrow.parquet.ParquetWriter.rst.txt | 24 -
.../pyarrow.parquet.read_metadata.rst.txt | 6 -
.../generated/pyarrow.parquet.read_pandas.rst.txt | 6 -
.../generated/pyarrow.parquet.read_schema.rst.txt | 6 -
.../generated/pyarrow.parquet.read_table.rst.txt | 6 -
.../pyarrow.parquet.write_metadata.rst.txt | 6 -
.../generated/pyarrow.parquet.write_table.rst.txt | 6 -
.../pyarrow.parquet.write_to_dataset.rst.txt | 6 -
.../generated/pyarrow.plasma.ObjectID.rst.txt | 23 -
.../generated/pyarrow.plasma.PlasmaBuffer.rst.txt | 33 -
.../generated/pyarrow.plasma.PlasmaClient.rst.txt | 46 -
.../python/generated/pyarrow.py_buffer.rst.txt | 6 -
.../python/generated/pyarrow.read_message.rst.txt | 6 -
.../generated/pyarrow.read_record_batch.rst.txt | 6 -
.../generated/pyarrow.read_serialized.rst.txt | 6 -
.../python/generated/pyarrow.read_tensor.rst.txt | 6 -
.../python/generated/pyarrow.serialize.rst.txt | 6 -
.../python/generated/pyarrow.serialize_to.rst.txt | 6 -
.../python/generated/pyarrow.set_cpu_count.rst.txt | 6 -
.../generated/pyarrow.set_memory_pool.rst.txt | 6 -
.../python/generated/pyarrow.string.rst.txt | 6 -
.../python/generated/pyarrow.struct.rst.txt | 6 -
.../python/generated/pyarrow.time32.rst.txt | 6 -
.../python/generated/pyarrow.time64.rst.txt | 6 -
.../python/generated/pyarrow.timestamp.rst.txt | 6 -
.../pyarrow.total_allocated_bytes.rst.txt | 6 -
.../generated/pyarrow.types.is_binary.rst.txt | 6 -
.../generated/pyarrow.types.is_boolean.rst.txt | 6 -
.../python/generated/pyarrow.types.is_date.rst.txt | 6 -
.../generated/pyarrow.types.is_date32.rst.txt | 6 -
.../generated/pyarrow.types.is_date64.rst.txt | 6 -
.../generated/pyarrow.types.is_decimal.rst.txt | 6 -
.../generated/pyarrow.types.is_dictionary.rst.txt | 6 -
.../pyarrow.types.is_fixed_size_binary.rst.txt | 6 -
.../generated/pyarrow.types.is_float16.rst.txt | 6 -
.../generated/pyarrow.types.is_float32.rst.txt | 6 -
.../generated/pyarrow.types.is_float64.rst.txt | 6 -
.../generated/pyarrow.types.is_floating.rst.txt | 6 -
.../generated/pyarrow.types.is_int16.rst.txt | 6 -
.../generated/pyarrow.types.is_int32.rst.txt | 6 -
.../generated/pyarrow.types.is_int64.rst.txt | 6 -
.../python/generated/pyarrow.types.is_int8.rst.txt | 6 -
.../generated/pyarrow.types.is_integer.rst.txt | 6 -
.../python/generated/pyarrow.types.is_list.rst.txt | 6 -
.../python/generated/pyarrow.types.is_map.rst.txt | 6 -
.../generated/pyarrow.types.is_nested.rst.txt | 6 -
.../python/generated/pyarrow.types.is_null.rst.txt | 6 -
.../pyarrow.types.is_signed_integer.rst.txt | 6 -
.../generated/pyarrow.types.is_string.rst.txt | 6 -
.../generated/pyarrow.types.is_struct.rst.txt | 6 -
.../generated/pyarrow.types.is_temporal.rst.txt | 6 -
.../python/generated/pyarrow.types.is_time.rst.txt | 6 -
.../generated/pyarrow.types.is_time32.rst.txt | 6 -
.../generated/pyarrow.types.is_time64.rst.txt | 6 -
.../generated/pyarrow.types.is_timestamp.rst.txt | 6 -
.../generated/pyarrow.types.is_uint16.rst.txt | 6 -
.../generated/pyarrow.types.is_uint32.rst.txt | 6 -
.../generated/pyarrow.types.is_uint64.rst.txt | 6 -
.../generated/pyarrow.types.is_uint8.rst.txt | 6 -
.../generated/pyarrow.types.is_unicode.rst.txt | 6 -
.../generated/pyarrow.types.is_union.rst.txt | 6 -
.../pyarrow.types.is_unsigned_integer.rst.txt | 6 -
.../python/generated/pyarrow.uint16.rst.txt | 6 -
.../python/generated/pyarrow.uint32.rst.txt | 6 -
.../python/generated/pyarrow.uint64.rst.txt | 6 -
.../python/generated/pyarrow.uint8.rst.txt | 6 -
.../_sources/python/generated/pyarrow.utf8.rst.txt | 6 -
.../python/generated/pyarrow.write_tensor.rst.txt | 6 -
.../_sources/python/getting_involved.rst.txt | 35 -
docs/latest/_sources/python/index.rst.txt | 49 -
docs/latest/_sources/python/install.rst.txt | 51 -
docs/latest/_sources/python/ipc.rst.txt | 383 -
docs/latest/_sources/python/memory.rst.txt | 284 -
docs/latest/_sources/python/numpy.rst.txt | 75 -
docs/latest/_sources/python/pandas.rst.txt | 124 -
docs/latest/_sources/python/parquet.rst.txt | 402 -
docs/latest/_sources/python/plasma.rst.txt | 467 -
docs/latest/_static/ajax-loader.gif | Bin 673 -> 0 bytes
docs/latest/_static/basic.css | 676 --
docs/latest/_static/comment-bright.png | Bin 756 -> 0 bytes
docs/latest/_static/comment-close.png | Bin 829 -> 0 bytes
docs/latest/_static/comment.png | Bin 641 -> 0 bytes
docs/latest/_static/css/badge_only.css | 1 -
docs/latest/_static/css/theme.css | 6 -
docs/latest/_static/doctools.js | 315 -
docs/latest/_static/documentation_options.js | 296 -
docs/latest/_static/down-pressed.png | Bin 222 -> 0 bytes
docs/latest/_static/down.png | Bin 202 -> 0 bytes
docs/latest/_static/file.png | Bin 286 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bold.eot | Bin 256056 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bold.ttf | Bin 600856 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bold.woff | Bin 309728 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bold.woff2 | Bin 184912 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bolditalic.eot | Bin 266158 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-bolditalic.ttf | Bin 622572 -> 0 bytes
.../latest/_static/fonts/Lato/lato-bolditalic.woff | Bin 323344 -> 0 bytes
.../_static/fonts/Lato/lato-bolditalic.woff2 | Bin 193308 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-italic.eot | Bin 268604 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-italic.ttf | Bin 639388 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-italic.woff | Bin 328412 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-italic.woff2 | Bin 195704 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-regular.eot | Bin 253461 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-regular.ttf | Bin 607720 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-regular.woff | Bin 309192 -> 0 bytes
docs/latest/_static/fonts/Lato/lato-regular.woff2 | Bin 182708 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-bold.eot | Bin 79520 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-bold.ttf | Bin 170616 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-bold.woff | Bin 87624 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-bold.woff2 | Bin 67312 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-regular.eot | Bin 78331 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-regular.ttf | Bin 169064 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-regular.woff | Bin 86288 -> 0 bytes
.../fonts/RobotoSlab/roboto-slab-v7-regular.woff2 | Bin 66444 -> 0 bytes
docs/latest/_static/fonts/fontawesome-webfont.eot | Bin 165742 -> 0 bytes
docs/latest/_static/fonts/fontawesome-webfont.svg | 2671 -----
docs/latest/_static/fonts/fontawesome-webfont.ttf | Bin 165548 -> 0 bytes
docs/latest/_static/fonts/fontawesome-webfont.woff | Bin 98024 -> 0 bytes
.../latest/_static/fonts/fontawesome-webfont.woff2 | Bin 77160 -> 0 bytes
docs/latest/_static/jquery-3.2.1.js | 10253 -------------------
docs/latest/_static/jquery.js | 4 -
docs/latest/_static/js/modernizr.min.js | 4 -
docs/latest/_static/js/theme.js | 3 -
docs/latest/_static/minus.png | Bin 90 -> 0 bytes
docs/latest/_static/plus.png | Bin 90 -> 0 bytes
docs/latest/_static/pygments.css | 69 -
docs/latest/_static/searchtools.js | 482 -
docs/latest/_static/stub | 18 -
docs/latest/_static/underscore-1.3.1.js | 999 --
docs/latest/_static/underscore.js | 31 -
docs/latest/_static/up-pressed.png | Bin 214 -> 0 bytes
docs/latest/_static/up.png | Bin 203 -> 0 bytes
docs/latest/_static/websupport.js | 808 --
docs/latest/cpp/api.html | 297 -
docs/latest/cpp/api/array.html | 801 --
docs/latest/cpp/api/builder.html | 1067 --
docs/latest/cpp/api/datatype.html | 1599 ---
docs/latest/cpp/api/memory.html | 1075 --
docs/latest/cpp/api/support.html | 463 -
docs/latest/cpp/api/table.html | 919 --
docs/latest/cpp/arrays.html | 430 -
docs/latest/cpp/conventions.html | 317 -
docs/latest/cpp/datatypes.html | 292 -
docs/latest/cpp/examples.html | 440 -
docs/latest/cpp/getting_started.html | 308 -
docs/latest/cpp/index.html | 264 -
docs/latest/cpp/memory.html | 348 -
docs/latest/cpp/overview.html | 316 -
docs/latest/cpp/tables.html | 313 -
docs/latest/format/Guidelines.html | 261 -
docs/latest/format/IPC.html | 450 -
docs/latest/format/Layout.html | 876 --
docs/latest/format/Metadata.html | 606 --
docs/latest/format/README.html | 266 -
docs/latest/genindex.html | 3948 -------
docs/latest/index.html | 273 -
docs/latest/objects.inv | Bin 30022 -> 0 bytes
docs/latest/python/api.html | 1290 ---
docs/latest/python/benchmarks.html | 247 -
docs/latest/python/csv.html | 324 -
docs/latest/python/data.html | 982 --
docs/latest/python/development.html | 589 --
docs/latest/python/extending.html | 671 --
docs/latest/python/filesystems.html | 359 -
docs/latest/python/generated/pyarrow.Array.html | 644 --
.../python/generated/pyarrow.ArrayValue.html | 309 -
.../python/generated/pyarrow.BinaryArray.html | 647 --
.../python/generated/pyarrow.BinaryValue.html | 336 -
.../python/generated/pyarrow.BooleanArray.html | 647 --
.../python/generated/pyarrow.BooleanValue.html | 327 -
docs/latest/python/generated/pyarrow.Buffer.html | 396 -
.../generated/pyarrow.BufferOutputStream.html | 628 --
.../python/generated/pyarrow.BufferReader.html | 620 --
.../python/generated/pyarrow.ChunkedArray.html | 525 -
docs/latest/python/generated/pyarrow.Column.html | 558 -
.../generated/pyarrow.CompressedInputStream.html | 625 --
.../generated/pyarrow.CompressedOutputStream.html | 625 --
docs/latest/python/generated/pyarrow.DataType.html | 353 -
.../python/generated/pyarrow.Date32Array.html | 647 --
.../python/generated/pyarrow.Date32Value.html | 327 -
.../python/generated/pyarrow.Date64Array.html | 647 --
.../python/generated/pyarrow.Date64Value.html | 327 -
.../python/generated/pyarrow.Decimal128Array.html | 647 --
.../python/generated/pyarrow.DecimalValue.html | 324 -
.../python/generated/pyarrow.DictionaryArray.html | 693 --
.../python/generated/pyarrow.DoubleValue.html | 327 -
docs/latest/python/generated/pyarrow.Field.html | 417 -
.../generated/pyarrow.FixedSizeBinaryArray.html | 647 --
.../generated/pyarrow.FixedSizeBinaryValue.html | 327 -
.../generated/pyarrow.FixedSizeBufferWriter.html | 635 --
.../python/generated/pyarrow.FloatValue.html | 327 -
.../generated/pyarrow.FloatingPointArray.html | 647 --
.../generated/pyarrow.HadoopFileSystem.cat.html | 273 -
.../generated/pyarrow.HadoopFileSystem.chmod.html | 277 -
.../generated/pyarrow.HadoopFileSystem.chown.html | 278 -
.../generated/pyarrow.HadoopFileSystem.delete.html | 277 -
.../generated/pyarrow.HadoopFileSystem.df.html | 273 -
.../pyarrow.HadoopFileSystem.disk_usage.html | 275 -
.../pyarrow.HadoopFileSystem.download.html | 264 -
.../generated/pyarrow.HadoopFileSystem.exists.html | 266 -
.../pyarrow.HadoopFileSystem.get_capacity.html | 273 -
.../pyarrow.HadoopFileSystem.get_space_used.html | 273 -
.../generated/pyarrow.HadoopFileSystem.info.html | 275 -
.../generated/pyarrow.HadoopFileSystem.ls.html | 280 -
.../generated/pyarrow.HadoopFileSystem.mkdir.html | 275 -
.../generated/pyarrow.HadoopFileSystem.open.html | 275 -
.../generated/pyarrow.HadoopFileSystem.rename.html | 277 -
.../generated/pyarrow.HadoopFileSystem.rm.html | 265 -
.../generated/pyarrow.HadoopFileSystem.upload.html | 265 -
docs/latest/python/generated/pyarrow.HdfsFile.html | 598 --
.../python/generated/pyarrow.Int16Array.html | 647 --
.../python/generated/pyarrow.Int16Value.html | 327 -
.../python/generated/pyarrow.Int32Array.html | 647 --
.../python/generated/pyarrow.Int32Value.html | 327 -
.../python/generated/pyarrow.Int64Array.html | 647 --
.../python/generated/pyarrow.Int64Value.html | 327 -
.../latest/python/generated/pyarrow.Int8Array.html | 647 --
.../latest/python/generated/pyarrow.Int8Value.html | 327 -
.../python/generated/pyarrow.IntegerArray.html | 647 --
.../latest/python/generated/pyarrow.ListArray.html | 685 --
.../latest/python/generated/pyarrow.ListValue.html | 344 -
.../python/generated/pyarrow.LocalFileSystem.html | 535 -
.../python/generated/pyarrow.MemoryMappedFile.html | 636 --
.../python/generated/pyarrow.MemoryPool.html | 323 -
docs/latest/python/generated/pyarrow.Message.html | 395 -
.../python/generated/pyarrow.MessageReader.html | 338 -
docs/latest/python/generated/pyarrow.NA.html | 303 -
.../python/generated/pyarrow.NativeFile.html | 611 --
.../latest/python/generated/pyarrow.NullArray.html | 647 --
.../python/generated/pyarrow.NumericArray.html | 647 --
docs/latest/python/generated/pyarrow.OSFile.html | 611 --
.../python/generated/pyarrow.PythonFile.html | 598 --
.../python/generated/pyarrow.RecordBatch.html | 580 --
.../generated/pyarrow.RecordBatchFileReader.html | 406 -
.../generated/pyarrow.RecordBatchFileWriter.html | 394 -
.../generated/pyarrow.RecordBatchStreamReader.html | 394 -
.../generated/pyarrow.RecordBatchStreamWriter.html | 394 -
.../python/generated/pyarrow.ResizableBuffer.html | 418 -
docs/latest/python/generated/pyarrow.Scalar.html | 309 -
docs/latest/python/generated/pyarrow.Schema.html | 597 --
.../generated/pyarrow.SerializationContext.html | 420 -
.../generated/pyarrow.SerializedPyObject.html | 404 -
.../python/generated/pyarrow.StringArray.html | 648 --
.../python/generated/pyarrow.StringValue.html | 327 -
docs/latest/python/generated/pyarrow.Table.html | 738 --
docs/latest/python/generated/pyarrow.Tensor.html | 388 -
.../python/generated/pyarrow.Time32Array.html | 647 --
.../python/generated/pyarrow.Time64Array.html | 647 --
.../python/generated/pyarrow.TimestampArray.html | 647 --
.../python/generated/pyarrow.TimestampValue.html | 345 -
.../python/generated/pyarrow.UInt16Array.html | 647 --
.../python/generated/pyarrow.UInt16Value.html | 327 -
.../python/generated/pyarrow.UInt32Array.html | 647 --
.../python/generated/pyarrow.UInt32Value.html | 327 -
.../python/generated/pyarrow.UInt64Array.html | 647 --
.../python/generated/pyarrow.UInt64Value.html | 327 -
.../python/generated/pyarrow.UInt8Array.html | 647 --
.../python/generated/pyarrow.UInt8Value.html | 327 -
.../python/generated/pyarrow.allocate_buffer.html | 304 -
docs/latest/python/generated/pyarrow.binary.html | 317 -
docs/latest/python/generated/pyarrow.bool_.html | 307 -
.../python/generated/pyarrow.chunked_array.html | 299 -
docs/latest/python/generated/pyarrow.compress.html | 307 -
.../python/generated/pyarrow.concat_tables.html | 300 -
.../latest/python/generated/pyarrow.cpu_count.html | 288 -
.../generated/pyarrow.create_memory_map.html | 310 -
.../generated/pyarrow.csv.ConvertOptions.html | 341 -
.../python/generated/pyarrow.csv.ParseOptions.html | 395 -
.../python/generated/pyarrow.csv.ReadOptions.html | 335 -
.../python/generated/pyarrow.csv.read_csv.html | 307 -
docs/latest/python/generated/pyarrow.date32.html | 307 -
docs/latest/python/generated/pyarrow.date64.html | 307 -
.../python/generated/pyarrow.decimal128.html | 322 -
.../python/generated/pyarrow.decompress.html | 308 -
.../generated/pyarrow.default_memory_pool.html | 285 -
.../python/generated/pyarrow.deserialize.html | 319 -
.../generated/pyarrow.deserialize_components.html | 318 -
.../python/generated/pyarrow.deserialize_from.html | 320 -
.../python/generated/pyarrow.dictionary.html | 320 -
.../generated/pyarrow.feather.read_feather.html | 300 -
.../generated/pyarrow.feather.write_feather.html | 295 -
docs/latest/python/generated/pyarrow.float16.html | 307 -
docs/latest/python/generated/pyarrow.float32.html | 307 -
docs/latest/python/generated/pyarrow.float64.html | 307 -
.../python/generated/pyarrow.foreign_buffer.html | 292 -
.../python/generated/pyarrow.from_numpy_dtype.html | 304 -
.../python/generated/pyarrow.get_include.html | 285 -
.../python/generated/pyarrow.get_libraries.html | 285 -
.../python/generated/pyarrow.get_library_dirs.html | 285 -
.../generated/pyarrow.get_record_batch_size.html | 303 -
.../python/generated/pyarrow.get_tensor_size.html | 303 -
.../python/generated/pyarrow.hdfs.connect.html | 298 -
.../python/generated/pyarrow.input_stream.html | 310 -
docs/latest/python/generated/pyarrow.int16.html | 307 -
docs/latest/python/generated/pyarrow.int32.html | 307 -
docs/latest/python/generated/pyarrow.int64.html | 307 -
docs/latest/python/generated/pyarrow.int8.html | 307 -
.../python/generated/pyarrow.ipc.open_file.html | 319 -
.../python/generated/pyarrow.ipc.open_stream.html | 319 -
docs/latest/python/generated/pyarrow.list_.html | 317 -
.../generated/pyarrow.log_memory_allocations.html | 294 -
.../python/generated/pyarrow.memory_map.html | 309 -
docs/latest/python/generated/pyarrow.null.html | 307 -
.../latest/python/generated/pyarrow.open_file.html | 228 -
.../python/generated/pyarrow.open_stream.html | 228 -
.../python/generated/pyarrow.output_stream.html | 310 -
.../generated/pyarrow.parquet.ParquetDataset.html | 396 -
.../generated/pyarrow.parquet.ParquetFile.html | 442 -
.../generated/pyarrow.parquet.ParquetWriter.html | 351 -
.../generated/pyarrow.parquet.read_metadata.html | 306 -
.../generated/pyarrow.parquet.read_pandas.html | 316 -
.../generated/pyarrow.parquet.read_schema.html | 306 -
.../generated/pyarrow.parquet.read_table.html | 316 -
.../generated/pyarrow.parquet.write_metadata.html | 309 -
.../generated/pyarrow.parquet.write_table.html | 316 -
.../pyarrow.parquet.write_to_dataset.html | 335 -
.../python/generated/pyarrow.plasma.ObjectID.html | 334 -
.../generated/pyarrow.plasma.PlasmaBuffer.html | 425 -
.../generated/pyarrow.plasma.PlasmaClient.html | 733 --
.../latest/python/generated/pyarrow.py_buffer.html | 288 -
.../python/generated/pyarrow.read_message.html | 313 -
.../generated/pyarrow.read_record_batch.html | 318 -
.../python/generated/pyarrow.read_serialized.html | 319 -
.../python/generated/pyarrow.read_tensor.html | 316 -
.../latest/python/generated/pyarrow.serialize.html | 319 -
.../python/generated/pyarrow.serialize_to.html | 317 -
.../python/generated/pyarrow.set_cpu_count.html | 283 -
.../python/generated/pyarrow.set_memory_pool.html | 285 -
docs/latest/python/generated/pyarrow.string.html | 307 -
docs/latest/python/generated/pyarrow.struct.html | 338 -
docs/latest/python/generated/pyarrow.time32.html | 320 -
docs/latest/python/generated/pyarrow.time64.html | 320 -
.../latest/python/generated/pyarrow.timestamp.html | 333 -
.../generated/pyarrow.total_allocated_bytes.html | 287 -
.../python/generated/pyarrow.types.is_binary.html | 317 -
.../python/generated/pyarrow.types.is_boolean.html | 317 -
.../python/generated/pyarrow.types.is_date.html | 317 -
.../python/generated/pyarrow.types.is_date32.html | 317 -
.../python/generated/pyarrow.types.is_date64.html | 317 -
.../python/generated/pyarrow.types.is_decimal.html | 317 -
.../generated/pyarrow.types.is_dictionary.html | 317 -
.../pyarrow.types.is_fixed_size_binary.html | 317 -
.../python/generated/pyarrow.types.is_float16.html | 317 -
.../python/generated/pyarrow.types.is_float32.html | 317 -
.../python/generated/pyarrow.types.is_float64.html | 317 -
.../generated/pyarrow.types.is_floating.html | 317 -
.../python/generated/pyarrow.types.is_int16.html | 317 -
.../python/generated/pyarrow.types.is_int32.html | 317 -
.../python/generated/pyarrow.types.is_int64.html | 317 -
.../python/generated/pyarrow.types.is_int8.html | 317 -
.../python/generated/pyarrow.types.is_integer.html | 317 -
.../python/generated/pyarrow.types.is_list.html | 317 -
.../python/generated/pyarrow.types.is_map.html | 317 -
.../python/generated/pyarrow.types.is_nested.html | 317 -
.../python/generated/pyarrow.types.is_null.html | 317 -
.../generated/pyarrow.types.is_signed_integer.html | 317 -
.../python/generated/pyarrow.types.is_string.html | 317 -
.../python/generated/pyarrow.types.is_struct.html | 317 -
.../generated/pyarrow.types.is_temporal.html | 318 -
.../python/generated/pyarrow.types.is_time.html | 317 -
.../python/generated/pyarrow.types.is_time32.html | 317 -
.../python/generated/pyarrow.types.is_time64.html | 317 -
.../generated/pyarrow.types.is_timestamp.html | 317 -
.../python/generated/pyarrow.types.is_uint16.html | 317 -
.../python/generated/pyarrow.types.is_uint32.html | 317 -
.../python/generated/pyarrow.types.is_uint64.html | 317 -
.../python/generated/pyarrow.types.is_uint8.html | 317 -
.../python/generated/pyarrow.types.is_unicode.html | 317 -
.../python/generated/pyarrow.types.is_union.html | 317 -
.../pyarrow.types.is_unsigned_integer.html | 317 -
docs/latest/python/generated/pyarrow.uint16.html | 307 -
docs/latest/python/generated/pyarrow.uint32.html | 307 -
docs/latest/python/generated/pyarrow.uint64.html | 307 -
docs/latest/python/generated/pyarrow.uint8.html | 307 -
docs/latest/python/generated/pyarrow.utf8.html | 307 -
.../python/generated/pyarrow.write_tensor.html | 318 -
docs/latest/python/getting_involved.html | 264 -
docs/latest/python/index.html | 364 -
docs/latest/python/install.html | 281 -
docs/latest/python/ipc.html | 655 --
docs/latest/python/memory.html | 532 -
docs/latest/python/numpy.html | 302 -
docs/latest/python/pandas.html | 394 -
docs/latest/python/parquet.html | 672 --
docs/latest/python/plasma.html | 669 --
docs/latest/search.html | 237 -
docs/latest/searchindex.js | 1 -
819 files changed, 136948 deletions(-)
diff --git a/docs/latest/.buildinfo b/docs/latest/.buildinfo
deleted file mode 100644
index dc1b213..0000000
--- a/docs/latest/.buildinfo
+++ /dev/null
@@ -1,4 +0,0 @@
-# Sphinx build info version 1
-# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: a5756bf3bbfdff060c47e74ac722eb4a
-tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/docs/latest/.doctrees/cpp/api.doctree b/docs/latest/.doctrees/cpp/api.doctree
deleted file mode 100644
index 20fac52..0000000
Binary files a/docs/latest/.doctrees/cpp/api.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/array.doctree b/docs/latest/.doctrees/cpp/api/array.doctree
deleted file mode 100644
index 47e28a2..0000000
Binary files a/docs/latest/.doctrees/cpp/api/array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/builder.doctree b/docs/latest/.doctrees/cpp/api/builder.doctree
deleted file mode 100644
index ae634b0..0000000
Binary files a/docs/latest/.doctrees/cpp/api/builder.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/datatype.doctree b/docs/latest/.doctrees/cpp/api/datatype.doctree
deleted file mode 100644
index 0043fc6..0000000
Binary files a/docs/latest/.doctrees/cpp/api/datatype.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/memory.doctree b/docs/latest/.doctrees/cpp/api/memory.doctree
deleted file mode 100644
index ba38149..0000000
Binary files a/docs/latest/.doctrees/cpp/api/memory.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/support.doctree b/docs/latest/.doctrees/cpp/api/support.doctree
deleted file mode 100644
index de9959c..0000000
Binary files a/docs/latest/.doctrees/cpp/api/support.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/api/table.doctree b/docs/latest/.doctrees/cpp/api/table.doctree
deleted file mode 100644
index 1c51f96..0000000
Binary files a/docs/latest/.doctrees/cpp/api/table.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/arrays.doctree b/docs/latest/.doctrees/cpp/arrays.doctree
deleted file mode 100644
index f870d5f..0000000
Binary files a/docs/latest/.doctrees/cpp/arrays.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/conventions.doctree b/docs/latest/.doctrees/cpp/conventions.doctree
deleted file mode 100644
index 07e4f1c..0000000
Binary files a/docs/latest/.doctrees/cpp/conventions.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/datatypes.doctree b/docs/latest/.doctrees/cpp/datatypes.doctree
deleted file mode 100644
index e39e00d..0000000
Binary files a/docs/latest/.doctrees/cpp/datatypes.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/examples.doctree b/docs/latest/.doctrees/cpp/examples.doctree
deleted file mode 100644
index 8b45945..0000000
Binary files a/docs/latest/.doctrees/cpp/examples.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/getting_started.doctree b/docs/latest/.doctrees/cpp/getting_started.doctree
deleted file mode 100644
index f636154..0000000
Binary files a/docs/latest/.doctrees/cpp/getting_started.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/index.doctree b/docs/latest/.doctrees/cpp/index.doctree
deleted file mode 100644
index 2b752c9..0000000
Binary files a/docs/latest/.doctrees/cpp/index.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/memory.doctree b/docs/latest/.doctrees/cpp/memory.doctree
deleted file mode 100644
index 2916308..0000000
Binary files a/docs/latest/.doctrees/cpp/memory.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/overview.doctree b/docs/latest/.doctrees/cpp/overview.doctree
deleted file mode 100644
index bb9cff9..0000000
Binary files a/docs/latest/.doctrees/cpp/overview.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/cpp/tables.doctree b/docs/latest/.doctrees/cpp/tables.doctree
deleted file mode 100644
index c839903..0000000
Binary files a/docs/latest/.doctrees/cpp/tables.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/environment.pickle b/docs/latest/.doctrees/environment.pickle
deleted file mode 100644
index 1ef0540..0000000
Binary files a/docs/latest/.doctrees/environment.pickle and /dev/null differ
diff --git a/docs/latest/.doctrees/format/Guidelines.doctree b/docs/latest/.doctrees/format/Guidelines.doctree
deleted file mode 100644
index 2d80ef2..0000000
Binary files a/docs/latest/.doctrees/format/Guidelines.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/format/IPC.doctree b/docs/latest/.doctrees/format/IPC.doctree
deleted file mode 100644
index 61dbcc8..0000000
Binary files a/docs/latest/.doctrees/format/IPC.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/format/Layout.doctree b/docs/latest/.doctrees/format/Layout.doctree
deleted file mode 100644
index a048ab7..0000000
Binary files a/docs/latest/.doctrees/format/Layout.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/format/Metadata.doctree b/docs/latest/.doctrees/format/Metadata.doctree
deleted file mode 100644
index c0f3fc3..0000000
Binary files a/docs/latest/.doctrees/format/Metadata.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/format/README.doctree b/docs/latest/.doctrees/format/README.doctree
deleted file mode 100644
index 4f47af2..0000000
Binary files a/docs/latest/.doctrees/format/README.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/index.doctree b/docs/latest/.doctrees/index.doctree
deleted file mode 100644
index 162322f..0000000
Binary files a/docs/latest/.doctrees/index.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/api.doctree b/docs/latest/.doctrees/python/api.doctree
deleted file mode 100644
index a5e4bb2..0000000
Binary files a/docs/latest/.doctrees/python/api.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/benchmarks.doctree b/docs/latest/.doctrees/python/benchmarks.doctree
deleted file mode 100644
index f03ac4b..0000000
Binary files a/docs/latest/.doctrees/python/benchmarks.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/csv.doctree b/docs/latest/.doctrees/python/csv.doctree
deleted file mode 100644
index f3237a3..0000000
Binary files a/docs/latest/.doctrees/python/csv.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/data.doctree b/docs/latest/.doctrees/python/data.doctree
deleted file mode 100644
index ab295db..0000000
Binary files a/docs/latest/.doctrees/python/data.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/development.doctree b/docs/latest/.doctrees/python/development.doctree
deleted file mode 100644
index 65d9e56..0000000
Binary files a/docs/latest/.doctrees/python/development.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/extending.doctree b/docs/latest/.doctrees/python/extending.doctree
deleted file mode 100644
index ac60ccc..0000000
Binary files a/docs/latest/.doctrees/python/extending.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/filesystems.doctree b/docs/latest/.doctrees/python/filesystems.doctree
deleted file mode 100644
index 8454a24..0000000
Binary files a/docs/latest/.doctrees/python/filesystems.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Array.doctree
deleted file mode 100644
index 043a92d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ArrayValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ArrayValue.doctree
deleted file mode 100644
index c19456c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ArrayValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BinaryArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BinaryArray.doctree
deleted file mode 100644
index 3f297d8..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BinaryArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BinaryValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BinaryValue.doctree
deleted file mode 100644
index 4f4ae11..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BinaryValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BooleanArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BooleanArray.doctree
deleted file mode 100644
index 827e5dd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BooleanArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BooleanValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BooleanValue.doctree
deleted file mode 100644
index 6f3f1ea..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BooleanValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Buffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Buffer.doctree
deleted file mode 100644
index f1d6b83..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Buffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BufferOutputStream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BufferOutputStream.doctree
deleted file mode 100644
index 9baf8ae..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BufferOutputStream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.BufferReader.doctree b/docs/latest/.doctrees/python/generated/pyarrow.BufferReader.doctree
deleted file mode 100644
index e755eea..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.BufferReader.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ChunkedArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ChunkedArray.doctree
deleted file mode 100644
index f2f0c39..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ChunkedArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Column.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Column.doctree
deleted file mode 100644
index 07f7a83..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Column.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.CompressedInputStream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.CompressedInputStream.doctree
deleted file mode 100644
index 6b2a594..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.CompressedInputStream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.CompressedOutputStream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.CompressedOutputStream.doctree
deleted file mode 100644
index 4f2fbb6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.CompressedOutputStream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.DataType.doctree b/docs/latest/.doctrees/python/generated/pyarrow.DataType.doctree
deleted file mode 100644
index 1a8190d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.DataType.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Date32Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Date32Array.doctree
deleted file mode 100644
index 3b6114a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Date32Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Date32Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Date32Value.doctree
deleted file mode 100644
index a1bcd14..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Date32Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Date64Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Date64Array.doctree
deleted file mode 100644
index 49390ba..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Date64Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Date64Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Date64Value.doctree
deleted file mode 100644
index 300ef06..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Date64Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Decimal128Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Decimal128Array.doctree
deleted file mode 100644
index 9ba644f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Decimal128Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.DecimalValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.DecimalValue.doctree
deleted file mode 100644
index 9a9fd2a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.DecimalValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.DictionaryArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.DictionaryArray.doctree
deleted file mode 100644
index 1bcfeaa..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.DictionaryArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.DoubleValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.DoubleValue.doctree
deleted file mode 100644
index e11acac..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.DoubleValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Field.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Field.doctree
deleted file mode 100644
index 797f721..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Field.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryArray.doctree
deleted file mode 100644
index 250075a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryValue.doctree
deleted file mode 100644
index 512275d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBinaryValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBufferWriter.doctree b/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBufferWriter.doctree
deleted file mode 100644
index 907f26b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.FixedSizeBufferWriter.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.FloatValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.FloatValue.doctree
deleted file mode 100644
index f28baf4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.FloatValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.FloatingPointArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.FloatingPointArray.doctree
deleted file mode 100644
index 497ba0e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.FloatingPointArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.cat.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.cat.doctree
deleted file mode 100644
index f6aff17..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.cat.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chmod.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chmod.doctree
deleted file mode 100644
index f972772..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chmod.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chown.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chown.doctree
deleted file mode 100644
index 7d93b77..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.chown.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.delete.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.delete.doctree
deleted file mode 100644
index 07c2b09..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.delete.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.df.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.df.doctree
deleted file mode 100644
index 67856a5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.df.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.disk_usage.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.disk_usage.doctree
deleted file mode 100644
index 31a72f6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.disk_usage.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.download.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.download.doctree
deleted file mode 100644
index a1b0d94..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.download.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.exists.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.exists.doctree
deleted file mode 100644
index 3b0ddb7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.exists.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_capacity.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_capacity.doctree
deleted file mode 100644
index fb7e2e6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_capacity.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_space_used.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_space_used.doctree
deleted file mode 100644
index d00aea9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.get_space_used.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.info.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.info.doctree
deleted file mode 100644
index a00ab67..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.info.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.ls.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.ls.doctree
deleted file mode 100644
index 2b0664b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.ls.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.mkdir.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.mkdir.doctree
deleted file mode 100644
index 5e8811c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.mkdir.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.open.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.open.doctree
deleted file mode 100644
index 5f62ab7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.open.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rename.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rename.doctree
deleted file mode 100644
index 59d93d2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rename.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rm.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rm.doctree
deleted file mode 100644
index d5edf9e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.rm.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.upload.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.upload.doctree
deleted file mode 100644
index f3e18c2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HadoopFileSystem.upload.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.HdfsFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.HdfsFile.doctree
deleted file mode 100644
index df02c8b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.HdfsFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int16Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int16Array.doctree
deleted file mode 100644
index d785ca4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int16Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int16Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int16Value.doctree
deleted file mode 100644
index 9567652..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int16Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int32Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int32Array.doctree
deleted file mode 100644
index f05f105..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int32Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int32Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int32Value.doctree
deleted file mode 100644
index 314e161..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int32Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int64Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int64Array.doctree
deleted file mode 100644
index 8cb4b3a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int64Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int64Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int64Value.doctree
deleted file mode 100644
index da3c674..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int64Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int8Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int8Array.doctree
deleted file mode 100644
index 78ace1d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int8Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Int8Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Int8Value.doctree
deleted file mode 100644
index aedb203..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Int8Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.IntegerArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.IntegerArray.doctree
deleted file mode 100644
index c6a6017..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.IntegerArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ListArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ListArray.doctree
deleted file mode 100644
index 126cdb9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ListArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ListValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ListValue.doctree
deleted file mode 100644
index 26bd7e3..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ListValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.LocalFileSystem.doctree b/docs/latest/.doctrees/python/generated/pyarrow.LocalFileSystem.doctree
deleted file mode 100644
index 5bdf90e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.LocalFileSystem.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.MemoryMappedFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.MemoryMappedFile.doctree
deleted file mode 100644
index 33f704c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.MemoryMappedFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.MemoryPool.doctree b/docs/latest/.doctrees/python/generated/pyarrow.MemoryPool.doctree
deleted file mode 100644
index cd7dab0..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.MemoryPool.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Message.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Message.doctree
deleted file mode 100644
index 6d74ba9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Message.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.MessageReader.doctree b/docs/latest/.doctrees/python/generated/pyarrow.MessageReader.doctree
deleted file mode 100644
index 3d906a4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.MessageReader.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.NA.doctree b/docs/latest/.doctrees/python/generated/pyarrow.NA.doctree
deleted file mode 100644
index 705a16f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.NA.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.NativeFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.NativeFile.doctree
deleted file mode 100644
index eef4fb1..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.NativeFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.NullArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.NullArray.doctree
deleted file mode 100644
index f130dad..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.NullArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.NumericArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.NumericArray.doctree
deleted file mode 100644
index f184dcc..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.NumericArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.OSFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.OSFile.doctree
deleted file mode 100644
index 1fe0137..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.OSFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.PythonFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.PythonFile.doctree
deleted file mode 100644
index 01fbf8d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.PythonFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatch.doctree b/docs/latest/.doctrees/python/generated/pyarrow.RecordBatch.doctree
deleted file mode 100644
index 430f896..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatch.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileReader.doctree b/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileReader.doctree
deleted file mode 100644
index c32be4e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileReader.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileWriter.doctree b/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileWriter.doctree
deleted file mode 100644
index 31b9412..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchFileWriter.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamReader.doctree b/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamReader.doctree
deleted file mode 100644
index 41d7ccc..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamReader.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamWriter.doctree b/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamWriter.doctree
deleted file mode 100644
index cba2a1b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.RecordBatchStreamWriter.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ResizableBuffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ResizableBuffer.doctree
deleted file mode 100644
index 43dd92d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ResizableBuffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Scalar.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Scalar.doctree
deleted file mode 100644
index 79f9a60..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Scalar.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Schema.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Schema.doctree
deleted file mode 100644
index e8cb1ef..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Schema.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.SerializationContext.doctree b/docs/latest/.doctrees/python/generated/pyarrow.SerializationContext.doctree
deleted file mode 100644
index c5b4158..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.SerializationContext.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.SerializedPyObject.doctree b/docs/latest/.doctrees/python/generated/pyarrow.SerializedPyObject.doctree
deleted file mode 100644
index 544d24c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.SerializedPyObject.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.StringArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.StringArray.doctree
deleted file mode 100644
index 60b1d48..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.StringArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.StringValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.StringValue.doctree
deleted file mode 100644
index b50b75c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.StringValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Table.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Table.doctree
deleted file mode 100644
index 4dc1a7f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Table.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Tensor.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Tensor.doctree
deleted file mode 100644
index 05ef7f9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Tensor.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Time32Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Time32Array.doctree
deleted file mode 100644
index f25292a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Time32Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.Time64Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.Time64Array.doctree
deleted file mode 100644
index 62d43d6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.Time64Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.TimestampArray.doctree b/docs/latest/.doctrees/python/generated/pyarrow.TimestampArray.doctree
deleted file mode 100644
index ded15d0..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.TimestampArray.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.TimestampValue.doctree b/docs/latest/.doctrees/python/generated/pyarrow.TimestampValue.doctree
deleted file mode 100644
index 1d06f77..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.TimestampValue.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt16Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt16Array.doctree
deleted file mode 100644
index 3335c5a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt16Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt16Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt16Value.doctree
deleted file mode 100644
index c7279d7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt16Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt32Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt32Array.doctree
deleted file mode 100644
index 6533125..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt32Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt32Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt32Value.doctree
deleted file mode 100644
index d1a4d90..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt32Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt64Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt64Array.doctree
deleted file mode 100644
index 7f8969c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt64Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt64Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt64Value.doctree
deleted file mode 100644
index 673f7e5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt64Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt8Array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt8Array.doctree
deleted file mode 100644
index f63079f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt8Array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.UInt8Value.doctree b/docs/latest/.doctrees/python/generated/pyarrow.UInt8Value.doctree
deleted file mode 100644
index 9ad10e7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.UInt8Value.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.allocate_buffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.allocate_buffer.doctree
deleted file mode 100644
index fe9effa..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.allocate_buffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.binary.doctree b/docs/latest/.doctrees/python/generated/pyarrow.binary.doctree
deleted file mode 100644
index 1e08f12..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.binary.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.bool_.doctree b/docs/latest/.doctrees/python/generated/pyarrow.bool_.doctree
deleted file mode 100644
index 88e0569..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.bool_.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.chunked_array.doctree b/docs/latest/.doctrees/python/generated/pyarrow.chunked_array.doctree
deleted file mode 100644
index 4895ee3..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.chunked_array.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.compress.doctree b/docs/latest/.doctrees/python/generated/pyarrow.compress.doctree
deleted file mode 100644
index 1919fb5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.compress.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.concat_tables.doctree b/docs/latest/.doctrees/python/generated/pyarrow.concat_tables.doctree
deleted file mode 100644
index 925124b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.concat_tables.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.cpu_count.doctree b/docs/latest/.doctrees/python/generated/pyarrow.cpu_count.doctree
deleted file mode 100644
index b2d8a7e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.cpu_count.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.create_memory_map.doctree b/docs/latest/.doctrees/python/generated/pyarrow.create_memory_map.doctree
deleted file mode 100644
index 91b2800..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.create_memory_map.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.csv.ConvertOptions.doctree b/docs/latest/.doctrees/python/generated/pyarrow.csv.ConvertOptions.doctree
deleted file mode 100644
index d89595a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.csv.ConvertOptions.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.csv.ParseOptions.doctree b/docs/latest/.doctrees/python/generated/pyarrow.csv.ParseOptions.doctree
deleted file mode 100644
index 1e49791..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.csv.ParseOptions.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.csv.ReadOptions.doctree b/docs/latest/.doctrees/python/generated/pyarrow.csv.ReadOptions.doctree
deleted file mode 100644
index e21d8e4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.csv.ReadOptions.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.csv.read_csv.doctree b/docs/latest/.doctrees/python/generated/pyarrow.csv.read_csv.doctree
deleted file mode 100644
index bffba88..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.csv.read_csv.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.date32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.date32.doctree
deleted file mode 100644
index 047a3a5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.date32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.date64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.date64.doctree
deleted file mode 100644
index 8ce4cea..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.date64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.decimal128.doctree b/docs/latest/.doctrees/python/generated/pyarrow.decimal128.doctree
deleted file mode 100644
index a965c4e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.decimal128.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.decompress.doctree b/docs/latest/.doctrees/python/generated/pyarrow.decompress.doctree
deleted file mode 100644
index b210b18..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.decompress.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.default_memory_pool.doctree b/docs/latest/.doctrees/python/generated/pyarrow.default_memory_pool.doctree
deleted file mode 100644
index 444022d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.default_memory_pool.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.deserialize.doctree b/docs/latest/.doctrees/python/generated/pyarrow.deserialize.doctree
deleted file mode 100644
index 86421ad..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.deserialize.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.deserialize_components.doctree b/docs/latest/.doctrees/python/generated/pyarrow.deserialize_components.doctree
deleted file mode 100644
index deb10fa..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.deserialize_components.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.deserialize_from.doctree b/docs/latest/.doctrees/python/generated/pyarrow.deserialize_from.doctree
deleted file mode 100644
index 3e0b1d6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.deserialize_from.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.dictionary.doctree b/docs/latest/.doctrees/python/generated/pyarrow.dictionary.doctree
deleted file mode 100644
index b72b4e2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.dictionary.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.feather.read_feather.doctree b/docs/latest/.doctrees/python/generated/pyarrow.feather.read_feather.doctree
deleted file mode 100644
index 78fcbe0..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.feather.read_feather.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.feather.write_feather.doctree b/docs/latest/.doctrees/python/generated/pyarrow.feather.write_feather.doctree
deleted file mode 100644
index 935e37e..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.feather.write_feather.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.float16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.float16.doctree
deleted file mode 100644
index b9260c4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.float16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.float32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.float32.doctree
deleted file mode 100644
index 96dea64..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.float32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.float64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.float64.doctree
deleted file mode 100644
index c97caef..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.float64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.foreign_buffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.foreign_buffer.doctree
deleted file mode 100644
index 9aef807..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.foreign_buffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.from_numpy_dtype.doctree b/docs/latest/.doctrees/python/generated/pyarrow.from_numpy_dtype.doctree
deleted file mode 100644
index caeb055..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.from_numpy_dtype.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.get_include.doctree b/docs/latest/.doctrees/python/generated/pyarrow.get_include.doctree
deleted file mode 100644
index bc488cd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.get_include.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.get_libraries.doctree b/docs/latest/.doctrees/python/generated/pyarrow.get_libraries.doctree
deleted file mode 100644
index 73cd95a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.get_libraries.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.get_library_dirs.doctree b/docs/latest/.doctrees/python/generated/pyarrow.get_library_dirs.doctree
deleted file mode 100644
index 6464fbc..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.get_library_dirs.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.get_record_batch_size.doctree b/docs/latest/.doctrees/python/generated/pyarrow.get_record_batch_size.doctree
deleted file mode 100644
index 2be3b34..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.get_record_batch_size.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.get_tensor_size.doctree b/docs/latest/.doctrees/python/generated/pyarrow.get_tensor_size.doctree
deleted file mode 100644
index 6cb3053..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.get_tensor_size.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.hdfs.connect.doctree b/docs/latest/.doctrees/python/generated/pyarrow.hdfs.connect.doctree
deleted file mode 100644
index fa425cf..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.hdfs.connect.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.input_stream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.input_stream.doctree
deleted file mode 100644
index 7a1db68..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.input_stream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.int16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.int16.doctree
deleted file mode 100644
index 18dd4fe..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.int16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.int32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.int32.doctree
deleted file mode 100644
index a540cef..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.int32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.int64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.int64.doctree
deleted file mode 100644
index a718cbe..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.int64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.int8.doctree b/docs/latest/.doctrees/python/generated/pyarrow.int8.doctree
deleted file mode 100644
index 337e517..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.int8.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_file.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_file.doctree
deleted file mode 100644
index 4603d44..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_file.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_stream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_stream.doctree
deleted file mode 100644
index 3de49a1..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.ipc.open_stream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.list_.doctree b/docs/latest/.doctrees/python/generated/pyarrow.list_.doctree
deleted file mode 100644
index 39f9736..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.list_.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.log_memory_allocations.doctree b/docs/latest/.doctrees/python/generated/pyarrow.log_memory_allocations.doctree
deleted file mode 100644
index e3a81a2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.log_memory_allocations.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.memory_map.doctree b/docs/latest/.doctrees/python/generated/pyarrow.memory_map.doctree
deleted file mode 100644
index 0209f2c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.memory_map.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.null.doctree b/docs/latest/.doctrees/python/generated/pyarrow.null.doctree
deleted file mode 100644
index b096659..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.null.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.open_file.doctree b/docs/latest/.doctrees/python/generated/pyarrow.open_file.doctree
deleted file mode 100644
index 5e810b1..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.open_file.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.open_stream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.open_stream.doctree
deleted file mode 100644
index cf463fd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.open_stream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.output_stream.doctree b/docs/latest/.doctrees/python/generated/pyarrow.output_stream.doctree
deleted file mode 100644
index a8703f8..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.output_stream.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetDataset.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetDataset.doctree
deleted file mode 100644
index e3d26b6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetDataset.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetFile.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetFile.doctree
deleted file mode 100644
index 968f795..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetFile.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetWriter.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetWriter.doctree
deleted file mode 100644
index 70d5c5c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.ParquetWriter.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_metadata.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_metadata.doctree
deleted file mode 100644
index e0227b2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_metadata.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_pandas.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_pandas.doctree
deleted file mode 100644
index 8771b77..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_pandas.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_schema.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_schema.doctree
deleted file mode 100644
index 6eb538a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_schema.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_table.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_table.doctree
deleted file mode 100644
index 8bf764a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.read_table.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_metadata.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_metadata.doctree
deleted file mode 100644
index 0b625f7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_metadata.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_table.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_table.doctree
deleted file mode 100644
index 723fb10..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_table.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_to_dataset.doctree b/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_to_dataset.doctree
deleted file mode 100644
index 94454c3..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.parquet.write_to_dataset.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.plasma.ObjectID.doctree b/docs/latest/.doctrees/python/generated/pyarrow.plasma.ObjectID.doctree
deleted file mode 100644
index cf92a80..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.plasma.ObjectID.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaBuffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaBuffer.doctree
deleted file mode 100644
index 6c796ba..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaBuffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaClient.doctree b/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaClient.doctree
deleted file mode 100644
index 1a5f90d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.plasma.PlasmaClient.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.py_buffer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.py_buffer.doctree
deleted file mode 100644
index 4dbbae8..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.py_buffer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.read_message.doctree b/docs/latest/.doctrees/python/generated/pyarrow.read_message.doctree
deleted file mode 100644
index 5ca5ea4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.read_message.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.read_record_batch.doctree b/docs/latest/.doctrees/python/generated/pyarrow.read_record_batch.doctree
deleted file mode 100644
index 872cc44..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.read_record_batch.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.read_serialized.doctree b/docs/latest/.doctrees/python/generated/pyarrow.read_serialized.doctree
deleted file mode 100644
index 0dca2f7..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.read_serialized.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.read_tensor.doctree b/docs/latest/.doctrees/python/generated/pyarrow.read_tensor.doctree
deleted file mode 100644
index edd8ce0..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.read_tensor.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.serialize.doctree b/docs/latest/.doctrees/python/generated/pyarrow.serialize.doctree
deleted file mode 100644
index 1280fba..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.serialize.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.serialize_to.doctree b/docs/latest/.doctrees/python/generated/pyarrow.serialize_to.doctree
deleted file mode 100644
index a44eabd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.serialize_to.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.set_cpu_count.doctree b/docs/latest/.doctrees/python/generated/pyarrow.set_cpu_count.doctree
deleted file mode 100644
index e92aac5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.set_cpu_count.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.set_memory_pool.doctree b/docs/latest/.doctrees/python/generated/pyarrow.set_memory_pool.doctree
deleted file mode 100644
index dffb195..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.set_memory_pool.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.string.doctree b/docs/latest/.doctrees/python/generated/pyarrow.string.doctree
deleted file mode 100644
index 212bb45..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.string.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.struct.doctree b/docs/latest/.doctrees/python/generated/pyarrow.struct.doctree
deleted file mode 100644
index 9109b79..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.struct.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.time32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.time32.doctree
deleted file mode 100644
index 2e0f3b3..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.time32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.time64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.time64.doctree
deleted file mode 100644
index 197c17b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.time64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.timestamp.doctree b/docs/latest/.doctrees/python/generated/pyarrow.timestamp.doctree
deleted file mode 100644
index 507ba00..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.timestamp.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.total_allocated_bytes.doctree b/docs/latest/.doctrees/python/generated/pyarrow.total_allocated_bytes.doctree
deleted file mode 100644
index b5538cf..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.total_allocated_bytes.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_binary.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_binary.doctree
deleted file mode 100644
index 5b1ad7f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_binary.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_boolean.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_boolean.doctree
deleted file mode 100644
index 58b258a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_boolean.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_date.doctree
deleted file mode 100644
index 581f9e1..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_date32.doctree
deleted file mode 100644
index e7d4673..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_date64.doctree
deleted file mode 100644
index 452126f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_date64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_decimal.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_decimal.doctree
deleted file mode 100644
index f105bdc..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_decimal.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_dictionary.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_dictionary.doctree
deleted file mode 100644
index ee911c4..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_dictionary.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_fixed_size_binary.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_fixed_size_binary.doctree
deleted file mode 100644
index 65a4854..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_fixed_size_binary.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_float16.doctree
deleted file mode 100644
index 8e13f75..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_float32.doctree
deleted file mode 100644
index aea0e46..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_float64.doctree
deleted file mode 100644
index a7e71b0..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_float64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_floating.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_floating.doctree
deleted file mode 100644
index 93bde72..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_floating.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_int16.doctree
deleted file mode 100644
index 238f078..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_int32.doctree
deleted file mode 100644
index 66fbf89..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_int64.doctree
deleted file mode 100644
index aadc598..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int8.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_int8.doctree
deleted file mode 100644
index 7020f7a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_int8.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_integer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_integer.doctree
deleted file mode 100644
index b91ca8f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_integer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_list.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_list.doctree
deleted file mode 100644
index 49904e2..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_list.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_map.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_map.doctree
deleted file mode 100644
index eeefade..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_map.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_nested.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_nested.doctree
deleted file mode 100644
index 7b7f06c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_nested.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_null.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_null.doctree
deleted file mode 100644
index 73feac9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_null.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_signed_integer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_signed_integer.doctree
deleted file mode 100644
index a9ad325..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_signed_integer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_string.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_string.doctree
deleted file mode 100644
index 82925dd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_string.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_struct.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_struct.doctree
deleted file mode 100644
index eefa60c..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_struct.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_temporal.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_temporal.doctree
deleted file mode 100644
index e814f7a..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_temporal.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_time.doctree
deleted file mode 100644
index ebbe207..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_time32.doctree
deleted file mode 100644
index 48aa601..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_time64.doctree
deleted file mode 100644
index 4a3af3b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_time64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_timestamp.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_timestamp.doctree
deleted file mode 100644
index 99f4440..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_timestamp.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint16.doctree
deleted file mode 100644
index 38ebc63..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint32.doctree
deleted file mode 100644
index 2a9039b..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint64.doctree
deleted file mode 100644
index f38384d..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint8.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint8.doctree
deleted file mode 100644
index 1309858..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_uint8.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_unicode.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_unicode.doctree
deleted file mode 100644
index 717f7fc..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_unicode.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_union.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_union.doctree
deleted file mode 100644
index 00cb6a6..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_union.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.types.is_unsigned_integer.doctree b/docs/latest/.doctrees/python/generated/pyarrow.types.is_unsigned_integer.doctree
deleted file mode 100644
index d85c756..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.types.is_unsigned_integer.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.uint16.doctree b/docs/latest/.doctrees/python/generated/pyarrow.uint16.doctree
deleted file mode 100644
index b20fcc9..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.uint16.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.uint32.doctree b/docs/latest/.doctrees/python/generated/pyarrow.uint32.doctree
deleted file mode 100644
index 8102e32..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.uint32.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.uint64.doctree b/docs/latest/.doctrees/python/generated/pyarrow.uint64.doctree
deleted file mode 100644
index 410835f..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.uint64.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.uint8.doctree b/docs/latest/.doctrees/python/generated/pyarrow.uint8.doctree
deleted file mode 100644
index 0e905e5..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.uint8.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.utf8.doctree b/docs/latest/.doctrees/python/generated/pyarrow.utf8.doctree
deleted file mode 100644
index 97cddbd..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.utf8.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/generated/pyarrow.write_tensor.doctree b/docs/latest/.doctrees/python/generated/pyarrow.write_tensor.doctree
deleted file mode 100644
index f8ff880..0000000
Binary files a/docs/latest/.doctrees/python/generated/pyarrow.write_tensor.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/getting_involved.doctree b/docs/latest/.doctrees/python/getting_involved.doctree
deleted file mode 100644
index 142af7d..0000000
Binary files a/docs/latest/.doctrees/python/getting_involved.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/index.doctree b/docs/latest/.doctrees/python/index.doctree
deleted file mode 100644
index 142b658..0000000
Binary files a/docs/latest/.doctrees/python/index.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/install.doctree b/docs/latest/.doctrees/python/install.doctree
deleted file mode 100644
index 7ec4cb6..0000000
Binary files a/docs/latest/.doctrees/python/install.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/ipc.doctree b/docs/latest/.doctrees/python/ipc.doctree
deleted file mode 100644
index d14e388..0000000
Binary files a/docs/latest/.doctrees/python/ipc.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/memory.doctree b/docs/latest/.doctrees/python/memory.doctree
deleted file mode 100644
index b7c9366..0000000
Binary files a/docs/latest/.doctrees/python/memory.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/numpy.doctree b/docs/latest/.doctrees/python/numpy.doctree
deleted file mode 100644
index 697bd1b..0000000
Binary files a/docs/latest/.doctrees/python/numpy.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/pandas.doctree b/docs/latest/.doctrees/python/pandas.doctree
deleted file mode 100644
index a37184c..0000000
Binary files a/docs/latest/.doctrees/python/pandas.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/parquet.doctree b/docs/latest/.doctrees/python/parquet.doctree
deleted file mode 100644
index c02b5af..0000000
Binary files a/docs/latest/.doctrees/python/parquet.doctree and /dev/null differ
diff --git a/docs/latest/.doctrees/python/plasma.doctree b/docs/latest/.doctrees/python/plasma.doctree
deleted file mode 100644
index 4d57de8..0000000
Binary files a/docs/latest/.doctrees/python/plasma.doctree and /dev/null differ
diff --git a/docs/latest/_modules/index.html b/docs/latest/_modules/index.html
deleted file mode 100644
index befce19..0000000
--- a/docs/latest/_modules/index.html
+++ /dev/null
@@ -1,226 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>Overview: module code — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../genindex.html" />
- <link rel="search" title="Search" href="../search.html" />
-
-
- <script src="../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../index.html">Docs</a> »</li>
-
- <li>Overview: module code</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>All modules for which code is available</h1>
-<ul><li><a href="pyarrow.html">pyarrow</a></li>
-<ul><li><a href="pyarrow/_csv.html">pyarrow._csv</a></li>
-<li><a href="pyarrow/_plasma.html">pyarrow._plasma</a></li>
-<li><a href="pyarrow/feather.html">pyarrow.feather</a></li>
-<li><a href="pyarrow/filesystem.html">pyarrow.filesystem</a></li>
-<li><a href="pyarrow/hdfs.html">pyarrow.hdfs</a></li>
-<li><a href="pyarrow/ipc.html">pyarrow.ipc</a></li>
-<li><a href="pyarrow/lib.html">pyarrow.lib</a></li>
-<li><a href="pyarrow/parquet.html">pyarrow.parquet</a></li>
-<li><a href="pyarrow/types.html">pyarrow.types</a></li>
-</ul></ul>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../_static/jquery.js"></script>
- <script type="text/javascript" src="../_static/underscore.js"></script>
- <script type="text/javascript" src="../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow.html b/docs/latest/_modules/pyarrow.html
deleted file mode 100644
index 7b0004b..0000000
--- a/docs/latest/_modules/pyarrow.html
+++ /dev/null
@@ -1,478 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../genindex.html" />
- <link rel="search" title="Search" href="../search.html" />
-
-
- <script src="../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../index.html">Docs</a> »</li>
-
- <li><a href="index.html">Module code</a> »</li>
-
- <li>pyarrow</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="c1"># flake8: noqa</span>
-
-<span class="kn">import</span> <span class="nn">os</span> <span class="k">as</span> <span class="nn">_os</span>
-<span class="kn">import</span> <span class="nn">sys</span> <span class="k">as</span> <span class="nn">_sys</span>
-
-<span class="k">try</span><span class="p">:</span>
- <span class="kn">from</span> <span class="nn">._generated_version</span> <span class="k">import</span> <span class="n">version</span> <span class="k">as</span> <span class="n">__version__</span>
-<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
- <span class="c1"># Package is not installed, parse git tag at runtime</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="kn">import</span> <span class="nn">setuptools_scm</span>
- <span class="c1"># Code duplicated from setup.py to avoid a dependency on each other</span>
- <span class="k">def</span> <span class="nf">parse_git</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Parse function for setuptools_scm that ignores tags for non-C++</span>
-<span class="sd"> subprojects, e.g. apache-arrow-js-XXX tags.</span>
-<span class="sd"> """</span>
- <span class="kn">from</span> <span class="nn">setuptools_scm.git</span> <span class="k">import</span> <span class="n">parse</span>
- <span class="n">kwargs</span><span class="p">[</span><span class="s1">'describe_command'</span><span class="p">]</span> <span class="o">=</span> \
- <span class="s2">"git describe --dirty --tags --long --match 'apache-arrow-[0-9].*'"</span>
- <span class="k">return</span> <span class="n">parse</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
- <span class="n">__version__</span> <span class="o">=</span> <span class="n">setuptools_scm</span><span class="o">.</span><span class="n">get_version</span><span class="p">(</span><span class="s1">'../'</span><span class="p">,</span>
- <span class="n">parse</span><span class="o">=</span><span class="n">parse_git</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
- <span class="n">__version__</span> <span class="o">=</span> <span class="kc">None</span>
-
-
-<span class="kn">import</span> <span class="nn">pyarrow.compat</span> <span class="k">as</span> <span class="nn">compat</span>
-
-<span class="c1"># Workaround for https://issues.apache.org/jira/browse/ARROW-2657</span>
-<span class="c1"># and https://issues.apache.org/jira/browse/ARROW-2920</span>
-<span class="k">if</span> <span class="n">_sys</span><span class="o">.</span><span class="n">platform</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'linux'</span><span class="p">,</span> <span class="s1">'linux2'</span><span class="p">):</span>
- <span class="n">compat</span><span class="o">.</span><span class="n">import_tensorflow_extension</span><span class="p">()</span>
- <span class="n">compat</span><span class="o">.</span><span class="n">import_pytorch_extension</span><span class="p">()</span>
-
-
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="n">cpu_count</span><span class="p">,</span> <span class="n">set_cpu_count</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">null</span><span class="p">,</span> <span class="n">bool_</span><span class="p">,</span>
- <span class="n">int8</span><span class="p">,</span> <span class="n">int16</span><span class="p">,</span> <span class="n">int32</span><span class="p">,</span> <span class="n">int64</span><span class="p">,</span>
- <span class="n">uint8</span><span class="p">,</span> <span class="n">uint16</span><span class="p">,</span> <span class="n">uint32</span><span class="p">,</span> <span class="n">uint64</span><span class="p">,</span>
- <span class="n">time32</span><span class="p">,</span> <span class="n">time64</span><span class="p">,</span> <span class="n">timestamp</span><span class="p">,</span> <span class="n">date32</span><span class="p">,</span> <span class="n">date64</span><span class="p">,</span>
- <span class="n">float16</span><span class="p">,</span> <span class="n">float32</span><span class="p">,</span> <span class="n">float64</span><span class="p">,</span>
- <span class="n">binary</span><span class="p">,</span> <span class="n">string</span><span class="p">,</span> <span class="n">utf8</span><span class="p">,</span> <span class="n">decimal128</span><span class="p">,</span>
- <span class="n">list_</span><span class="p">,</span> <span class="n">struct</span><span class="p">,</span> <span class="n">union</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">,</span> <span class="n">field</span><span class="p">,</span>
- <span class="n">type_for_alias</span><span class="p">,</span>
- <span class="n">DataType</span><span class="p">,</span>
- <span class="n">Field</span><span class="p">,</span>
- <span class="n">Schema</span><span class="p">,</span>
- <span class="n">schema</span><span class="p">,</span>
- <span class="n">Array</span><span class="p">,</span> <span class="n">Tensor</span><span class="p">,</span>
- <span class="n">array</span><span class="p">,</span> <span class="n">chunked_array</span><span class="p">,</span> <span class="n">column</span><span class="p">,</span>
- <span class="n">from_numpy_dtype</span><span class="p">,</span>
- <span class="n">NullArray</span><span class="p">,</span>
- <span class="n">NumericArray</span><span class="p">,</span> <span class="n">IntegerArray</span><span class="p">,</span> <span class="n">FloatingPointArray</span><span class="p">,</span>
- <span class="n">BooleanArray</span><span class="p">,</span>
- <span class="n">Int8Array</span><span class="p">,</span> <span class="n">UInt8Array</span><span class="p">,</span>
- <span class="n">Int16Array</span><span class="p">,</span> <span class="n">UInt16Array</span><span class="p">,</span>
- <span class="n">Int32Array</span><span class="p">,</span> <span class="n">UInt32Array</span><span class="p">,</span>
- <span class="n">Int64Array</span><span class="p">,</span> <span class="n">UInt64Array</span><span class="p">,</span>
- <span class="n">ListArray</span><span class="p">,</span> <span class="n">UnionArray</span><span class="p">,</span>
- <span class="n">BinaryArray</span><span class="p">,</span> <span class="n">StringArray</span><span class="p">,</span>
- <span class="n">FixedSizeBinaryArray</span><span class="p">,</span>
- <span class="n">DictionaryArray</span><span class="p">,</span>
- <span class="n">Date32Array</span><span class="p">,</span> <span class="n">Date64Array</span><span class="p">,</span>
- <span class="n">TimestampArray</span><span class="p">,</span> <span class="n">Time32Array</span><span class="p">,</span> <span class="n">Time64Array</span><span class="p">,</span>
- <span class="n">Decimal128Array</span><span class="p">,</span> <span class="n">StructArray</span><span class="p">,</span>
- <span class="n">ArrayValue</span><span class="p">,</span> <span class="n">Scalar</span><span class="p">,</span> <span class="n">NA</span><span class="p">,</span> <span class="n">_NULL</span> <span class="k">as</span> <span class="n">NULL</span><span class="p">,</span>
- <span class="n">BooleanValue</span><span class="p">,</span>
- <span class="n">Int8Value</span><span class="p">,</span> <span class="n">Int16Value</span><span class="p">,</span> <span class="n">Int32Value</span><span class="p">,</span> <span class="n">Int64Value</span><span class="p">,</span>
- <span class="n">UInt8Value</span><span class="p">,</span> <span class="n">UInt16Value</span><span class="p">,</span> <span class="n">UInt32Value</span><span class="p">,</span> <span class="n">UInt64Value</span><span class="p">,</span>
- <span class="n">HalfFloatValue</span><span class="p">,</span> <span class="n">FloatValue</span><span class="p">,</span> <span class="n">DoubleValue</span><span class="p">,</span> <span class="n">ListValue</span><span class="p">,</span>
- <span class="n">BinaryValue</span><span class="p">,</span> <span class="n">StringValue</span><span class="p">,</span> <span class="n">FixedSizeBinaryValue</span><span class="p">,</span>
- <span class="n">DecimalValue</span><span class="p">,</span> <span class="n">UnionValue</span><span class="p">,</span> <span class="n">StructValue</span><span class="p">,</span> <span class="n">DictionaryValue</span><span class="p">,</span>
- <span class="n">Date32Value</span><span class="p">,</span> <span class="n">Date64Value</span><span class="p">,</span>
- <span class="n">Time32Value</span><span class="p">,</span> <span class="n">Time64Value</span><span class="p">,</span>
- <span class="n">TimestampValue</span><span class="p">)</span>
-
-<span class="c1"># Buffers, allocation</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">Buffer</span><span class="p">,</span> <span class="n">ResizableBuffer</span><span class="p">,</span> <span class="n">foreign_buffer</span><span class="p">,</span> <span class="n">py_buffer</span><span class="p">,</span>
- <span class="n">compress</span><span class="p">,</span> <span class="n">decompress</span><span class="p">,</span> <span class="n">allocate_buffer</span><span class="p">)</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">MemoryPool</span><span class="p">,</span> <span class="n">LoggingMemoryPool</span><span class="p">,</span> <span class="n">ProxyMemoryPool</span><span class="p">,</span>
- <span class="n">total_allocated_bytes</span><span class="p">,</span> <span class="n">set_memory_pool</span><span class="p">,</span>
- <span class="n">default_memory_pool</span><span class="p">,</span> <span class="n">logging_memory_pool</span><span class="p">,</span>
- <span class="n">proxy_memory_pool</span><span class="p">,</span> <span class="n">log_memory_allocations</span><span class="p">)</span>
-
-<span class="c1"># I/O</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">HdfsFile</span><span class="p">,</span> <span class="n">NativeFile</span><span class="p">,</span> <span class="n">PythonFile</span><span class="p">,</span>
- <span class="n">CompressedInputStream</span><span class="p">,</span> <span class="n">CompressedOutputStream</span><span class="p">,</span>
- <span class="n">FixedSizeBufferWriter</span><span class="p">,</span>
- <span class="n">BufferReader</span><span class="p">,</span> <span class="n">BufferOutputStream</span><span class="p">,</span>
- <span class="n">OSFile</span><span class="p">,</span> <span class="n">MemoryMappedFile</span><span class="p">,</span> <span class="n">memory_map</span><span class="p">,</span>
- <span class="n">create_memory_map</span><span class="p">,</span> <span class="n">have_libhdfs</span><span class="p">,</span> <span class="n">have_libhdfs3</span><span class="p">,</span>
- <span class="n">MockOutputStream</span><span class="p">,</span> <span class="n">input_stream</span><span class="p">,</span> <span class="n">output_stream</span><span class="p">)</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">ChunkedArray</span><span class="p">,</span> <span class="n">Column</span><span class="p">,</span> <span class="n">RecordBatch</span><span class="p">,</span> <span class="n">Table</span><span class="p">,</span>
- <span class="n">concat_tables</span><span class="p">)</span>
-
-<span class="c1"># Exceptions</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">ArrowException</span><span class="p">,</span>
- <span class="n">ArrowKeyError</span><span class="p">,</span>
- <span class="n">ArrowInvalid</span><span class="p">,</span>
- <span class="n">ArrowIOError</span><span class="p">,</span>
- <span class="n">ArrowMemoryError</span><span class="p">,</span>
- <span class="n">ArrowNotImplementedError</span><span class="p">,</span>
- <span class="n">ArrowTypeError</span><span class="p">,</span>
- <span class="n">ArrowSerializationError</span><span class="p">,</span>
- <span class="n">PlasmaObjectExists</span><span class="p">)</span>
-
-<span class="c1"># Serialization</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">deserialize_from</span><span class="p">,</span> <span class="n">deserialize</span><span class="p">,</span>
- <span class="n">deserialize_components</span><span class="p">,</span>
- <span class="n">serialize</span><span class="p">,</span> <span class="n">serialize_to</span><span class="p">,</span> <span class="n">read_serialized</span><span class="p">,</span>
- <span class="n">SerializedPyObject</span><span class="p">,</span> <span class="n">SerializationContext</span><span class="p">,</span>
- <span class="n">SerializationCallbackError</span><span class="p">,</span>
- <span class="n">DeserializationCallbackError</span><span class="p">)</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.filesystem</span> <span class="k">import</span> <span class="n">FileSystem</span><span class="p">,</span> <span class="n">LocalFileSystem</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.hdfs</span> <span class="k">import</span> <span class="n">HadoopFileSystem</span>
-<span class="kn">import</span> <span class="nn">pyarrow.hdfs</span> <span class="k">as</span> <span class="nn">hdfs</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.ipc</span> <span class="k">import</span> <span class="p">(</span><span class="n">Message</span><span class="p">,</span> <span class="n">MessageReader</span><span class="p">,</span>
- <span class="n">RecordBatchFileReader</span><span class="p">,</span> <span class="n">RecordBatchFileWriter</span><span class="p">,</span>
- <span class="n">RecordBatchStreamReader</span><span class="p">,</span> <span class="n">RecordBatchStreamWriter</span><span class="p">,</span>
- <span class="n">read_message</span><span class="p">,</span> <span class="n">read_record_batch</span><span class="p">,</span> <span class="n">read_schema</span><span class="p">,</span>
- <span class="n">read_tensor</span><span class="p">,</span> <span class="n">write_tensor</span><span class="p">,</span>
- <span class="n">get_record_batch_size</span><span class="p">,</span> <span class="n">get_tensor_size</span><span class="p">,</span>
- <span class="n">open_stream</span><span class="p">,</span>
- <span class="n">open_file</span><span class="p">,</span>
- <span class="n">serialize_pandas</span><span class="p">,</span> <span class="n">deserialize_pandas</span><span class="p">)</span>
-<span class="kn">import</span> <span class="nn">pyarrow.ipc</span> <span class="k">as</span> <span class="nn">ipc</span>
-
-
-<div class="viewcode-block" id="open_stream"><a class="viewcode-back" href="../python/generated/pyarrow.open_stream.html#pyarrow.open_stream">[docs]</a><span class="k">def</span> <span class="nf">open_stream</span><span class="p">(</span><span class="n">source</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> pyarrow.open_stream deprecated since 0.12, use pyarrow.ipc.open_stream</span>
-<span class="sd"> """</span>
- <span class="kn">import</span> <span class="nn">warnings</span>
- <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"pyarrow.open_stream is deprecated, please use "</span>
- <span class="s2">"pyarrow.ipc.open_stream"</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">ipc</span><span class="o">.</span><span class="n">open_stream</span><span class="p">(</span><span class="n">source</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="open_file"><a class="viewcode-back" href="../python/generated/pyarrow.open_file.html#pyarrow.open_file">[docs]</a><span class="k">def</span> <span class="nf">open_file</span><span class="p">(</span><span class="n">source</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> pyarrow.open_file deprecated since 0.12, use pyarrow.ipc.open_file</span>
-<span class="sd"> """</span>
- <span class="kn">import</span> <span class="nn">warnings</span>
- <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"pyarrow.open_file is deprecated, please use "</span>
- <span class="s2">"pyarrow.ipc.open_file"</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">ipc</span><span class="o">.</span><span class="n">open_file</span><span class="p">(</span><span class="n">source</span><span class="p">)</span></div>
-
-
-<span class="n">localfs</span> <span class="o">=</span> <span class="n">LocalFileSystem</span><span class="o">.</span><span class="n">get_instance</span><span class="p">()</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.serialization</span> <span class="k">import</span> <span class="p">(</span><span class="n">default_serialization_context</span><span class="p">,</span>
- <span class="n">register_default_serialization_handlers</span><span class="p">,</span>
- <span class="n">register_torch_serialization_handlers</span><span class="p">)</span>
-
-<span class="kn">import</span> <span class="nn">pyarrow.types</span> <span class="k">as</span> <span class="nn">types</span>
-
-<span class="c1"># Entry point for starting the plasma store</span>
-
-<span class="k">def</span> <span class="nf">_plasma_store_entry_point</span><span class="p">():</span>
- <span class="sd">"""Entry point for starting the plasma store.</span>
-
-<span class="sd"> This can be used by invoking e.g.</span>
-<span class="sd"> ``plasma_store -s /tmp/plasma -m 1000000000``</span>
-<span class="sd"> from the command line and will start the plasma_store executable with the</span>
-<span class="sd"> given arguments.</span>
-<span class="sd"> """</span>
- <span class="kn">import</span> <span class="nn">pyarrow</span>
- <span class="n">plasma_store_executable</span> <span class="o">=</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">__path__</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span>
- <span class="s2">"plasma_store_server"</span><span class="p">)</span>
- <span class="n">_os</span><span class="o">.</span><span class="n">execv</span><span class="p">(</span><span class="n">plasma_store_executable</span><span class="p">,</span> <span class="n">_sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
-
-<span class="c1"># ----------------------------------------------------------------------</span>
-<span class="c1"># Deprecations</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.util</span> <span class="k">import</span> <span class="n">_deprecate_api</span> <span class="c1"># noqa</span>
-
-<span class="c1"># ----------------------------------------------------------------------</span>
-<span class="c1"># Returning absolute path to the pyarrow include directory (if bundled, e.g. in</span>
-<span class="c1"># wheels)</span>
-
-<div class="viewcode-block" id="get_include"><a class="viewcode-back" href="../python/generated/pyarrow.get_include.html#pyarrow.get_include">[docs]</a><span class="k">def</span> <span class="nf">get_include</span><span class="p">():</span>
- <span class="sd">"""</span>
-<span class="sd"> Return absolute path to directory containing Arrow C++ include</span>
-<span class="sd"> headers. Similar to numpy.get_include</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="vm">__file__</span><span class="p">),</span> <span class="s1">'include'</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="get_libraries"><a class="viewcode-back" href="../python/generated/pyarrow.get_libraries.html#pyarrow.get_libraries">[docs]</a><span class="k">def</span> <span class="nf">get_libraries</span><span class="p">():</span>
- <span class="sd">"""</span>
-<span class="sd"> Return list of library names to include in the `libraries` argument for C</span>
-<span class="sd"> or Cython extensions using pyarrow</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="p">[</span><span class="s1">'arrow'</span><span class="p">,</span> <span class="s1">'arrow_python'</span><span class="p">]</span></div>
-
-
-<div class="viewcode-block" id="get_library_dirs"><a class="viewcode-back" href="../python/generated/pyarrow.get_library_dirs.html#pyarrow.get_library_dirs">[docs]</a><span class="k">def</span> <span class="nf">get_library_dirs</span><span class="p">():</span>
- <span class="sd">"""</span>
-<span class="sd"> Return lists of directories likely to contain Arrow C++ libraries for</span>
-<span class="sd"> linking C or Cython extensions using pyarrow</span>
-<span class="sd"> """</span>
- <span class="n">package_cwd</span> <span class="o">=</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="vm">__file__</span><span class="p">)</span>
-
- <span class="n">library_dirs</span> <span class="o">=</span> <span class="p">[</span><span class="n">package_cwd</span><span class="p">]</span>
-
- <span class="c1"># Search library paths via pkg-config. This is necessary if the user</span>
- <span class="c1"># installed libarrow and the other shared libraries manually and they</span>
- <span class="c1"># are not shipped inside the pyarrow package (see also ARROW-2976).</span>
- <span class="kn">from</span> <span class="nn">subprocess</span> <span class="k">import</span> <span class="n">call</span><span class="p">,</span> <span class="n">PIPE</span><span class="p">,</span> <span class="n">Popen</span>
- <span class="n">pkg_config_executable</span> <span class="o">=</span> <span class="n">_os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'PKG_CONFIG'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="ow">or</span> <span class="s1">'pkg-config'</span>
- <span class="k">for</span> <span class="n">package</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">"arrow"</span><span class="p">,</span> <span class="s2">"plasma"</span><span class="p">,</span> <span class="s2">"arrow_python"</span><span class="p">]:</span>
- <span class="n">cmd</span> <span class="o">=</span> <span class="s1">'</span><span class="si">{0}</span><span class="s1"> --exists </span><span class="si">{1}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">pkg_config_executable</span><span class="p">,</span> <span class="n">package</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">call</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">cmd</span> <span class="o">=</span> <span class="p">[</span><span class="n">pkg_config_executable</span><span class="p">,</span> <span class="s2">"--libs-only-L"</span><span class="p">,</span> <span class="n">package</span><span class="p">]</span>
- <span class="n">proc</span> <span class="o">=</span> <span class="n">Popen</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">PIPE</span><span class="p">)</span>
- <span class="n">out</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">proc</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
- <span class="n">library_dir</span> <span class="o">=</span> <span class="n">out</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)[</span><span class="mi">2</span><span class="p">:]</span> <span class="c1"># strip "-L"</span>
- <span class="k">if</span> <span class="n">library_dir</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">library_dirs</span><span class="p">:</span>
- <span class="n">library_dirs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">library_dir</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
- <span class="k">pass</span>
-
- <span class="k">if</span> <span class="n">_sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s1">'win32'</span><span class="p">:</span>
- <span class="c1"># TODO(wesm): Is this necessary, or does setuptools within a conda</span>
- <span class="c1"># installation add Library\lib to the linker path for MSVC?</span>
- <span class="n">python_base_install</span> <span class="o">=</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">_sys</span><span class="o">.</span><span class="n">executable</span><span class="p">)</span>
- <span class="n">library_lib</span> <span class="o">=</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">python_base_install</span><span class="p">,</span> <span class="s1">'Library'</span><span class="p">,</span> <span class="s1">'lib'</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">library_lib</span><span class="p">,</span> <span class="s1">'arrow.lib'</span><span class="p">)):</span>
- <span class="n">library_dirs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">library_lib</span><span class="p">)</span>
-
- <span class="c1"># ARROW-4074: Allow for ARROW_HOME to be set to some other directory</span>
- <span class="k">if</span> <span class="s1">'ARROW_HOME'</span> <span class="ow">in</span> <span class="n">_os</span><span class="o">.</span><span class="n">environ</span><span class="p">:</span>
- <span class="n">library_dirs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">_os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'ARROW_HOME'</span><span class="p">],</span> <span class="s1">'lib'</span><span cla [...]
-
- <span class="k">return</span> <span class="n">library_dirs</span></div>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../_static/jquery.js"></script>
- <script type="text/javascript" src="../_static/underscore.js"></script>
- <script type="text/javascript" src="../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/feather.html b/docs/latest/_modules/pyarrow/feather.html
deleted file mode 100644
index 0e38446..0000000
--- a/docs/latest/_modules/pyarrow/feather.html
+++ /dev/null
@@ -1,455 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.feather — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.feather</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.feather</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="kn">from</span> <span class="nn">distutils.version</span> <span class="k">import</span> <span class="n">LooseVersion</span>
-<span class="kn">import</span> <span class="nn">os</span>
-
-<span class="kn">import</span> <span class="nn">six</span>
-<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.compat</span> <span class="k">import</span> <span class="n">pdapi</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="n">FeatherError</span> <span class="c1"># noqa</span>
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="n">Table</span><span class="p">,</span> <span class="n">concat_tables</span>
-<span class="kn">import</span> <span class="nn">pyarrow.lib</span> <span class="k">as</span> <span class="nn">ext</span>
-
-
-<span class="k">try</span><span class="p">:</span>
- <span class="n">infer_dtype</span> <span class="o">=</span> <span class="n">pdapi</span><span class="o">.</span><span class="n">infer_dtype</span>
-<span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
- <span class="n">infer_dtype</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">infer_dtype</span>
-
-
-<span class="k">if</span> <span class="n">LooseVersion</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">__version__</span><span class="p">)</span> <span class="o"><</span> <span class="s1">'0.17.0'</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ImportError</span><span class="p">(</span><span class="s2">"feather requires pandas >= 0.17.0"</span><span class="p">)</span>
-
-
-<span class="k">class</span> <span class="nc">FeatherReader</span><span class="p">(</span><span class="n">ext</span><span class="o">.</span><span class="n">FeatherReader</span><span class="p">):</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">source</span> <span class="o">=</span> <span class="n">source</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">source</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">read_table</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">columns</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_read</span><span class="p">()</span>
- <span class="n">column_types</span> <span class="o">=</span> <span class="p">[</span><span class="nb">type</span><span class="p">(</span><span class="n">column</span><span class="p">)</span> <span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">columns</span><span class="p">]</span>
- <span class="k">if</span> <span class="nb">all</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">t</span><span class="p">:</span> <span class="n">t</span> <span class="o">==</span> <span class="nb">int</span><span class="p">,</span> <span class="n">column_types</span><span class="p">)):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_read_indices</span><span class="p">(</span><span class="n">columns</span><span class="p">)</span>
- <span class="k">elif</span> <span class="nb">all</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">t</span><span class="p">:</span> <span class="n">t</span> <span class="o">==</span> <span class="nb">str</span><span class="p">,</span> <span class="n">column_types</span><span class="p">)):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_read_names</span><span class="p">(</span><span class="n">columns</span><span class="p">)</span>
-
- <span class="n">column_type_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">t</span><span class="o">.</span><span class="vm">__name__</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">column_types</span><span class="p">]</span>
- <span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"Columns must be indices or names. "</span>
- <span class="s2">"Got columns </span><span class="si">{}</span><span class="s2"> of types </span><span class="si">{}</span><span class="s2">"</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">columns</span><span class="p">,</span> <span class="n">column_type_names</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">read_pandas</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">(</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">check_chunked_overflow</span><span class="p">(</span><span class="n">col</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">col</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">num_chunks</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
- <span class="k">return</span>
-
- <span class="k">if</span> <span class="n">col</span><span class="o">.</span><span class="n">type</span> <span class="ow">in</span> <span class="p">(</span><span class="n">ext</span><span class="o">.</span><span class="n">binary</span><span class="p">(),</span> <span class="n">ext</span><span class="o">.</span><span class="n">string</span><span class="p">()):</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Column '</span><span class="si">{0}</span><span class="s2">' exceeds 2GB maximum capacity of "</span>
- <span class="s2">"a Feather binary column. This restriction may be "</span>
- <span class="s2">"lifted in the future"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">col</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="c1"># TODO(wesm): Not sure when else this might be reached</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Column '</span><span class="si">{0}</span><span class="s2">' of type </span><span class="si">{1}</span><span class="s2"> was chunked on conversion "</span>
- <span class="s2">"to Arrow and cannot be currently written to "</span>
- <span class="s2">"Feather format"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">col</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">col</span><span class="o">.</span><span class="n">type</span><span class="p">)))</span>
-
-
-<span class="k">class</span> <span class="nc">FeatherWriter</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dest</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">dest</span> <span class="o">=</span> <span class="n">dest</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span> <span class="o">=</span> <span class="n">ext</span><span class="o">.</span><span class="n">FeatherWriter</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">dest</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">write</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">df</span><span class="p">):</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">pd</span><span class="o">.</span><span class="n">SparseDataFrame</span><span class="p">):</span>
- <span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">to_dense</span><span class="p">()</span>
-
- <span class="k">if</span> <span class="ow">not</span> <span class="n">df</span><span class="o">.</span><span class="n">columns</span><span class="o">.</span><span class="n">is_unique</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"cannot serialize duplicate column names"</span><span class="p">)</span>
-
- <span class="c1"># TODO(wesm): Remove this length check, see ARROW-1732</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">columns</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">Table</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">preserve_index</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">name</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">names</span><span class="p">):</span>
- <span class="n">col</span> <span class="o">=</span> <span class="n">table</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
- <span class="n">check_chunked_overflow</span><span class="p">(</span><span class="n">col</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span><span class="o">.</span><span class="n">write_array</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">col</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">chunk</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
-
-
-<span class="k">class</span> <span class="nc">FeatherDataset</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Encapsulates details of reading a list of Feather files.</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path_or_paths : List[str]</span>
-<span class="sd"> A list of file names</span>
-<span class="sd"> validate_schema : boolean, default True</span>
-<span class="sd"> Check that individual file schemas are all the same / compatible</span>
-<span class="sd"> """</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path_or_paths</span><span class="p">,</span> <span class="n">validate_schema</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">paths</span> <span class="o">=</span> <span class="n">path_or_paths</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">validate_schema</span> <span class="o">=</span> <span class="n">validate_schema</span>
-
- <span class="k">def</span> <span class="nf">read_table</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read multiple feather files as a single pyarrow.Table</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns : List[str]</span>
-<span class="sd"> Names of columns to read from the file</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pyarrow.Table</span>
-<span class="sd"> Content of the file as a table (of columns)</span>
-<span class="sd"> """</span>
- <span class="n">_fil</span> <span class="o">=</span> <span class="n">FeatherReader</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">paths</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_tables</span> <span class="o">=</span> <span class="p">[</span><span class="n">_fil</span><span class="p">]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">_fil</span><span class="o">.</span><span class="n">schema</span>
-
- <span class="k">for</span> <span class="n">fil</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">paths</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
- <span class="n">fil_table</span> <span class="o">=</span> <span class="n">FeatherReader</span><span class="p">(</span><span class="n">fil</span><span class="p">)</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validate_schema</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">validate_schemas</span><span class="p">(</span><span class="n">fil</span><span class="p">,</span> <span class="n">fil_table</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">fil_table</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">concat_tables</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_tables</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">validate_schemas</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">piece</span><span class="p">,</span> <span class="n">table</span><span class="p">):</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">equals</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Schema in </span><span class="si">{0!s}</span><span class="s1"> was different. </span><span class="se">\n</span><span class="s1">'</span>
- <span class="s1">'</span><span class="si">{1!s}</span><span class="se">\n\n</span><span class="s1">vs</span><span class="se">\n\n</span><span class="si">{2!s}</span><span class="s1">'</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">piece</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
- <span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">read_pandas</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read multiple Parquet files as a single pandas DataFrame</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns : List[str]</span>
-<span class="sd"> Names of columns to read from the file</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Use multiple threads when converting to pandas</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pandas.DataFrame</span>
-<span class="sd"> Content of the file as a pandas DataFrame (of columns)</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">(</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span>
-
-
-<div class="viewcode-block" id="write_feather"><a class="viewcode-back" href="../../python/generated/pyarrow.feather.write_feather.html#pyarrow.feather.write_feather">[docs]</a><span class="k">def</span> <span class="nf">write_feather</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">dest</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Write a pandas.DataFrame to Feather format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> df : pandas.DataFrame</span>
-<span class="sd"> dest : string</span>
-<span class="sd"> Local file path</span>
-<span class="sd"> """</span>
- <span class="n">writer</span> <span class="o">=</span> <span class="n">FeatherWriter</span><span class="p">(</span><span class="n">dest</span><span class="p">)</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">writer</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
- <span class="c1"># Try to make sure the resource is closed</span>
- <span class="kn">import</span> <span class="nn">gc</span>
- <span class="n">writer</span> <span class="o">=</span> <span class="kc">None</span>
- <span class="n">gc</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">dest</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">string_types</span><span class="p">):</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">dest</span><span class="p">)</span>
- <span class="k">except</span> <span class="n">os</span><span class="o">.</span><span class="n">error</span><span class="p">:</span>
- <span class="k">pass</span>
- <span class="k">raise</span></div>
-
-
-<div class="viewcode-block" id="read_feather"><a class="viewcode-back" href="../../python/generated/pyarrow.feather.read_feather.html#pyarrow.feather.read_feather">[docs]</a><span class="k">def</span> <span class="nf">read_feather</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class= [...]
- <span class="sd">"""</span>
-<span class="sd"> Read a pandas.DataFrame from Feather format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : string file path, or file-like object</span>
-<span class="sd"> columns : sequence, optional</span>
-<span class="sd"> Only read a specific set of columns. If not provided, all columns are</span>
-<span class="sd"> read</span>
-<span class="sd"> use_threads: bool, default True</span>
-<span class="sd"> Whether to parallelize reading using multiple threads</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> df : pandas.DataFrame</span>
-<span class="sd"> """</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">FeatherReader</span><span class="p">(</span><span class="n">source</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_pandas</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span></div>
-
-
-<span class="k">def</span> <span class="nf">read_table</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read a pyarrow.Table from Feather format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : string file path, or file-like object</span>
-<span class="sd"> columns : sequence, optional</span>
-<span class="sd"> Only read a specific set of columns. If not provided, all columns are</span>
-<span class="sd"> read</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> table : pyarrow.Table</span>
-<span class="sd"> """</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">FeatherReader</span><span class="p">(</span><span class="n">source</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">)</span>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/filesystem.html b/docs/latest/_modules/pyarrow/filesystem.html
deleted file mode 100644
index 29ee84c..0000000
--- a/docs/latest/_modules/pyarrow/filesystem.html
+++ /dev/null
@@ -1,634 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.filesystem — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.filesystem</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.filesystem</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="kn">import</span> <span class="nn">os</span>
-<span class="kn">import</span> <span class="nn">inspect</span>
-<span class="kn">import</span> <span class="nn">posixpath</span>
-
-<span class="kn">from</span> <span class="nn">os.path</span> <span class="k">import</span> <span class="n">join</span> <span class="k">as</span> <span class="n">pjoin</span>
-<span class="kn">from</span> <span class="nn">six.moves.urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
-
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="kn">from</span> <span class="nn">pyarrow.util</span> <span class="k">import</span> <span class="n">implements</span><span class="p">,</span> <span class="n">_stringify_path</span>
-
-
-<span class="k">class</span> <span class="nc">FileSystem</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Abstract filesystem interface</span>
-<span class="sd"> """</span>
- <span class="k">def</span> <span class="nf">cat</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return contents of file as a bytes object</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> contents : bytes</span>
-<span class="sd"> """</span>
- <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">ls</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return list of file paths</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Delete the indicated file or directory</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : string</span>
-<span class="sd"> recursive : boolean, default False</span>
-<span class="sd"> If True, also delete child paths for directories</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">disk_usage</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Compute bytes used by all contents under indicated path in file tree</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : string</span>
-<span class="sd"> Can be a file path or directory</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> usage : int</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="n">path_info</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">path_info</span><span class="p">[</span><span class="s1">'kind'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'file'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">path_info</span><span class="p">[</span><span class="s1">'size'</span><span class="p">]</span>
-
- <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">for</span> <span class="n">root</span><span class="p">,</span> <span class="n">directories</span><span class="p">,</span> <span class="n">files</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="k">for</span> <span class="n">child_path</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
- <span class="n">abspath</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_path_join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">child_path</span><span class="p">)</span>
- <span class="n">total</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">abspath</span><span class="p">)[</span><span class="s1">'size'</span><span class="p">]</span>
-
- <span class="k">return</span> <span class="n">total</span>
-
- <span class="k">def</span> <span class="nf">_path_join</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">pathsep</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">stat</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> stat : dict</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s1">'FileSystem.stat'</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">rm</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Alias for FileSystem.delete</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">mv</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Alias for FileSystem.rename</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">rename</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Rename file, like UNIX mv command</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : string</span>
-<span class="sd"> Path to alter</span>
-<span class="sd"> new_path : string</span>
-<span class="sd"> Path to move to</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s1">'FileSystem.rename'</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">mkdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">create_parents</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">isdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if path is a directory</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">isfile</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if path is a file</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">_isfilestore</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Returns True if this FileSystem is a unix-style file store with</span>
-<span class="sd"> directories.</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="k">def</span> <span class="nf">read_parquet</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="kc">None</span> [...]
- <span class="n">use_threads</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read Parquet data from path in file system. Can read from a single file</span>
-<span class="sd"> or a directory of files</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : str</span>
-<span class="sd"> Single file path or directory</span>
-<span class="sd"> columns : List[str], optional</span>
-<span class="sd"> Subset of columns to read</span>
-<span class="sd"> metadata : pyarrow.parquet.FileMetaData</span>
-<span class="sd"> Known metadata to validate files against</span>
-<span class="sd"> schema : pyarrow.parquet.Schema</span>
-<span class="sd"> Known schema to validate files against. Alternative to metadata</span>
-<span class="sd"> argument</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Perform multi-threaded column reads</span>
-<span class="sd"> use_pandas_metadata : boolean, default False</span>
-<span class="sd"> If True and file has custom pandas schema metadata, ensure that</span>
-<span class="sd"> index columns are also loaded</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> table : pyarrow.Table</span>
-<span class="sd"> """</span>
- <span class="kn">from</span> <span class="nn">pyarrow.parquet</span> <span class="k">import</span> <span class="n">ParquetDataset</span>
- <span class="n">dataset</span> <span class="o">=</span> <span class="n">ParquetDataset</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">metadata</span><span class="p">,</span>
- <span class="n">filesystem</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">dataset</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">open</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'rb'</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Open file for reading or writing</span>
-<span class="sd"> """</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">pathsep</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="s1">'/'</span>
-
-
-<div class="viewcode-block" id="LocalFileSystem"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem">[docs]</a><span class="k">class</span> <span class="nc">LocalFileSystem</span><span class="p">(</span><span class="n">FileSystem</span><span class="p">):</span>
-
- <span class="n">_instance</span> <span class="o">=</span> <span class="kc">None</span>
-
-<div class="viewcode-block" id="LocalFileSystem.get_instance"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.get_instance">[docs]</a> <span class="nd">@classmethod</span>
- <span class="k">def</span> <span class="nf">get_instance</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span> <span class="o">=</span> <span class="n">LocalFileSystem</span><span class="p">()</span>
- <span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span></div>
-
-<div class="viewcode-block" id="LocalFileSystem.ls"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.ls">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">ls</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">ls</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">pjoin</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">path</span><span class="p">))</span></div>
-
-<div class="viewcode-block" id="LocalFileSystem.mkdir"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.mkdir">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">mkdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">mkdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">create_parents</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">create_parents</span><span class="p">:</span>
- <span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="LocalFileSystem.isdir"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.isdir">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="LocalFileSystem.isfile"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.isfile">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isfile</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isfile</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">_isfilestore</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">_isfilestore</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="kc">True</span>
-
-<div class="viewcode-block" id="LocalFileSystem.exists"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.exists">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">exists</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="LocalFileSystem.open"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.open">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">open</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">open</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'rb'</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Open file for reading or writing</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="nb">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">mode</span><span class="p">)</span></div>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">pathsep</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">sep</span>
-
-<div class="viewcode-block" id="LocalFileSystem.walk"><a class="viewcode-back" href="../../python/generated/pyarrow.LocalFileSystem.html#pyarrow.LocalFileSystem.walk">[docs]</a> <span class="k">def</span> <span class="nf">walk</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Directory tree generator, see os.walk</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div></div>
-
-
-<span class="k">class</span> <span class="nc">DaskFileSystem</span><span class="p">(</span><span class="n">FileSystem</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Wraps s3fs Dask filesystem implementation like s3fs, gcsfs, etc.</span>
-<span class="sd"> """</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fs</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">fs</span> <span class="o">=</span> <span class="n">fs</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">"Unsupported file system API"</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isfile</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isfile</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">"Unsupported file system API"</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">_isfilestore</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">_isfilestore</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Object Stores like S3 and GCSFS are based on key lookups, not true</span>
-<span class="sd"> file-paths</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="kc">False</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">delete</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">rm</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="n">recursive</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">exists</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">mkdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">mkdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">create_parents</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">create_parents</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">mkdirs</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">open</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">open</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'rb'</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Open file for reading or writing</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="n">mode</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">ls</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">ls</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="n">detail</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">walk</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Directory tree generator, like os.walk</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
-
-<span class="k">class</span> <span class="nc">S3FSWrapper</span><span class="p">(</span><span class="n">DaskFileSystem</span><span class="p">):</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">contents</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">ls</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">contents</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">path</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">False</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">True</span>
- <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">False</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isfile</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isfile</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">contents</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">ls</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">contents</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">path</span>
- <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">False</span>
-
- <span class="k">def</span> <span class="nf">walk</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">refresh</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Directory tree generator, like os.walk</span>
-
-<span class="sd"> Generator version of what is in s3fs, which yields a flattened list of</span>
-<span class="sd"> files</span>
-<span class="sd"> """</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'s3://'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
- <span class="n">directories</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
- <span class="n">files</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
-
- <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">_ls</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">refresh</span><span class="o">=</span><span class="n">refresh</span><span class="p">)):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">key</span><span class="p">[</span><span class="s1">'Key'</span><span class="p">]</span>
- <span class="k">if</span> <span class="n">key</span><span class="p">[</span><span class="s1">'StorageClass'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'DIRECTORY'</span><span class="p">:</span>
- <span class="n">directories</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">elif</span> <span class="n">key</span><span class="p">[</span><span class="s1">'StorageClass'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'BUCKET'</span><span class="p">:</span>
- <span class="k">pass</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">files</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
- <span class="c1"># s3fs creates duplicate 'DIRECTORY' entries</span>
- <span class="n">files</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">([</span><span class="n">posixpath</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">f</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span>
- <span class="k">if</span> <span class="n">f</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">directories</span><span class="p">])</span>
- <span class="n">directories</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">([</span><span class="n">posixpath</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">x</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span>
- <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">directories</span><span class="p">])</span>
-
- <span class="k">yield</span> <span class="n">path</span><span class="p">,</span> <span class="n">directories</span><span class="p">,</span> <span class="n">files</span>
-
- <span class="k">for</span> <span class="n">directory</span> <span class="ow">in</span> <span class="n">directories</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">tup</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">directory</span><span class="p">,</span> <span class="n">refresh</span><span class="o">=</span><span class="n">refresh</span><span class="p">):</span>
- <span class="k">yield</span> <span class="n">tup</span>
-
-
-<span class="k">def</span> <span class="nf">_ensure_filesystem</span><span class="p">(</span><span class="n">fs</span><span class="p">):</span>
- <span class="n">fs_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">fs</span><span class="p">)</span>
-
- <span class="c1"># If the arrow filesystem was subclassed, assume it supports the full</span>
- <span class="c1"># interface and return it</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">fs_type</span><span class="p">,</span> <span class="n">FileSystem</span><span class="p">):</span>
- <span class="k">for</span> <span class="n">mro</span> <span class="ow">in</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getmro</span><span class="p">(</span><span class="n">fs_type</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">mro</span><span class="o">.</span><span class="vm">__name__</span> <span class="ow">is</span> <span class="s1">'S3FileSystem'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">S3FSWrapper</span><span class="p">(</span><span class="n">fs</span><span class="p">)</span>
- <span class="c1"># In case its a simple LocalFileSystem (e.g. dask) use native arrow</span>
- <span class="c1"># FS</span>
- <span class="k">elif</span> <span class="n">mro</span><span class="o">.</span><span class="vm">__name__</span> <span class="ow">is</span> <span class="s1">'LocalFileSystem'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">LocalFileSystem</span><span class="o">.</span><span class="n">get_instance</span><span class="p">()</span>
-
- <span class="k">raise</span> <span class="ne">IOError</span><span class="p">(</span><span class="s1">'Unrecognized filesystem: </span><span class="si">{0}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">fs_type</span><span class="p">))</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">fs</span>
-
-
-<span class="k">def</span> <span class="nf">get_filesystem_from_uri</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> return filesystem from path which could be an HDFS URI</span>
-<span class="sd"> """</span>
- <span class="c1"># input can be hdfs URI such as hdfs://host:port/myfile.parquet</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="c1"># if _has_pathlib and isinstance(path, pathlib.Path):</span>
- <span class="c1"># path = str(path)</span>
- <span class="n">parsed_uri</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">parsed_uri</span><span class="o">.</span><span class="n">scheme</span> <span class="o">==</span> <span class="s1">'hdfs'</span><span class="p">:</span>
- <span class="n">netloc_split</span> <span class="o">=</span> <span class="n">parsed_uri</span><span class="o">.</span><span class="n">netloc</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">':'</span><span class="p">)</span>
- <span class="n">host</span> <span class="o">=</span> <span class="n">netloc_split</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
- <span class="k">if</span> <span class="n">host</span> <span class="o">==</span> <span class="s1">''</span><span class="p">:</span>
- <span class="n">host</span> <span class="o">=</span> <span class="s1">'default'</span>
- <span class="n">port</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">netloc_split</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">netloc_split</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">isnumeric</span><span class="p">():</span>
- <span class="n">port</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">netloc_split</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
- <span class="n">fs</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">hdfs</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">port</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">fs</span> <span class="o">=</span> <span class="n">LocalFileSystem</span><span class="o">.</span><span class="n">get_instance</span><span class="p">()</span>
-
- <span class="k">return</span> <span class="n">fs</span><span class="p">,</span> <span class="n">parsed_uri</span><span class="o">.</span><span class="n">path</span>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/hdfs.html b/docs/latest/_modules/pyarrow/hdfs.html
deleted file mode 100644
index 589b4cc..0000000
--- a/docs/latest/_modules/pyarrow/hdfs.html
+++ /dev/null
@@ -1,428 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.hdfs — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.hdfs</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.hdfs</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="kn">import</span> <span class="nn">os</span>
-<span class="kn">import</span> <span class="nn">posixpath</span>
-<span class="kn">import</span> <span class="nn">sys</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.util</span> <span class="k">import</span> <span class="n">implements</span>
-<span class="kn">from</span> <span class="nn">pyarrow.filesystem</span> <span class="k">import</span> <span class="n">FileSystem</span>
-<span class="kn">import</span> <span class="nn">pyarrow.lib</span> <span class="k">as</span> <span class="nn">lib</span>
-
-
-<div class="viewcode-block" id="HadoopFileSystem"><a class="viewcode-back" href="../../python/api.html#pyarrow.HadoopFileSystem">[docs]</a><span class="k">class</span> <span class="nc">HadoopFileSystem</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="n">FileSystem</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> FileSystem interface for HDFS cluster. See pyarrow.hdfs.connect for full</span>
-<span class="sd"> connection details</span>
-<span class="sd"> """</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">kerb_ticket< [...]
- <span class="n">driver</span><span class="o">=</span><span class="s1">'libhdfs'</span><span class="p">,</span> <span class="n">extra_conf</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">driver</span> <span class="o">==</span> <span class="s1">'libhdfs'</span><span class="p">:</span>
- <span class="n">_maybe_set_hadoop_classpath</span><span class="p">()</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">_connect</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span> <span class="n">user</span><span class="p">,</span> <span class="n">kerb_ticket</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">extra_conf</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">__reduce__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">port</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">user</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">kerb_ticket</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">driver</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">extra_conf</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">_isfilestore</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Returns True if this FileSystem is a unix-style file store with</span>
-<span class="sd"> directories.</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="kc">True</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isdir</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">isdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
- <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">isfile</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">isfile</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
-<div class="viewcode-block" id="HadoopFileSystem.delete"><a class="viewcode-back" href="../../python/generated/pyarrow.HadoopFileSystem.delete.html#pyarrow.HadoopFileSystem.delete">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">delete</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">recursive</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="HadoopFileSystem.mkdir"><a class="viewcode-back" href="../../python/generated/pyarrow.HadoopFileSystem.mkdir.html#pyarrow.HadoopFileSystem.mkdir">[docs]</a> <span class="k">def</span> <span class="nf">mkdir</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create directory in HDFS</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : string</span>
-<span class="sd"> Directory path to create, including any parent directories</span>
-
-<span class="sd"> Notes</span>
-<span class="sd"> -----</span>
-<span class="sd"> libhdfs does not support create_parents=False, so we ignore this here</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="HadoopFileSystem.rename"><a class="viewcode-back" href="../../python/generated/pyarrow.HadoopFileSystem.rename.html#pyarrow.HadoopFileSystem.rename">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">rename</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">rename</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">rename</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">new_path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="HadoopFileSystem.exists"><a class="viewcode-back" href="../../python/generated/pyarrow.HadoopFileSystem.exists.html#pyarrow.HadoopFileSystem.exists">[docs]</a> <span class="nd">@implements</span><span class="p">(</span><span class="n">FileSystem</span><span class="o">.</span><span class="n">exists</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="HadoopFileSystem.ls"><a class="viewcode-back" href="../../python/generated/pyarrow.HadoopFileSystem.ls.html#pyarrow.HadoopFileSystem.ls">[docs]</a> <span class="k">def</span> <span class="nf">ls</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Retrieve directory contents and metadata, if requested.</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : HDFS path</span>
-<span class="sd"> detail : boolean, default False</span>
-<span class="sd"> If False, only return list of paths</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> result : list of dicts (detail=True) or strings (detail=False)</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">HadoopFileSystem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">ls</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">detail</span><span class="p">)</span></div>
-
- <span class="k">def</span> <span class="nf">walk</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">top_path</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Directory tree generator for HDFS, like os.walk</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> top_path : string</span>
-<span class="sd"> Root directory for tree traversal</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> Generator yielding 3-tuple (dirpath, dirnames, filename)</span>
-<span class="sd"> """</span>
- <span class="n">contents</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ls</span><span class="p">(</span><span class="n">top_path</span><span class="p">,</span> <span class="n">detail</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
-
- <span class="n">directories</span><span class="p">,</span> <span class="n">files</span> <span class="o">=</span> <span class="n">_libhdfs_walk_files_dirs</span><span class="p">(</span><span class="n">top_path</span><span class="p">,</span> <span class="n">contents</span><span class="p">)</span>
- <span class="k">yield</span> <span class="n">top_path</span><span class="p">,</span> <span class="n">directories</span><span class="p">,</span> <span class="n">files</span>
- <span class="k">for</span> <span class="n">dirname</span> <span class="ow">in</span> <span class="n">directories</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">tup</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_path_join</span><span class="p">(</span><span class="n">top_path</span><span class="p">,</span> <span class="n">dirname</span><span class="p">)):</span>
- <span class="k">yield</span> <span class="n">tup</span></div>
-
-
-<span class="k">def</span> <span class="nf">_maybe_set_hadoop_classpath</span><span class="p">():</span>
- <span class="k">if</span> <span class="s1">'hadoop'</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'CLASSPATH'</span><span class="p">,</span> <span class="s1">''</span><span class="p">):</span>
- <span class="k">return</span>
-
- <span class="k">if</span> <span class="s1">'HADOOP_HOME'</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">!=</span> <span class="s1">'win32'</span><span class="p">:</span>
- <span class="n">classpath</span> <span class="o">=</span> <span class="n">_derive_hadoop_classpath</span><span class="p">()</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">hadoop_bin</span> <span class="o">=</span> <span class="s1">'</span><span class="si">{0}</span><span class="s1">/bin/hadoop'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'HADOOP_HOME'</span><span class="p">])</span>
- <span class="n">classpath</span> <span class="o">=</span> <span class="n">_hadoop_classpath_glob</span><span class="p">(</span><span class="n">hadoop_bin</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">classpath</span> <span class="o">=</span> <span class="n">_hadoop_classpath_glob</span><span class="p">(</span><span class="s1">'hadoop'</span><span class="p">)</span>
-
- <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'CLASSPATH'</span><span class="p">]</span> <span class="o">=</span> <span class="n">classpath</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_derive_hadoop_classpath</span><span class="p">():</span>
- <span class="kn">import</span> <span class="nn">subprocess</span>
-
- <span class="n">find_args</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'find'</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s1">'HADOOP_HOME'</span><span class="p">],</span> <span class="s1">'-name'</span><span class="p">,</span> <span class="s1">'*.jar'</span><span class="p">)</span>
- <span class="n">find</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">find_args</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
- <span class="n">xargs_echo</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">((</span><span class="s1">'xargs'</span><span class="p">,</span> <span class="s1">'echo'</span><span class="p">),</span>
- <span class="n">stdin</span><span class="o">=</span><span class="n">find</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span>
- <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">((</span><span class="s1">'tr'</span><span class="p">,</span> <span class="s2">"' '"</span><span class="p">,</span> <span class="s2">"':'"</span><span class="p">),</span>
- <span class="n">stdin</span><span class="o">=</span><span class="n">xargs_echo</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_hadoop_classpath_glob</span><span class="p">(</span><span class="n">hadoop_bin</span><span class="p">):</span>
- <span class="kn">import</span> <span class="nn">subprocess</span>
-
- <span class="n">hadoop_classpath_args</span> <span class="o">=</span> <span class="p">(</span><span class="n">hadoop_bin</span><span class="p">,</span> <span class="s1">'classpath'</span><span class="p">,</span> <span class="s1">'--glob'</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">hadoop_classpath_args</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_libhdfs_walk_files_dirs</span><span class="p">(</span><span class="n">top_path</span><span class="p">,</span> <span class="n">contents</span><span class="p">):</span>
- <span class="n">files</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="n">directories</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">contents</span><span class="p">:</span>
- <span class="n">scrubbed_name</span> <span class="o">=</span> <span class="n">posixpath</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">c</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])[</span><span class="mi">1</span><span class="p">]</span>
- <span class="k">if</span> <span class="n">c</span><span class="p">[</span><span class="s1">'kind'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'file'</span><span class="p">:</span>
- <span class="n">files</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">scrubbed_name</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">directories</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">scrubbed_name</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">directories</span><span class="p">,</span> <span class="n">files</span>
-
-
-<div class="viewcode-block" id="connect"><a class="viewcode-back" href="../../python/generated/pyarrow.hdfs.connect.html#pyarrow.connect">[docs]</a><span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">user</span><span class="o" [...]
- <span class="n">driver</span><span class="o">=</span><span class="s1">'libhdfs'</span><span class="p">,</span> <span class="n">extra_conf</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Connect to an HDFS cluster. All parameters are optional and should</span>
-<span class="sd"> only be set if the defaults need to be overridden.</span>
-
-<span class="sd"> Authentication should be automatic if the HDFS cluster uses Kerberos.</span>
-<span class="sd"> However, if a username is specified, then the ticket cache will likely</span>
-<span class="sd"> be required.</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> host : NameNode. Set to "default" for fs.defaultFS from core-site.xml.</span>
-<span class="sd"> port : NameNode's port. Set to 0 for default or logical (HA) nodes.</span>
-<span class="sd"> user : Username when connecting to HDFS; None implies login user.</span>
-<span class="sd"> kerb_ticket : Path to Kerberos ticket cache.</span>
-<span class="sd"> driver : {'libhdfs', 'libhdfs3'}, default 'libhdfs'</span>
-<span class="sd"> Connect using libhdfs (JNI-based) or libhdfs3 (3rd-party C++</span>
-<span class="sd"> library from Apache HAWQ (incubating) )</span>
-<span class="sd"> extra_conf : dict, default None</span>
-<span class="sd"> extra Key/Value pairs for config; Will override any</span>
-<span class="sd"> hdfs-site.xml properties</span>
-
-<span class="sd"> Notes</span>
-<span class="sd"> -----</span>
-<span class="sd"> The first time you call this method, it will take longer than usual due</span>
-<span class="sd"> to JNI spin-up time.</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> filesystem : HadoopFileSystem</span>
-<span class="sd"> """</span>
- <span class="n">fs</span> <span class="o">=</span> <span class="n">HadoopFileSystem</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">port</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="n">user</span><span class="p">,</span>
- <span class="n">kerb_ticket</span><span class="o">=</span><span class="n">kerb_ticket</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="n">driver</span><span class="p">,</span>
- <span class="n">extra_conf</span><span class="o">=</span><span class="n">extra_conf</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">fs</span></div>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/ipc.html b/docs/latest/_modules/pyarrow/ipc.html
deleted file mode 100644
index 5ec980b..0000000
--- a/docs/latest/_modules/pyarrow/ipc.html
+++ /dev/null
@@ -1,408 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.ipc — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.ipc</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.ipc</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="c1"># Arrow file and stream reader/writer classes, and other messaging tools</span>
-
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">Message</span><span class="p">,</span> <span class="n">MessageReader</span><span class="p">,</span> <span class="c1"># noqa</span>
- <span class="n">read_message</span><span class="p">,</span> <span class="n">read_record_batch</span><span class="p">,</span> <span class="n">read_schema</span><span class="p">,</span>
- <span class="n">read_tensor</span><span class="p">,</span> <span class="n">write_tensor</span><span class="p">,</span>
- <span class="n">get_record_batch_size</span><span class="p">,</span> <span class="n">get_tensor_size</span><span class="p">)</span>
-<span class="kn">import</span> <span class="nn">pyarrow.lib</span> <span class="k">as</span> <span class="nn">lib</span>
-
-
-<span class="k">class</span> <span class="nc">_ReadPandasOption</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
-
- <span class="k">def</span> <span class="nf">read_pandas</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read contents of stream and convert to pandas.DataFrame using</span>
-<span class="sd"> Table.to_pandas</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> **options : arguments to forward to Table.to_pandas</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> df : pandas.DataFrame</span>
-<span class="sd"> """</span>
- <span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">read_all</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">table</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">(</span><span class="o">**</span><span class="n">options</span><span class="p">)</span>
-
-
-<div class="viewcode-block" id="RecordBatchStreamReader"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchStreamReader.html#pyarrow.RecordBatchStreamReader">[docs]</a><span class="k">class</span> <span class="nc">RecordBatchStreamReader</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">_RecordBatchReader</span><span class="p">,</span> <span class="n">_ReadPandasOption</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Reader for the Arrow streaming binary format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either an in-memory buffer, or a readable file object</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="RecordBatchStreamReader.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchStreamReader.html#pyarrow.RecordBatchStreamReader.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">source</span><span class="p">)</span></div></div>
-
-
-<div class="viewcode-block" id="RecordBatchStreamWriter"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchStreamWriter.html#pyarrow.RecordBatchStreamWriter">[docs]</a><span class="k">class</span> <span class="nc">RecordBatchStreamWriter</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">_RecordBatchWriter</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Writer for the Arrow streaming binary format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> sink : str, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either a file path, or a writable file object</span>
-<span class="sd"> schema : pyarrow.Schema</span>
-<span class="sd"> The Arrow schema for data to be written to the file</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="RecordBatchStreamWriter.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchStreamWriter.html#pyarrow.RecordBatchStreamWriter.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">)</span></div></div>
-
-
-<div class="viewcode-block" id="RecordBatchFileReader"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchFileReader.html#pyarrow.RecordBatchFileReader">[docs]</a><span class="k">class</span> <span class="nc">RecordBatchFileReader</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">_RecordBatchFileReader</span><span class="p">,</span> <span class="n">_ReadPandasOption</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Class for reading Arrow record batch data from the Arrow binary file format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either an in-memory buffer, or a readable file object</span>
-<span class="sd"> footer_offset : int, default None</span>
-<span class="sd"> If the file is embedded in some larger file, this is the byte offset to</span>
-<span class="sd"> the very end of the file data</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="RecordBatchFileReader.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchFileReader.html#pyarrow.RecordBatchFileReader.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">footer_offset</span><span class="o">=</span><span class="kc">None</span><span c [...]
- <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">footer_offset</span><span class="o">=</span><span class="n">footer_offset</span><span class="p">)</span></div></div>
-
-
-<div class="viewcode-block" id="RecordBatchFileWriter"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchFileWriter.html#pyarrow.RecordBatchFileWriter">[docs]</a><span class="k">class</span> <span class="nc">RecordBatchFileWriter</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">_RecordBatchFileWriter</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Writer to create the Arrow binary file format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> sink : str, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either a file path, or a writable file object</span>
-<span class="sd"> schema : pyarrow.Schema</span>
-<span class="sd"> The Arrow schema for data to be written to the file</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="RecordBatchFileWriter.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.RecordBatchFileWriter.html#pyarrow.RecordBatchFileWriter.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">)</span></div></div>
-
-
-<div class="viewcode-block" id="open_stream"><a class="viewcode-back" href="../../python/generated/pyarrow.ipc.open_stream.html#pyarrow.open_stream">[docs]</a><span class="k">def</span> <span class="nf">open_stream</span><span class="p">(</span><span class="n">source</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create reader for Arrow streaming format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either an in-memory buffer, or a readable file object</span>
-<span class="sd"> footer_offset : int, default None</span>
-<span class="sd"> If the file is embedded in some larger file, this is the byte offset to</span>
-<span class="sd"> the very end of the file data</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> reader : RecordBatchStreamReader</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">RecordBatchStreamReader</span><span class="p">(</span><span class="n">source</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="open_file"><a class="viewcode-back" href="../../python/generated/pyarrow.ipc.open_file.html#pyarrow.open_file">[docs]</a><span class="k">def</span> <span class="nf">open_file</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">footer_offset</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create reader for Arrow file format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object</span>
-<span class="sd"> Either an in-memory buffer, or a readable file object</span>
-<span class="sd"> footer_offset : int, default None</span>
-<span class="sd"> If the file is embedded in some larger file, this is the byte offset to</span>
-<span class="sd"> the very end of the file data</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> reader : RecordBatchFileReader</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">RecordBatchFileReader</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">footer_offset</span><span class="o">=</span><span class="n">footer_offset</span><span class="p">)</span></div>
-
-
-<span class="k">def</span> <span class="nf">serialize_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">nthreads</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">preserve_index</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="sd">"""Serialize a pandas DataFrame into a buffer protocol compatible object.</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> df : pandas.DataFrame</span>
-<span class="sd"> nthreads : int, default None</span>
-<span class="sd"> Number of threads to use for conversion to Arrow, default all CPUs</span>
-<span class="sd"> preserve_index : boolean, default True</span>
-<span class="sd"> If True, preserve the pandas index data, otherwise the result will have</span>
-<span class="sd"> a default RangeIndex</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> buf : buffer</span>
-<span class="sd"> An object compatible with the buffer protocol</span>
-<span class="sd"> """</span>
- <span class="n">batch</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">nthreads</span><span class="o">=</span><span class="n">nthreads</span><span class="p">,</span>
- <span class="n">preserve_index</span><span class="o">=</span><span class="n">preserve_index</span><span class="p">)</span>
- <span class="n">sink</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">BufferOutputStream</span><span class="p">()</span>
- <span class="n">writer</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchStreamWriter</span><span class="p">(</span><span class="n">sink</span><span class="p">,</span> <span class="n">batch</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span>
- <span class="n">writer</span><span class="o">.</span><span class="n">write_batch</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span>
- <span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">sink</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span>
-
-
-<span class="k">def</span> <span class="nf">deserialize_pandas</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="sd">"""Deserialize a buffer protocol compatible object into a pandas DataFrame.</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> buf : buffer</span>
-<span class="sd"> An object compatible with the buffer protocol</span>
-<span class="sd"> use_threads: boolean, default True</span>
-<span class="sd"> Whether to parallelize the conversion using multiple threads</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> df : pandas.DataFrame</span>
-<span class="sd"> """</span>
- <span class="n">buffer_reader</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">BufferReader</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchStreamReader</span><span class="p">(</span><span class="n">buffer_reader</span><span class="p">)</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_all</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">table</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">(</span><span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/parquet.html b/docs/latest/_modules/pyarrow/parquet.html
deleted file mode 100644
index 70ad977..0000000
--- a/docs/latest/_modules/pyarrow/parquet.html
+++ /dev/null
@@ -1,1543 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.parquet — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.parquet</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.parquet</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">defaultdict</span>
-<span class="kn">from</span> <span class="nn">concurrent</span> <span class="k">import</span> <span class="n">futures</span>
-
-<span class="kn">from</span> <span class="nn">six.moves.urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
-<span class="kn">import</span> <span class="nn">json</span>
-<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
-<span class="kn">import</span> <span class="nn">os</span>
-<span class="kn">import</span> <span class="nn">re</span>
-<span class="kn">import</span> <span class="nn">six</span>
-
-<span class="kn">import</span> <span class="nn">pyarrow</span> <span class="k">as</span> <span class="nn">pa</span>
-<span class="kn">import</span> <span class="nn">pyarrow.lib</span> <span class="k">as</span> <span class="nn">lib</span>
-<span class="kn">import</span> <span class="nn">pyarrow._parquet</span> <span class="k">as</span> <span class="nn">_parquet</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow._parquet</span> <span class="k">import</span> <span class="p">(</span><span class="n">ParquetReader</span><span class="p">,</span> <span class="n">RowGroupStatistics</span><span class="p">,</span> <span class="c1"># noqa</span>
- <span class="n">FileMetaData</span><span class="p">,</span> <span class="n">RowGroupMetaData</span><span class="p">,</span>
- <span class="n">ColumnChunkMetaData</span><span class="p">,</span>
- <span class="n">ParquetSchema</span><span class="p">,</span> <span class="n">ColumnSchema</span><span class="p">)</span>
-<span class="kn">from</span> <span class="nn">pyarrow.compat</span> <span class="k">import</span> <span class="n">guid</span>
-<span class="kn">from</span> <span class="nn">pyarrow.filesystem</span> <span class="k">import</span> <span class="p">(</span><span class="n">LocalFileSystem</span><span class="p">,</span> <span class="n">_ensure_filesystem</span><span class="p">,</span>
- <span class="n">get_filesystem_from_uri</span><span class="p">)</span>
-<span class="kn">from</span> <span class="nn">pyarrow.util</span> <span class="k">import</span> <span class="n">_is_path_like</span><span class="p">,</span> <span class="n">_stringify_path</span>
-
-<span class="n">_URI_STRIP_SCHEMES</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'hdfs'</span><span class="p">,)</span>
-
-
-<span class="k">def</span> <span class="nf">_parse_uri</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="n">parsed_uri</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">parsed_uri</span><span class="o">.</span><span class="n">scheme</span> <span class="ow">in</span> <span class="n">_URI_STRIP_SCHEMES</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">parsed_uri</span><span class="o">.</span><span class="n">path</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="c1"># ARROW-4073: On Windows returning the path with the scheme</span>
- <span class="c1"># stripped removes the drive letter, if any</span>
- <span class="k">return</span> <span class="n">path</span>
-
-
-<span class="k">def</span> <span class="nf">_get_filesystem_and_path</span><span class="p">(</span><span class="n">passed_filesystem</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">passed_filesystem</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">get_filesystem_from_uri</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">passed_filesystem</span> <span class="o">=</span> <span class="n">_ensure_filesystem</span><span class="p">(</span><span class="n">passed_filesystem</span><span class="p">)</span>
- <span class="n">parsed_path</span> <span class="o">=</span> <span class="n">_parse_uri</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">passed_filesystem</span><span class="p">,</span> <span class="n">parsed_path</span>
-
-
-<span class="k">def</span> <span class="nf">_check_contains_null</span><span class="p">(</span><span class="n">val</span><span class="p">):</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">binary_type</span><span class="p">):</span>
- <span class="k">for</span> <span class="n">byte</span> <span class="ow">in</span> <span class="n">val</span><span class="p">:</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">byte</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">binary_type</span><span class="p">):</span>
- <span class="n">compare_to</span> <span class="o">=</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">compare_to</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">if</span> <span class="n">byte</span> <span class="o">==</span> <span class="n">compare_to</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">True</span>
- <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">text_type</span><span class="p">):</span>
- <span class="k">return</span> <span class="sa">u</span><span class="s1">'</span><span class="se">\x00</span><span class="s1">'</span> <span class="ow">in</span> <span class="n">val</span>
- <span class="k">return</span> <span class="kc">False</span>
-
-
-<span class="k">def</span> <span class="nf">_check_filters</span><span class="p">(</span><span class="n">filters</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Check if filters are well-formed.</span>
-<span class="sd"> """</span>
- <span class="k">if</span> <span class="n">filters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">filters</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">or</span> <span class="nb">any</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">f</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">i [...]
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Malformed filters"</span><span class="p">)</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">filters</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="n">six</span><span class="o">.</span><span class="n">string_types</span><span class="p">):</span>
- <span class="c1"># We have encountered the situation where we have one nesting level</span>
- <span class="c1"># too few:</span>
- <span class="c1"># We have [(,,), ..] instead of [[(,,), ..]]</span>
- <span class="n">filters</span> <span class="o">=</span> <span class="p">[</span><span class="n">filters</span><span class="p">]</span>
- <span class="k">for</span> <span class="n">conjunction</span> <span class="ow">in</span> <span class="n">filters</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">col</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">val</span> <span class="ow">in</span> <span class="n">conjunction</span><span class="p">:</span>
- <span class="k">if</span> <span class="p">(</span>
- <span class="nb">isinstance</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
- <span class="ow">and</span> <span class="nb">all</span><span class="p">(</span><span class="n">_check_contains_null</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">val</span><span class="p">)</span>
- <span class="ow">or</span> <span class="n">_check_contains_null</span><span class="p">(</span><span class="n">val</span><span class="p">)</span>
- <span class="p">):</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span>
- <span class="s2">"Null-terminated binary strings are not supported as"</span>
- <span class="s2">" filter values."</span>
- <span class="p">)</span>
- <span class="k">return</span> <span class="n">filters</span>
-
-<span class="c1"># ----------------------------------------------------------------------</span>
-<span class="c1"># Reading a single Parquet file</span>
-
-
-<div class="viewcode-block" id="ParquetFile"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetFile.html#pyarrow.parquet.ParquetFile">[docs]</a><span class="k">class</span> <span class="nc">ParquetFile</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Reader interface for a single Parquet file</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> source : str, pathlib.Path, pyarrow.NativeFile, or file-like object</span>
-<span class="sd"> Readable source. For passing bytes or buffer-like file containing a</span>
-<span class="sd"> Parquet file, use pyarorw.BufferReader</span>
-<span class="sd"> metadata : ParquetFileMetadata, default None</span>
-<span class="sd"> Use existing metadata object, rather than reading from file.</span>
-<span class="sd"> common_metadata : ParquetFileMetadata, default None</span>
-<span class="sd"> Will be used in reads for pandas schema metadata if not found in the</span>
-<span class="sd"> main file's metadata, no other uses at the moment</span>
-<span class="sd"> memory_map : boolean, default True</span>
-<span class="sd"> If the source is a file path, use a memory map to read file, which can</span>
-<span class="sd"> improve performance in some environments</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="ParquetFile.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetFile.html#pyarrow.parquet.ParquetFile.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> < [...]
- <span class="n">memory_map</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetReader</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">use_memory_map</span><span class="o">=</span><span class="n">memory_map</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">metadata</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="o">=</span> <span class="n">common_metadata</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_nested_paths_by_prefix</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_build_nested_paths</span><span class="p">()</span></div>
-
- <span class="k">def</span> <span class="nf">_build_nested_paths</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="n">paths</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">column_paths</span>
-
- <span class="n">result</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">_visit_piece</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">rest</span><span class="p">):</span>
- <span class="n">result</span><span class="p">[</span><span class="n">key</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">rest</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">nested_key</span> <span class="o">=</span> <span class="s1">'.'</span><span class="o">.</span><span class="n">join</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">rest</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span>
- <span class="n">_visit_piece</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">nested_key</span><span class="p">,</span> <span class="n">rest</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
-
- <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">path</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">paths</span><span class="p">):</span>
- <span class="n">_visit_piece</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">path</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">path</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
-
- <span class="k">return</span> <span class="n">result</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">metadata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">metadata</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">schema</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">schema</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">num_row_groups</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">num_row_groups</span>
-
-<div class="viewcode-block" id="ParquetFile.read_row_group"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetFile.html#pyarrow.parquet.ParquetFile.read_row_group">[docs]</a> <span class="k">def</span> <span class="nf">read_row_group</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p [...]
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read a single row group from a Parquet file</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns: list</span>
-<span class="sd"> If not None, only these columns will be read from the row group. A</span>
-<span class="sd"> column name may be a prefix of a nested field, e.g. 'a' will select</span>
-<span class="sd"> 'a.b', 'a.c', and 'a.d.e'</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Perform multi-threaded column reads</span>
-<span class="sd"> use_pandas_metadata : boolean, default False</span>
-<span class="sd"> If True and file has custom pandas schema metadata, ensure that</span>
-<span class="sd"> index columns are also loaded</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pyarrow.table.Table</span>
-<span class="sd"> Content of the row group as a table (of columns)</span>
-<span class="sd"> """</span>
- <span class="n">column_indices</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_column_indices</span><span class="p">(</span>
- <span class="n">columns</span><span class="p">,</span> <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">read_row_group</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">column_indices</span><span class="o">=</span><span class="n">column_indices</span><span class="p">,</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="ParquetFile.read"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetFile.html#pyarrow.parquet.ParquetFile.read">[docs]</a> <span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span clas [...]
- <span class="sd">"""</span>
-<span class="sd"> Read a Table from Parquet format</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns: list</span>
-<span class="sd"> If not None, only these columns will be read from the file. A</span>
-<span class="sd"> column name may be a prefix of a nested field, e.g. 'a' will select</span>
-<span class="sd"> 'a.b', 'a.c', and 'a.d.e'</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Perform multi-threaded column reads</span>
-<span class="sd"> use_pandas_metadata : boolean, default False</span>
-<span class="sd"> If True and file has custom pandas schema metadata, ensure that</span>
-<span class="sd"> index columns are also loaded</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pyarrow.table.Table</span>
-<span class="sd"> Content of the file as a table (of columns)</span>
-<span class="sd"> """</span>
- <span class="n">column_indices</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_column_indices</span><span class="p">(</span>
- <span class="n">columns</span><span class="p">,</span> <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">read_all</span><span class="p">(</span><span class="n">column_indices</span><span class="o">=</span><span class="n">column_indices</span><span class="p">,</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="ParquetFile.scan_contents"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetFile.html#pyarrow.parquet.ParquetFile.scan_contents">[docs]</a> <span class="k">def</span> <span class="nf">scan_contents</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">batch_size</span><span cl [...]
- <span class="sd">"""</span>
-<span class="sd"> Read contents of file with a single thread for indicated columns and</span>
-<span class="sd"> batch size. Number of rows in file is returned. This function is used</span>
-<span class="sd"> for benchmarking</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns : list of integers, default None</span>
-<span class="sd"> If None, scan all columns</span>
-<span class="sd"> batch_size : int, default 64K</span>
-<span class="sd"> Number of rows to read at a time internally</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> num_rows : number of rows in file</span>
-<span class="sd"> """</span>
- <span class="n">column_indices</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_column_indices</span><span class="p">(</span><span class="n">columns</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">scan_contents</span><span class="p">(</span><span class="n">column_indices</span><span class="p">,</span>
- <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">)</span></div>
-
- <span class="k">def</span> <span class="nf">_get_column_indices</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">column_names</span><span class="p">,</span> <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">column_names</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">None</span>
-
- <span class="n">indices</span> <span class="o">=</span> <span class="p">[]</span>
-
- <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">column_names</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_nested_paths_by_prefix</span><span class="p">:</span>
- <span class="n">indices</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_nested_paths_by_prefix</span><span class="p">[</span><span class="n">name</span><span class="p">])</span>
-
- <span class="k">if</span> <span class="n">use_pandas_metadata</span><span class="p">:</span>
- <span class="n">file_keyvalues</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">metadata</span>
- <span class="n">common_keyvalues</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span><span class="o">.</span><span class="n">metadata</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
- <span class="k">else</span> <span class="kc">None</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">file_keyvalues</span> <span class="ow">and</span> <span class="sa">b</span><span class="s1">'pandas'</span> <span class="ow">in</span> <span class="n">file_keyvalues</span><span class="p">:</span>
- <span class="n">index_columns</span> <span class="o">=</span> <span class="n">_get_pandas_index_columns</span><span class="p">(</span><span class="n">file_keyvalues</span><span class="p">)</span>
- <span class="k">elif</span> <span class="n">common_keyvalues</span> <span class="ow">and</span> <span class="sa">b</span><span class="s1">'pandas'</span> <span class="ow">in</span> <span class="n">common_keyvalues</span><span class="p">:</span>
- <span class="n">index_columns</span> <span class="o">=</span> <span class="n">_get_pandas_index_columns</span><span class="p">(</span><span class="n">common_keyvalues</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">index_columns</span> <span class="o">=</span> <span class="p">[]</span>
-
- <span class="k">if</span> <span class="n">indices</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">index_columns</span><span class="p">:</span>
- <span class="n">indices</span> <span class="o">+=</span> <span class="nb">map</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">reader</span><span class="o">.</span><span class="n">column_name_idx</span><span class="p">,</span> <span class="n">index_columns</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">indices</span></div>
-
-
-<span class="n">_SPARK_DISALLOWED_CHARS</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s1">'[ ,;</span><span class="si">{}</span><span class="s1">()</span><span class="se">\n\t</span><span class="s1">=]'</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_sanitized_spark_field_name</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">_SPARK_DISALLOWED_CHARS</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s1">'_'</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_sanitize_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">flavor</span><span class="p">):</span>
- <span class="k">if</span> <span class="s1">'spark'</span> <span class="ow">in</span> <span class="n">flavor</span><span class="p">:</span>
- <span class="n">sanitized_fields</span> <span class="o">=</span> <span class="p">[]</span>
-
- <span class="n">schema_changed</span> <span class="o">=</span> <span class="kc">False</span>
-
- <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">schema</span><span class="p">:</span>
- <span class="n">name</span> <span class="o">=</span> <span class="n">field</span><span class="o">.</span><span class="n">name</span>
- <span class="n">sanitized_name</span> <span class="o">=</span> <span class="n">_sanitized_spark_field_name</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">sanitized_name</span> <span class="o">!=</span> <span class="n">name</span><span class="p">:</span>
- <span class="n">schema_changed</span> <span class="o">=</span> <span class="kc">True</span>
- <span class="n">sanitized_field</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="n">sanitized_name</span><span class="p">,</span> <span class="n">field</span><span class="o">.</span><span class="n">type</span><span class="p">,</span>
- <span class="n">field</span><span class="o">.</span><span class="n">nullable</span><span class="p">,</span> <span class="n">field</span><span class="o">.</span><span class="n">metadata</span><span class="p">)</span>
- <span class="n">sanitized_fields</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">sanitized_field</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">sanitized_fields</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">field</span><span class="p">)</span>
-
- <span class="n">new_schema</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">schema</span><span class="p">(</span><span class="n">sanitized_fields</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">schema</span><span class="o">.</span><span class="n">metadata</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">new_schema</span><span class="p">,</span> <span class="n">schema_changed</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">schema</span><span class="p">,</span> <span class="kc">False</span>
-
-
-<span class="k">def</span> <span class="nf">_sanitize_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">new_schema</span><span class="p">,</span> <span class="n">flavor</span><span class="p">):</span>
- <span class="c1"># TODO: This will not handle prohibited characters in nested field names</span>
- <span class="k">if</span> <span class="s1">'spark'</span> <span class="ow">in</span> <span class="n">flavor</span><span class="p">:</span>
- <span class="n">column_data</span> <span class="o">=</span> <span class="p">[</span><span class="n">table</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">data</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">num_columns</span><span class="p">)]</span>
- <span class="k">return</span> <span class="n">pa</span><span class="o">.</span><span class="n">Table</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span><span class="n">column_data</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">new_schema</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">table</span>
-
-
-<span class="n">_parquet_writer_arg_docs</span> <span class="o">=</span> <span class="s2">"""version : {"1.0", "2.0"}, default "1.0"</span>
-<span class="s2"> The Parquet format version, defaults to 1.0</span>
-<span class="s2">use_dictionary : bool or list</span>
-<span class="s2"> Specify if we should use dictionary encoding in general or only for</span>
-<span class="s2"> some columns.</span>
-<span class="s2">use_deprecated_int96_timestamps : boolean, default None</span>
-<span class="s2"> Write timestamps to INT96 Parquet format. Defaults to False unless enabled</span>
-<span class="s2"> by flavor argument. This take priority over the coerce_timestamps option.</span>
-<span class="s2">coerce_timestamps : string, default None</span>
-<span class="s2"> Cast timestamps a particular resolution.</span>
-<span class="s2"> Valid values: {None, 'ms', 'us'}</span>
-<span class="s2">allow_truncated_timestamps : boolean, default False</span>
-<span class="s2"> Allow loss of data when coercing timestamps to a particular</span>
-<span class="s2"> resolution. E.g. if microsecond or nanosecond data is lost when coercing to</span>
-<span class="s2"> 'ms', do not raise an exception</span>
-<span class="s2">compression : str or dict</span>
-<span class="s2"> Specify the compression codec, either on a general basis or per-column.</span>
-<span class="s2"> Valid values: {'NONE', 'SNAPPY', 'GZIP', 'LZO', 'BROTLI', 'LZ4', 'ZSTD'}</span>
-<span class="s2">flavor : {'spark'}, default None</span>
-<span class="s2"> Sanitize schema or set other compatibility options for compatibility"""</span>
-
-
-<div class="viewcode-block" id="ParquetWriter"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow.parquet.ParquetWriter">[docs]</a><span class="k">class</span> <span class="nc">ParquetWriter</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
-
- <span class="vm">__doc__</span> <span class="o">=</span> <span class="s2">"""</span>
-<span class="s2">Class for incrementally building a Parquet file for Arrow tables</span>
-
-<span class="s2">Parameters</span>
-<span class="s2">----------</span>
-<span class="s2">where : path or file-like object</span>
-<span class="s2">schema : arrow Schema</span>
-<span class="si">{0}</span><span class="s2"></span>
-<span class="s2">"""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">_parquet_writer_arg_docs</span><span class="p">)</span>
-
-<div class="viewcode-block" id="ParquetWriter.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow.parquet.ParquetWriter.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">flavor</span><span class="o">=</sp [...]
- <span class="n">version</span><span class="o">=</span><span class="s1">'1.0'</span><span class="p">,</span>
- <span class="n">use_dictionary</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
- <span class="n">compression</span><span class="o">=</span><span class="s1">'snappy'</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">use_deprecated_int96_timestamps</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="c1"># Use int96 timestamps for Spark</span>
- <span class="k">if</span> <span class="n">flavor</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="s1">'spark'</span> <span class="ow">in</span> <span class="n">flavor</span><span class="p">:</span>
- <span class="n">use_deprecated_int96_timestamps</span> <span class="o">=</span> <span class="kc">True</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">use_deprecated_int96_timestamps</span> <span class="o">=</span> <span class="kc">False</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">flavor</span> <span class="o">=</span> <span class="n">flavor</span>
- <span class="k">if</span> <span class="n">flavor</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema_changed</span> <span class="o">=</span> <span class="n">_sanitize_schema</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">flavor</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">schema_changed</span> <span class="o">=</span> <span class="kc">False</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">where</span> <span class="o">=</span> <span class="n">where</span>
-
- <span class="c1"># If we open a file using an implied filesystem, so it can be assured</span>
- <span class="c1"># to be closed</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">file_handle</span> <span class="o">=</span> <span class="kc">None</span>
-
- <span class="k">if</span> <span class="n">_is_path_like</span><span class="p">(</span><span class="n">where</span><span class="p">):</span>
- <span class="n">fs</span><span class="p">,</span> <span class="n">path</span> <span class="o">=</span> <span class="n">_get_filesystem_and_path</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">where</span><span class="p">)</span>
- <span class="n">sink</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_handle</span> <span class="o">=</span> <span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">sink</span> <span class="o">=</span> <span class="n">where</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span> <span class="o">=</span> <span class="n">_parquet</span><span class="o">.</span><span class="n">ParquetWriter</span><span class="p">(</span>
- <span class="n">sink</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span>
- <span class="n">version</span><span class="o">=</span><span class="n">version</span><span class="p">,</span>
- <span class="n">compression</span><span class="o">=</span><span class="n">compression</span><span class="p">,</span>
- <span class="n">use_dictionary</span><span class="o">=</span><span class="n">use_dictionary</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="n">use_deprecated_int96_timestamps</span><span class="p">,</span>
- <span class="o">**</span><span class="n">options</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span> <span class="o">=</span> <span class="kc">True</span></div>
-
- <span class="k">def</span> <span class="nf">__del__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'is_open'</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span>
-
- <span class="k">def</span> <span class="nf">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="c1"># return false since we want to propagate exceptions</span>
- <span class="k">return</span> <span class="kc">False</span>
-
-<div class="viewcode-block" id="ParquetWriter.write_table"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow.parquet.ParquetWriter.write_table">[docs]</a> <span class="k">def</span> <span class="nf">write_table</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">row_group_size</span><span class="o">=</span><span class="kc">None</span><span [...]
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema_changed</span><span class="p">:</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">_sanitize_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">flavor</span><span class="p">)</span>
- <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span>
-
- <span class="k">if</span> <span class="ow">not</span> <span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">equals</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="n">check_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="n">msg</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'Table schema does not match schema used to create file: '</span>
- <span class="s1">'</span><span class="se">\n</span><span class="s1">table:</span><span class="se">\n</span><span class="si">{0!s}</span><span class="s1"> vs. </span><span class="se">\n</span><span class="s1">file:</span><span class="se">\n</span><span class="si">{1!s}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">schema</span><span c [...]
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">))</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">row_group_size</span><span class="o">=</span><span class="n">row_group_size</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="ParquetWriter.close"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow.parquet.ParquetWriter.close">[docs]</a> <span class="k">def</span> <span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span> <span class="o">=</span> <span class="kc">False</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_handle</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">file_handle</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div></div>
-
-
-<span class="k">def</span> <span class="nf">_get_pandas_index_columns</span><span class="p">(</span><span class="n">keyvalues</span><span class="p">):</span>
- <span class="k">return</span> <span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">keyvalues</span><span class="p">[</span><span class="sa">b</span><span class="s1">'pandas'</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf8'</span><span class="p">))</span>
- <span class="p">[</span><span class="s1">'index_columns'</span><span class="p">])</span>
-
-
-<span class="c1"># ----------------------------------------------------------------------</span>
-<span class="c1"># Metadata container providing instructions about reading a single Parquet</span>
-<span class="c1"># file, possibly part of a partitioned dataset</span>
-
-
-<span class="k">class</span> <span class="nc">ParquetDatasetPiece</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> A single chunk of a potentially larger Parquet dataset to read. The</span>
-<span class="sd"> arguments will indicate to read either a single row group or all row</span>
-<span class="sd"> groups, and whether to add partition keys to the resulting pyarrow.Table</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path : str or pathlib.Path</span>
-<span class="sd"> Path to file in the file system where this piece is located</span>
-<span class="sd"> partition_keys : list of tuples</span>
-<span class="sd"> [(column name, ordinal index)]</span>
-<span class="sd"> row_group : int, default None</span>
-<span class="sd"> Row group to load. By default, reads all row groups</span>
-<span class="sd"> """</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">row_group</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">partition_keys</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">path</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span> <span class="o">=</span> <span class="n">row_group</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span> <span class="o">=</span> <span class="n">partition_keys</span> <span class="ow">or</span> <span class="p">[]</span>
-
- <span class="k">def</span> <span class="nf">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">other</span><span class="p">,</span> <span class="n">ParquetDatasetPiece</span><span class="p">):</span>
- <span class="k">return</span> <span class="kc">False</span>
- <span class="k">return</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">path</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">path</span> <span class="ow">and</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">row_group</span> <span class="ow">and</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">__ne__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
- <span class="k">return</span> <span class="ow">not</span> <span class="p">(</span><span class="bp">self</span> <span class="o">==</span> <span class="n">other</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="p">(</span><span class="s1">'</span><span class="si">{0}</span><span class="s1">(</span><span class="si">{1!r}</span><span class="s1">, row_group=</span><span class="si">{2!r}</span><span class="s1">, partition_keys=</span><span class="si">{3!r}</span><span class="s1">)'</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="n">result</span> <span class="o">=</span> <span class="s1">''</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">partition_str</span> <span class="o">=</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">'</span><span class="si">{0}</span><span class="s1">=</span><span class="si">{1}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">index</span><span class=" [...]
- <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">)</span>
- <span class="n">result</span> <span class="o">+=</span> <span class="s1">'partition[</span><span class="si">{0}</span><span class="s1">] '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">partition_str</span><span class="p">)</span>
-
- <span class="n">result</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">path</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">result</span> <span class="o">+=</span> <span class="s1">' | row_group=</span><span class="si">{0}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">row_group</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">result</span>
-
- <span class="k">def</span> <span class="nf">get_metadata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">open_file_func</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Given a function that can create an open ParquetFile object, return the</span>
-<span class="sd"> file's metadata</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">open_file_func</span><span class="p">)</span><span class="o">.</span><span class="n">metadata</span>
-
- <span class="k">def</span> <span class="nf">_open</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">open_file_func</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Returns instance of ParquetFile</span>
-<span class="sd"> """</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">open_file_func</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">reader</span><span class="p">,</span> <span class="n">ParquetFile</span><span class="p">):</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">reader</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">reader</span>
-
- <span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">partitions</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">open_file_func</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read this piece as a pyarrow.Table</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns : list of column names, default None</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Perform multi-threaded column reads</span>
-<span class="sd"> partitions : ParquetPartitions, default None</span>
-<span class="sd"> open_file_func : function, default None</span>
-<span class="sd"> A function that knows how to construct a ParquetFile object given</span>
-<span class="sd"> the file path in this piece</span>
-<span class="sd"> file : file-like object</span>
-<span class="sd"> passed to ParquetFile</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> table : pyarrow.Table</span>
-<span class="sd"> """</span>
- <span class="k">if</span> <span class="n">open_file_func</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_open</span><span class="p">(</span><span class="n">open_file_func</span><span class="p">)</span>
- <span class="k">elif</span> <span class="n">file</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">file</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="c1"># try to read the local path</span>
- <span class="n">reader</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
-
- <span class="n">options</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">row_group</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">read_row_group</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">row_group</span><span class="p">,</span> <span class="o">**</span><span class="n">options</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">reader</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="o">**</span><span class="n">options</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">partitions</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Must pass partition sets'</span><span class="p">)</span>
-
- <span class="c1"># Here, the index is the categorical code of the partition where</span>
- <span class="c1"># this piece is located. Suppose we had</span>
- <span class="c1">#</span>
- <span class="c1"># /foo=a/0.parq</span>
- <span class="c1"># /foo=b/0.parq</span>
- <span class="c1"># /foo=c/0.parq</span>
- <span class="c1">#</span>
- <span class="c1"># Then we assign a=0, b=1, c=2. And the resulting Table pieces will</span>
- <span class="c1"># have a DictionaryArray column named foo having the constant index</span>
- <span class="c1"># value as indicated. The distinct categories of the partition have</span>
- <span class="c1"># been computed in the ParquetManifest</span>
- <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">):</span>
- <span class="c1"># The partition code is the same for all values in this piece</span>
- <span class="n">indices</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="n">index</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">'i4'</span><span class="p">)</span><span class="o">.</span><span class="n">repeat</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">table</s [...]
-
- <span class="c1"># This is set of all partition values, computed as part of the</span>
- <span class="c1"># manifest, so ['a', 'b', 'c'] as in our example above.</span>
- <span class="n">dictionary</span> <span class="o">=</span> <span class="n">partitions</span><span class="o">.</span><span class="n">levels</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">dictionary</span>
-
- <span class="n">arr</span> <span class="o">=</span> <span class="n">lib</span><span class="o">.</span><span class="n">DictionaryArray</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span><span class="n">indices</span><span class="p">,</span> <span class="n">dictionary</span><span class="p">)</span>
- <span class="n">col</span> <span class="o">=</span> <span class="n">lib</span><span class="o">.</span><span class="n">Column</span><span class="o">.</span><span class="n">from_array</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">arr</span><span class="p">)</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">append_column</span><span class="p">(</span><span class="n">col</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">table</span>
-
-
-<span class="k">class</span> <span class="nc">PartitionSet</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""A data structure for cataloguing the observed Parquet partitions at a</span>
-<span class="sd"> particular level. So if we have</span>
-
-<span class="sd"> /foo=a/bar=0</span>
-<span class="sd"> /foo=a/bar=1</span>
-<span class="sd"> /foo=a/bar=2</span>
-<span class="sd"> /foo=b/bar=0</span>
-<span class="sd"> /foo=b/bar=1</span>
-<span class="sd"> /foo=b/bar=2</span>
-
-<span class="sd"> Then we have two partition sets, one for foo, another for bar. As we visit</span>
-<span class="sd"> levels of the partition hierarchy, a PartitionSet tracks the distinct</span>
-<span class="sd"> values and assigns categorical codes to use when reading the pieces</span>
-<span class="sd"> """</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">keys</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">keys</span> <span class="o">=</span> <span class="n">keys</span> <span class="ow">or</span> <span class="p">[]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">key_indices</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">i</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span clas [...]
- <span class="bp">self</span><span class="o">.</span><span class="n">_dictionary</span> <span class="o">=</span> <span class="kc">None</span>
-
- <span class="k">def</span> <span class="nf">get_index</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Get the index of the partition value if it is known, otherwise assign</span>
-<span class="sd"> one</span>
-<span class="sd"> """</span>
- <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">key_indices</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">key_indices</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">index</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">key_indices</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">key_indices</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">index</span>
- <span class="k">return</span> <span class="n">index</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">dictionary</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_dictionary</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_dictionary</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'No known partition keys'</span><span class="p">)</span>
-
- <span class="c1"># Only integer and string partition types are supported right now</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">integer_keys</span> <span class="o">=</span> <span class="p">[</span><span class="nb">int</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="p">]</span>
- <span class="n">dictionary</span> <span class="o">=</span> <span class="n">lib</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">integer_keys</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
- <span class="n">dictionary</span> <span class="o">=</span> <span class="n">lib</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="p">)</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">_dictionary</span> <span class="o">=</span> <span class="n">dictionary</span>
- <span class="k">return</span> <span class="n">dictionary</span>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">is_sorted</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="p">)</span> <span class="o">==</span> <span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">keys</span><span class="p">)</span>
-
-
-<span class="k">class</span> <span class="nc">ParquetPartitions</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
-
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_names</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
-
- <span class="k">def</span> <span class="nf">get_index</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">level</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Record a partition value at a particular level, returning the distinct</span>
-<span class="sd"> code for that value at that level. Example:</span>
-
-<span class="sd"> partitions.get_index(1, 'foo', 'a') returns 0</span>
-<span class="sd"> partitions.get_index(1, 'foo', 'b') returns 1</span>
-<span class="sd"> partitions.get_index(1, 'foo', 'c') returns 2</span>
-<span class="sd"> partitions.get_index(1, 'foo', 'a') returns 0</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> level : int</span>
-<span class="sd"> The nesting level of the partition we are observing</span>
-<span class="sd"> name : string</span>
-<span class="sd"> The partition name</span>
-<span class="sd"> key : string or int</span>
-<span class="sd"> The partition value</span>
-<span class="sd"> """</span>
- <span class="k">if</span> <span class="n">level</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition_names</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'</span><span class="si">{0}</span><span class="s1"> was the name of the partition in '</span>
- <span class="s1">'another level'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">))</span>
-
- <span class="n">part_set</span> <span class="o">=</span> <span class="n">PartitionSet</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">part_set</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_names</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="p">[</span><span class="n">level</span><span class="p">]</span><span class="o">.</span><span class="n">get_index</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">filter_accepts_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">part_key</span><span class="p">,</span> <span class="nb">filter</span><span class="p">,</span> <span class="n">level</span><span class="p">):</span>
- <span class="n">p_column</span><span class="p">,</span> <span class="n">p_value_index</span> <span class="o">=</span> <span class="n">part_key</span>
- <span class="n">f_column</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">f_value</span> <span class="o">=</span> <span class="nb">filter</span>
- <span class="k">if</span> <span class="n">p_column</span> <span class="o">!=</span> <span class="n">f_column</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">True</span>
-
- <span class="n">f_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">f_value</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">f_value</span><span class="p">,</span> <span class="nb">set</span><span class="p">):</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">f_value</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Cannot use empty set as filter value"</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">op</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">{</span><span class="s1">'in'</span><span class="p">,</span> <span class="s1">'not in'</span><span class="p">}:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Op '</span><span class="si">%s</span><span class="s2">' not supported with set value"</span><span class="p">,</span>
- <span class="n">op</span><span class="p">)</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">([</span><span class="nb">type</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">f_value</span><span class="p">]))</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"All elements of set '</span><span class="si">%s</span><span class="s2">' must be of"</span>
- <span class="s2">" same type"</span><span class="p">,</span> <span class="n">f_value</span><span class="p">)</span>
- <span class="n">f_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="nb">next</span><span class="p">(</span><span class="nb">iter</span><span class="p">(</span><span class="n">f_value</span><span class="p">)))</span>
-
- <span class="n">p_value</span> <span class="o">=</span> <span class="n">f_type</span><span class="p">((</span><span class="bp">self</span><span class="o">.</span><span class="n">levels</span><span class="p">[</span><span class="n">level</span><span class="p">]</span>
- <span class="o">.</span><span class="n">dictionary</span><span class="p">[</span><span class="n">p_value_index</span><span class="p">]</span>
- <span class="o">.</span><span class="n">as_py</span><span class="p">()))</span>
-
- <span class="k">if</span> <span class="n">op</span> <span class="o">==</span> <span class="s2">"="</span> <span class="ow">or</span> <span class="n">op</span> <span class="o">==</span> <span class="s2">"=="</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o">==</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s2">"!="</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o">!=</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'<'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o"><</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'>'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o">></span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'<='</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o"><=</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'>='</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="o">>=</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'in'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="ow">in</span> <span class="n">f_value</span>
- <span class="k">elif</span> <span class="n">op</span> <span class="o">==</span> <span class="s1">'not in'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">p_value</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">f_value</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"'</span><span class="si">%s</span><span class="s2">' is not a valid operator in predicates."</span><span class="p">,</span>
- <span class="nb">filter</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
-
-
-<span class="k">class</span> <span class="nc">ParquetManifest</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-
-<span class="sd"> """</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dirpath</span><span class="p">,</span> <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">pathsep</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span>
- <span class="n">partition_scheme</span><span class="o">=</span><span class="s1">'hive'</span><span class="p">,</span> <span class="n">metadata_nthreads</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
- <span class="n">filesystem</span><span class="p">,</span> <span class="n">dirpath</span> <span class="o">=</span> <span class="n">_get_filesystem_and_path</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">dirpath</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">filesystem</span> <span class="o">=</span> <span class="n">filesystem</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">pathsep</span> <span class="o">=</span> <span class="n">pathsep</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">dirpath</span> <span class="o">=</span> <span class="n">_stringify_path</span><span class="p">(</span><span class="n">dirpath</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partition_scheme</span> <span class="o">=</span> <span class="n">partition_scheme</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span> <span class="o">=</span> <span class="n">ParquetPartitions</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_metadata_nthreads</span> <span class="o">=</span> <span class="n">metadata_nthreads</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_thread_pool</span> <span class="o">=</span> <span class="n">futures</span><span class="o">.</span><span class="n">ThreadPoolExecutor</span><span class="p">(</span>
- <span class="n">max_workers</span><span class="o">=</span><span class="n">metadata_nthreads</span><span class="p">)</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span> <span class="o">=</span> <span class="kc">None</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span> <span class="o">=</span> <span class="kc">None</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">_visit_level</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">dirpath</span><span class="p">,</span> <span class="p">[])</span>
-
- <span class="c1"># Due to concurrency, pieces will potentially by out of order if the</span>
- <span class="c1"># dataset is partitioned so we sort them to yield stable results</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">piece</span><span class="p">:</span> <span class="n">piece</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="c1"># _common_metadata is a subset of _metadata</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">_thread_pool</span><span class="o">.</span><span class="n">shutdown</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">_visit_level</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">level</span><span class="p">,</span> <span class="n">base_path</span><span class="p">,</span> <span class="n">part_keys</span><span class="p">):</span>
- <span class="n">fs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">filesystem</span>
-
- <span class="n">_</span><span class="p">,</span> <span class="n">directories</span><span class="p">,</span> <span class="n">files</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">fs</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">base_path</span><span class="p">))</span>
-
- <span class="n">filtered_files</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
- <span class="n">full_path</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pathsep</span><span class="o">.</span><span class="n">join</span><span class="p">((</span><span class="n">base_path</span><span class="p">,</span> <span class="n">path</span><span class="p">))</span>
- <span class="k">if</span> <span class="n">path</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'_common_metadata'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span> <span class="o">=</span> <span class="n">full_path</span>
- <span class="k">elif</span> <span class="n">path</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'_metadata'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span> <span class="o">=</span> <span class="n">full_path</span>
- <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_should_silently_exclude</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="k">continue</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">filtered_files</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">full_path</span><span class="p">)</span>
-
- <span class="c1"># ARROW-1079: Filter out "private" directories starting with underscore</span>
- <span class="n">filtered_directories</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">pathsep</span><span class="o">.</span><span class="n">join</span><span class="p">((</span><span class="n">base_path</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span>
- <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">directories</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">_is_private_directory</span><span class="p">(</span><span class="n">x</span><span class="p">)]</span>
-
- <span class="n">filtered_files</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
- <span class="n">filtered_directories</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">filtered_files</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">filtered_directories</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Found files in an intermediate '</span>
- <span class="s1">'directory: </span><span class="si">{0}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">base_path</span><span class="p">))</span>
- <span class="k">elif</span> <span class="nb">len</span><span class="p">(</span><span class="n">filtered_directories</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_visit_directories</span><span class="p">(</span><span class="n">level</span><span class="p">,</span> <span class="n">filtered_directories</span><span class="p">,</span> <span class="n">part_keys</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_push_pieces</span><span class="p">(</span><span class="n">filtered_files</span><span class="p">,</span> <span class="n">part_keys</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">_should_silently_exclude</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span>
- <span class="k">return</span> <span class="p">(</span><span class="n">file_name</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'.crc'</span><span class="p">)</span> <span class="ow">or</span> <span class="c1"># Checksums</span>
- <span class="n">file_name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span> <span class="ow">or</span> <span class="c1"># Hidden files</span>
- <span class="n">file_name</span> <span class="ow">in</span> <span class="n">EXCLUDED_PARQUET_PATHS</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">_visit_directories</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">level</span><span class="p">,</span> <span class="n">directories</span><span class="p">,</span> <span class="n">part_keys</span><span class="p">):</span>
- <span class="n">futures_list</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">directories</span><span class="p">:</span>
- <span class="n">head</span><span class="p">,</span> <span class="n">tail</span> <span class="o">=</span> <span class="n">_path_split</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pathsep</span><span class="p">)</span>
- <span class="n">name</span><span class="p">,</span> <span class="n">key</span> <span class="o">=</span> <span class="n">_parse_hive_partition</span><span class="p">(</span><span class="n">tail</span><span class="p">)</span>
-
- <span class="n">index</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span><span class="o">.</span><span class="n">get_index</span><span class="p">(</span><span class="n">level</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
- <span class="n">dir_part_keys</span> <span class="o">=</span> <span class="n">part_keys</span> <span class="o">+</span> <span class="p">[(</span><span class="n">name</span><span class="p">,</span> <span class="n">index</span><span class="p">)]</span>
- <span class="c1"># If you have less threads than levels, the wait call will block</span>
- <span class="c1"># indefinitely due to multiple waits within a thread.</span>
- <span class="k">if</span> <span class="n">level</span> <span class="o"><</span> <span class="bp">self</span><span class="o">.</span><span class="n">_metadata_nthreads</span><span class="p">:</span>
- <span class="n">future</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_thread_pool</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_visit_level</span><span class="p">,</span>
- <span class="n">level</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
- <span class="n">path</span><span class="p">,</span>
- <span class="n">dir_part_keys</span><span class="p">)</span>
- <span class="n">futures_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">future</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_visit_level</span><span class="p">(</span><span class="n">level</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">dir_part_keys</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">futures_list</span><span class="p">:</span>
- <span class="n">futures</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="n">futures_list</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">_parse_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dirname</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition_scheme</span> <span class="o">==</span> <span class="s1">'hive'</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">_parse_hive_partition</span><span class="p">(</span><span class="n">dirname</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s1">'partition schema: </span><span class="si">{0}</span><span class="s1">'</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">partition_scheme</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">_push_pieces</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">files</span><span class="p">,</span> <span class="n">part_keys</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span>
- <span class="n">ParquetDatasetPiece</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">partition_keys</span><span class="o">=</span><span class="n">part_keys</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">files</span>
- <span class="p">])</span>
-
-
-<span class="k">def</span> <span class="nf">_parse_hive_partition</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
- <span class="k">if</span> <span class="s1">'='</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">value</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Directory name did not appear to be a '</span>
- <span class="s1">'partition: </span><span class="si">{0}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">value</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">value</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'='</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_is_private_directory</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
- <span class="n">_</span><span class="p">,</span> <span class="n">tail</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">tail</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'_'</span><span class="p">)</span> <span class="ow">and</span> <span class="s1">'='</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">tail</span>
-
-
-<span class="k">def</span> <span class="nf">_path_split</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">sep</span><span class="p">):</span>
- <span class="n">i</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">rfind</span><span class="p">(</span><span class="n">sep</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span>
- <span class="n">head</span><span class="p">,</span> <span class="n">tail</span> <span class="o">=</span> <span class="n">path</span><span class="p">[:</span><span class="n">i</span><span class="p">],</span> <span class="n">path</span><span class="p">[</span><span class="n">i</span><span class="p">:]</span>
- <span class="n">head</span> <span class="o">=</span> <span class="n">head</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="n">sep</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">head</span><span class="p">,</span> <span class="n">tail</span>
-
-
-<span class="n">EXCLUDED_PARQUET_PATHS</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'_SUCCESS'</span><span class="p">}</span>
-
-
-<div class="viewcode-block" id="ParquetDataset"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset">[docs]</a><span class="k">class</span> <span class="nc">ParquetDataset</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Encapsulates details of reading a complete Parquet dataset possibly</span>
-<span class="sd"> consisting of multiple files and partitions in subdirectories</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> path_or_paths : str or List[str]</span>
-<span class="sd"> A directory name, single file name, or list of file names</span>
-<span class="sd"> filesystem : FileSystem, default None</span>
-<span class="sd"> If nothing passed, paths assumed to be found in the local on-disk</span>
-<span class="sd"> filesystem</span>
-<span class="sd"> metadata : pyarrow.parquet.FileMetaData</span>
-<span class="sd"> Use metadata obtained elsewhere to validate file schemas</span>
-<span class="sd"> schema : pyarrow.parquet.Schema</span>
-<span class="sd"> Use schema obtained elsewhere to validate file schemas. Alternative to</span>
-<span class="sd"> metadata parameter</span>
-<span class="sd"> split_row_groups : boolean, default False</span>
-<span class="sd"> Divide files into pieces for each row group in the file</span>
-<span class="sd"> validate_schema : boolean, default True</span>
-<span class="sd"> Check that individual file schemas are all the same / compatible</span>
-<span class="sd"> filters : List[Tuple] or List[List[Tuple]] or None (default)</span>
-<span class="sd"> List of filters to apply, like ``[[('x', '=', 0), ...], ...]``. This</span>
-<span class="sd"> implements partition-level (hive) filtering only, i.e., to prevent the</span>
-<span class="sd"> loading of some files of the dataset.</span>
-
-<span class="sd"> Predicates are expressed in disjunctive normal form (DNF). This means</span>
-<span class="sd"> that the innermost tuple describe a single column predicate. These</span>
-<span class="sd"> inner predicate make are all combined with a conjunction (AND) into a</span>
-<span class="sd"> larger predicate. The most outer list then combines all filters</span>
-<span class="sd"> with a disjunction (OR). By this, we should be able to express all</span>
-<span class="sd"> kinds of filters that are possible using boolean logic.</span>
-
-<span class="sd"> This function also supports passing in as List[Tuple]. These predicates</span>
-<span class="sd"> are evaluated as a conjunction. To express OR in predictates, one must</span>
-<span class="sd"> use the (preferred) List[List[Tuple]] notation.</span>
-<span class="sd"> metadata_nthreads: int, default 1</span>
-<span class="sd"> How many threads to allow the thread pool which is used to read the</span>
-<span class="sd"> dataset metadata. Increasing this is helpful to read partitioned</span>
-<span class="sd"> datasets.</span>
-<span class="sd"> """</span>
-<div class="viewcode-block" id="ParquetDataset.__init__"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset.__init__">[docs]</a> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path_or_paths</span><span class="p">,</span> <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span cl [...]
- <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">split_row_groups</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">validate_schema</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
- <span class="n">filters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">metadata_nthreads</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
- <span class="n">a_path</span> <span class="o">=</span> <span class="n">path_or_paths</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">a_path</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
- <span class="n">a_path</span> <span class="o">=</span> <span class="n">a_path</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">_get_filesystem_and_path</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">a_path</span><span class="p">)</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">paths</span> <span class="o">=</span> <span class="p">[</span><span class="n">_parse_uri</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">path_or_paths</span><span class="p">]</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">paths</span> <span class="o">=</span> <span class="n">_parse_uri</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">)</span>
-
- <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span><span class="p">)</span> <span class="o">=</span> <span class="n">_make_manifest</span><span class="p">(</span>
- <span class="n">path_or_paths</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="p">,</span> <span class="n">metadata_nthreads</span><span class="o">=</span><span class="n">metadata_nthreads</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">common_metadata_path</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="n">metadata</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="o">=</span> <span class="kc">None</span>
-
- <span class="k">if</span> <span class="n">metadata</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">metadata_path</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="o">.</span><span class="n">metadata</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span> <span class="o">=</span> <span class="n">metadata</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">split_row_groups</span> <span class="o">=</span> <span class="n">split_row_groups</span>
-
- <span class="k">if</span> <span class="n">split_row_groups</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">"split_row_groups not yet implemented"</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">validate_schema</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">validate_schemas</span><span class="p">()</span>
-
- <span class="k">if</span> <span class="n">filters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">filters</span> <span class="o">=</span> <span class="n">_check_filters</span><span class="p">(</span><span class="n">filters</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_filter</span><span class="p">(</span><span class="n">filters</span><span class="p">)</span></div>
-
-<div class="viewcode-block" id="ParquetDataset.validate_schemas"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset.validate_schemas">[docs]</a> <span class="k">def</span> <span class="nf">validate_schemas</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="n">open_file</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_open_file_func</span><span class="p">()</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span><span class="o">.</span><span class="n">schema</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">get_metadata</span><span class="p">(</span><span class="n">open_file</span><span class="p">)</span><span class="o">.</span><span class="n">schema</span>
- <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">schema</span>
-
- <span class="c1"># Verify schemas are all compatible</span>
- <span class="n">dataset_schema</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">to_arrow_schema</span><span class="p">()</span>
- <span class="c1"># Exclude the partition columns from the schema, they are provided</span>
- <span class="c1"># by the path, not the DatasetPiece</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">partition_name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span><span class="o">.</span><span class="n">partition_names</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">dataset_schema</span><span class="o">.</span><span class="n">get_field_index</span><span class="p">(</span><span class="n">partition_name</span><span class="p">)</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
- <span class="n">field_idx</span> <span class="o">=</span> <span class="n">dataset_schema</span><span class="o">.</span><span class="n">get_field_index</span><span class="p">(</span><span class="n">partition_name</span><span class="p">)</span>
- <span class="n">dataset_schema</span> <span class="o">=</span> <span class="n">dataset_schema</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">field_idx</span><span class="p">)</span>
-
- <span class="k">for</span> <span class="n">piece</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="p">:</span>
- <span class="n">file_metadata</span> <span class="o">=</span> <span class="n">piece</span><span class="o">.</span><span class="n">get_metadata</span><span class="p">(</span><span class="n">open_file</span><span class="p">)</span>
- <span class="n">file_schema</span> <span class="o">=</span> <span class="n">file_metadata</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">to_arrow_schema</span><span class="p">()</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">dataset_schema</span><span class="o">.</span><span class="n">equals</span><span class="p">(</span><span class="n">file_schema</span><span class="p">,</span> <span class="n">check_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Schema in </span><span class="si">{0!s}</span><span class="s1"> was different. </span><span class="se">\n</span><span class="s1">'</span>
- <span class="s1">'</span><span class="si">{1!s}</span><span class="se">\n\n</span><span class="s1">vs</span><span class="se">\n\n</span><span class="si">{2!s}</span><span class="s1">'</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">piece</span><span class="p">,</span> <span class="n">file_schema</span><span class="p">,</span>
- <span class="n">dataset_schema</span><span class="p">))</span></div>
-
-<div class="viewcode-block" id="ParquetDataset.read"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset.read">[docs]</a> <span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span>< [...]
- <span class="sd">"""</span>
-<span class="sd"> Read multiple Parquet files as a single pyarrow.Table</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> columns : List[str]</span>
-<span class="sd"> Names of columns to read from the file</span>
-<span class="sd"> use_threads : boolean, default True</span>
-<span class="sd"> Perform multi-threaded column reads</span>
-<span class="sd"> use_pandas_metadata : bool, default False</span>
-<span class="sd"> Passed through to each dataset piece</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pyarrow.Table</span>
-<span class="sd"> Content of the file as a table (of columns)</span>
-<span class="sd"> """</span>
- <span class="n">open_file</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_open_file_func</span><span class="p">()</span>
-
- <span class="n">tables</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">piece</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span><span class="p">:</span>
- <span class="n">table</span> <span class="o">=</span> <span class="n">piece</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span>
- <span class="n">partitions</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partitions</span><span class="p">,</span>
- <span class="n">open_file_func</span><span class="o">=</span><span class="n">open_file</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
- <span class="n">tables</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">table</span><span class="p">)</span>
-
- <span class="n">all_data</span> <span class="o">=</span> <span class="n">lib</span><span class="o">.</span><span class="n">concat_tables</span><span class="p">(</span><span class="n">tables</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">use_pandas_metadata</span><span class="p">:</span>
- <span class="c1"># We need to ensure that this metadata is set in the Table's schema</span>
- <span class="c1"># so that Table.to_pandas will construct pandas.DataFrame with the</span>
- <span class="c1"># right index</span>
- <span class="n">common_metadata</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_common_pandas_metadata</span><span class="p">()</span>
- <span class="n">current_metadata</span> <span class="o">=</span> <span class="n">all_data</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">metadata</span> <span class="ow">or</span> <span class="p">{}</span>
-
- <span class="k">if</span> <span class="n">common_metadata</span> <span class="ow">and</span> <span class="sa">b</span><span class="s1">'pandas'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">current_metadata</span><span class="p">:</span>
- <span class="n">all_data</span> <span class="o">=</span> <span class="n">all_data</span><span class="o">.</span><span class="n">replace_schema_metadata</span><span class="p">({</span>
- <span class="sa">b</span><span class="s1">'pandas'</span><span class="p">:</span> <span class="n">common_metadata</span><span class="p">})</span>
-
- <span class="k">return</span> <span class="n">all_data</span></div>
-
-<div class="viewcode-block" id="ParquetDataset.read_pandas"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset.read_pandas">[docs]</a> <span class="k">def</span> <span class="nf">read_pandas</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read dataset including pandas metadata, if any. Other arguments passed</span>
-<span class="sd"> through to ParquetDataset.read, see docstring for further details</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> pyarrow.Table</span>
-<span class="sd"> Content of the file as a table (of columns)</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div>
-
- <span class="k">def</span> <span class="nf">_get_common_pandas_metadata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">None</span>
-
- <span class="n">keyvalues</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span><span class="o">.</span><span class="n">metadata</span>
- <span class="k">return</span> <span class="n">keyvalues</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="sa">b</span><span class="s1">'pandas'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
-
- <span class="k">def</span> <span class="nf">_get_open_file_func</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">fs</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="p">,</span> <span class="n">LocalFileSystem</span><span class="p">):</span>
- <span class="k">def</span> <span class="nf">open_file</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">meta</span><span class="p">,</span>
- <span class="n">common_metadata</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">def</span> <span class="nf">open_file</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">meta</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s1">'rb'</span><span class="p">),</span>
- <span class="n">metadata</span><span class="o">=</span><span class="n">meta</span><span class="p">,</span>
- <span class="n">common_metadata</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">common_metadata</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">open_file</span>
-
- <span class="k">def</span> <span class="nf">_filter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filters</span><span class="p">):</span>
- <span class="n">accepts_filter</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">partitions</span><span class="o">.</span><span class="n">filter_accepts_partition</span>
-
- <span class="k">def</span> <span class="nf">one_filter_accepts</span><span class="p">(</span><span class="n">piece</span><span class="p">,</span> <span class="nb">filter</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">all</span><span class="p">(</span><span class="n">accepts_filter</span><span class="p">(</span><span class="n">part_key</span><span class="p">,</span> <span class="nb">filter</span><span class="p">,</span> <span class="n">level</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">level</span><span class="p">,</span> <span class="n">part_key</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">piece</span><span class="o">.</span><span class="n">partition_keys</span><span class="p">))</span>
-
- <span class="k">def</span> <span class="nf">all_filters_accept</span><span class="p">(</span><span class="n">piece</span><span class="p">):</span>
- <span class="k">return</span> <span class="nb">any</span><span class="p">(</span><span class="nb">all</span><span class="p">(</span><span class="n">one_filter_accepts</span><span class="p">(</span><span class="n">piece</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">conjunction</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">conjunction</span> <span class="ow">in</span> <span class="n">filters</span><span class="p">)</span>
-
- <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">pieces</span> <span class="k">if</span> <span class="n">all_filters_accept</span><span class="p">(</span><span class="n">p</span><span class="p">)]</span></div>
-
-
-<span class="k">def</span> <span class="nf">_make_manifest</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">,</span> <span class="n">fs</span><span class="p">,</span> <span class="n">pathsep</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span> <span class="n">metadata_nthreads</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
- <span class="n">partitions</span> <span class="o">=</span> <span class="kc">None</span>
- <span class="n">common_metadata_path</span> <span class="o">=</span> <span class="kc">None</span>
- <span class="n">metadata_path</span> <span class="o">=</span> <span class="kc">None</span>
-
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
- <span class="c1"># Dask passes a directory as a list of length 1</span>
- <span class="n">path_or_paths</span> <span class="o">=</span> <span class="n">path_or_paths</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
-
- <span class="k">if</span> <span class="n">_is_path_like</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">)</span> <span class="ow">and</span> <span class="n">fs</span><span class="o">.</span><span class="n">isdir</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">):</span>
- <span class="n">manifest</span> <span class="o">=</span> <span class="n">ParquetManifest</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">,</span> <span class="n">filesystem</span><span class="o">=</span><span class="n">fs</span><span class="p">,</span>
- <span class="n">pathsep</span><span class="o">=</span><span class="n">fs</span><span class="o">.</span><span class="n">pathsep</span><span class="p">,</span>
- <span class="n">metadata_nthreads</span><span class="o">=</span><span class="n">metadata_nthreads</span><span class="p">)</span>
- <span class="n">common_metadata_path</span> <span class="o">=</span> <span class="n">manifest</span><span class="o">.</span><span class="n">common_metadata_path</span>
- <span class="n">metadata_path</span> <span class="o">=</span> <span class="n">manifest</span><span class="o">.</span><span class="n">metadata_path</span>
- <span class="n">pieces</span> <span class="o">=</span> <span class="n">manifest</span><span class="o">.</span><span class="n">pieces</span>
- <span class="n">partitions</span> <span class="o">=</span> <span class="n">manifest</span><span class="o">.</span><span class="n">partitions</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
- <span class="n">path_or_paths</span> <span class="o">=</span> <span class="p">[</span><span class="n">path_or_paths</span><span class="p">]</span>
-
- <span class="c1"># List of paths</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">path_or_paths</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Must pass at least one file path'</span><span class="p">)</span>
-
- <span class="n">pieces</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">path_or_paths</span><span class="p">:</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">fs</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="k">raise</span> <span class="ne">IOError</span><span class="p">(</span><span class="s1">'Passed non-file path: </span><span class="si">{0}</span><span class="s1">'</span>
- <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>
- <span class="n">piece</span> <span class="o">=</span> <span class="n">ParquetDatasetPiece</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="n">pieces</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">piece</span><span class="p">)</span>
-
- <span class="k">return</span> <span class="n">pieces</span><span class="p">,</span> <span class="n">partitions</span><span class="p">,</span> <span class="n">common_metadata_path</span><span class="p">,</span> <span class="n">metadata_path</span>
-
-
-<span class="n">_read_table_docstring</span> <span class="o">=</span> <span class="s2">"""</span>
-<span class="si">{0}</span><span class="s2"></span>
-
-<span class="s2">Parameters</span>
-<span class="s2">----------</span>
-<span class="s2">source: str, pyarrow.NativeFile, or file-like object</span>
-<span class="s2"> If a string passed, can be a single file name or directory name. For</span>
-<span class="s2"> file-like objects, only read a single file. Use pyarrow.BufferReader to</span>
-<span class="s2"> read a file contained in a bytes or buffer-like object</span>
-<span class="s2">columns: list</span>
-<span class="s2"> If not None, only these columns will be read from the file. A column</span>
-<span class="s2"> name may be a prefix of a nested field, e.g. 'a' will select 'a.b',</span>
-<span class="s2"> 'a.c', and 'a.d.e'</span>
-<span class="s2">use_threads : boolean, default True</span>
-<span class="s2"> Perform multi-threaded column reads</span>
-<span class="s2">metadata : FileMetaData</span>
-<span class="s2"> If separately computed</span>
-<span class="s2">memory_map : boolean, default True</span>
-<span class="s2"> If the source is a file path, use a memory map to read file, which can</span>
-<span class="s2"> improve performance in some environments</span>
-<span class="si">{1}</span><span class="s2"></span>
-
-<span class="s2">Returns</span>
-<span class="s2">-------</span>
-<span class="si">{2}</span><span class="s2"></span>
-<span class="s2">"""</span>
-
-
-<div class="viewcode-block" id="read_table"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.read_table.html#pyarrow.parquet.read_table">[docs]</a><span class="k">def</span> <span class="nf">read_table</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc">Tru [...]
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
- <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">_is_path_like</span><span class="p">(</span><span class="n">source</span><span class="p">):</span>
- <span class="n">fs</span><span class="p">,</span> <span class="n">path</span> <span class="o">=</span> <span class="n">_get_filesystem_and_path</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">source</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">fs</span><span class="o">.</span><span class="n">read_parquet</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">metadata</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span>
-
- <span class="n">pf</span> <span class="o">=</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">metadata</span><span class="o">=</span><span class="n">metadata</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">pf</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="n">use_pandas_metadata</span><span class="p">)</span></div>
-
-
-<span class="n">read_table</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="n">_read_table_docstring</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="s1">'Read a Table from Parquet format'</span><span class="p">,</span>
- <span class="sd">"""use_pandas_metadata : boolean, default False</span>
-<span class="sd"> If True and file has custom pandas schema metadata, ensure that</span>
-<span class="sd"> index columns are also loaded"""</span><span class="p">,</span>
- <span class="sd">"""pyarrow.Table</span>
-<span class="sd"> Content of the file as a table (of columns)"""</span><span class="p">)</span>
-
-
-<div class="viewcode-block" id="read_pandas"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.read_pandas.html#pyarrow.parquet.read_pandas">[docs]</a><span class="k">def</span> <span class="nf">read_pandas</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_threads</span><span class="o">=</span><span class="kc" [...]
- <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">read_table</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
- <span class="n">use_threads</span><span class="o">=</span><span class="n">use_threads</span><span class="p">,</span>
- <span class="n">metadata</span><span class="o">=</span><span class="n">metadata</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
- <span class="n">use_pandas_metadata</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span></div>
-
-
-<span class="n">read_pandas</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="n">_read_table_docstring</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="s1">'Read a Table from Parquet format, also reading DataFrame</span><span class="se">\n</span><span class="s1">'</span>
- <span class="s1">'index values if known in the file metadata'</span><span class="p">,</span>
- <span class="s1">''</span><span class="p">,</span>
- <span class="sd">"""pyarrow.Table</span>
-<span class="sd"> Content of the file as a Table of Columns, including DataFrame</span>
-<span class="sd"> indexes as columns"""</span><span class="p">)</span>
-
-
-<div class="viewcode-block" id="write_table"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table">[docs]</a><span class="k">def</span> <span class="nf">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">row_group_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n" [...]
- <span class="n">use_dictionary</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">compression</span><span class="o">=</span><span class="s1">'snappy'</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">coerce_timestamps</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">allow_truncated_timestamps</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
- <span class="n">flavor</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="n">row_group_size</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'chunk_size'</span><span class="p">,</span> <span class="n">row_group_size</span><span class="p">)</span>
- <span class="n">use_int96</span> <span class="o">=</span> <span class="n">use_deprecated_int96_timestamps</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="k">with</span> <span class="n">ParquetWriter</span><span class="p">(</span>
- <span class="n">where</span><span class="p">,</span> <span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
- <span class="n">filesystem</span><span class="o">=</span><span class="n">filesystem</span><span class="p">,</span>
- <span class="n">version</span><span class="o">=</span><span class="n">version</span><span class="p">,</span>
- <span class="n">flavor</span><span class="o">=</span><span class="n">flavor</span><span class="p">,</span>
- <span class="n">use_dictionary</span><span class="o">=</span><span class="n">use_dictionary</span><span class="p">,</span>
- <span class="n">coerce_timestamps</span><span class="o">=</span><span class="n">coerce_timestamps</span><span class="p">,</span>
- <span class="n">allow_truncated_timestamps</span><span class="o">=</span><span class="n">allow_truncated_timestamps</span><span class="p">,</span>
- <span class="n">compression</span><span class="o">=</span><span class="n">compression</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="n">use_int96</span><span class="p">,</span>
- <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">as</span> <span class="n">writer</span><span class="p">:</span>
- <span class="n">writer</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">row_group_size</span><span class="o">=</span><span class="n">row_group_size</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">_is_path_like</span><span class="p">(</span><span class="n">where</span><span class="p">):</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">_stringify_path</span><span class="p">(</span><span class="n">where</span><span class="p">))</span>
- <span class="k">except</span> <span class="n">os</span><span class="o">.</span><span class="n">error</span><span class="p">:</span>
- <span class="k">pass</span>
- <span class="k">raise</span></div>
-
-
-<span class="n">write_table</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="s2">"""</span>
-<span class="s2">Write a Table to Parquet format</span>
-
-<span class="s2">Parameters</span>
-<span class="s2">----------</span>
-<span class="s2">table : pyarrow.Table</span>
-<span class="s2">where: string or pyarrow.NativeFile</span>
-<span class="si">{0}</span><span class="s2"></span>
-<span class="s2">"""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">_parquet_writer_arg_docs</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">_mkdir_if_not_exists</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">fs</span><span class="o">.</span><span class="n">_isfilestore</span><span class="p">()</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">fs</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">fs</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
- <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
- <span class="k">assert</span> <span class="n">fs</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
-
-
-<div class="viewcode-block" id="write_to_dataset"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.write_to_dataset.html#pyarrow.parquet.write_to_dataset">[docs]</a><span class="k">def</span> <span class="nf">write_to_dataset</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">root_path</span><span class="p">,</span> <span class="n">partition_cols</span><span class="o">=</span><span class="kc">None</span><span class="p"> [...]
- <span class="n">filesystem</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">preserve_index</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Wrapper around parquet.write_table for writing a Table to</span>
-<span class="sd"> Parquet format by partitions.</span>
-<span class="sd"> For each combination of partition columns and values,</span>
-<span class="sd"> a subdirectories are created in the following</span>
-<span class="sd"> manner:</span>
-
-<span class="sd"> root_dir/</span>
-<span class="sd"> group1=value1</span>
-<span class="sd"> group2=value1</span>
-<span class="sd"> <uuid>.parquet</span>
-<span class="sd"> group2=value2</span>
-<span class="sd"> <uuid>.parquet</span>
-<span class="sd"> group1=valueN</span>
-<span class="sd"> group2=value1</span>
-<span class="sd"> <uuid>.parquet</span>
-<span class="sd"> group2=valueN</span>
-<span class="sd"> <uuid>.parquet</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> table : pyarrow.Table</span>
-<span class="sd"> root_path : string,</span>
-<span class="sd"> The root directory of the dataset</span>
-<span class="sd"> filesystem : FileSystem, default None</span>
-<span class="sd"> If nothing passed, paths assumed to be found in the local on-disk</span>
-<span class="sd"> filesystem</span>
-<span class="sd"> partition_cols : list,</span>
-<span class="sd"> Column names by which to partition the dataset</span>
-<span class="sd"> Columns are partitioned in the order they are given</span>
-<span class="sd"> preserve_index : bool,</span>
-<span class="sd"> Parameter for instantiating Table; preserve pandas index or not.</span>
-<span class="sd"> **kwargs : dict, kwargs for write_table function.</span>
-<span class="sd"> """</span>
- <span class="n">fs</span><span class="p">,</span> <span class="n">root_path</span> <span class="o">=</span> <span class="n">_get_filesystem_and_path</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">root_path</span><span class="p">)</span>
-
- <span class="n">_mkdir_if_not_exists</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="n">root_path</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="n">partition_cols</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">partition_cols</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">df</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
- <span class="n">partition_keys</span> <span class="o">=</span> <span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="n">col</span><span class="p">]</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">partition_cols</span><span class="p">]</span>
- <span class="n">data_df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">drop</span><span class="p">(</span><span class="n">partition_cols</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="s1">'columns'</span><span class="p">)</span>
- <span class="n">data_cols</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">columns</span><span class="o">.</span><span class="n">drop</span><span class="p">(</span><span class="n">partition_cols</span><span class="p">)</span>
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">data_cols</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'No data left to save outside partition columns'</span><span class="p">)</span>
-
- <span class="n">subschema</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">schema</span>
- <span class="c1"># ARROW-2891: Ensure the output_schema is preserved when writing a</span>
- <span class="c1"># partitioned dataset</span>
- <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">table</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">names</span><span class="p">:</span>
- <span class="k">if</span> <span class="p">(</span><span class="n">col</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'__index_level_'</span><span class="p">)</span> <span class="ow">or</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">partition_cols</span><span class="p">):</span>
- <span class="n">subschema</span> <span class="o">=</span> <span class="n">subschema</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">subschema</span><span class="o">.</span><span class="n">get_field_index</span><span class="p">(</span><span class="n">col</span><span class="p">))</span>
-
- <span class="k">for</span> <span class="n">keys</span><span class="p">,</span> <span class="n">subgroup</span> <span class="ow">in</span> <span class="n">data_df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="n">partition_keys</span><span class="p">):</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">):</span>
- <span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="n">keys</span><span class="p">,)</span>
- <span class="n">subdir</span> <span class="o">=</span> <span class="s1">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
- <span class="p">[</span><span class="s1">'</span><span class="si">{colname}</span><span class="s1">=</span><span class="si">{value}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">colname</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">val</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">val</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">partition_cols</span><span class="p">,</span> <span class="n">keys</span><span class="p">)])</span>
- <span class="n">subtable</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">Table</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">subgroup</span><span class="p">,</span>
- <span class="n">preserve_index</span><span class="o">=</span><span class="n">preserve_index</span><span class="p">,</span>
- <span class="n">schema</span><span class="o">=</span><span class="n">subschema</span><span class="p">,</span>
- <span class="n">safe</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
- <span class="n">prefix</span> <span class="o">=</span> <span class="s1">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">root_path</span><span class="p">,</span> <span class="n">subdir</span><span class="p">])</span>
- <span class="n">_mkdir_if_not_exists</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="n">prefix</span><span class="p">)</span>
- <span class="n">outfile</span> <span class="o">=</span> <span class="n">guid</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'.parquet'</span>
- <span class="n">full_path</span> <span class="o">=</span> <span class="s1">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">prefix</span><span class="p">,</span> <span class="n">outfile</span><span class="p">])</span>
- <span class="k">with</span> <span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">full_path</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="n">write_table</span><span class="p">(</span><span class="n">subtable</span><span class="p">,</span> <span class="n">f</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">outfile</span> <span class="o">=</span> <span class="n">guid</span><span class="p">()</span> <span class="o">+</span> <span class="s1">'.parquet'</span>
- <span class="n">full_path</span> <span class="o">=</span> <span class="s1">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">root_path</span><span class="p">,</span> <span class="n">outfile</span><span class="p">])</span>
- <span class="k">with</span> <span class="n">fs</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">full_path</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="n">write_table</span><span class="p">(</span><span class="n">table</span><span class="p">,</span> <span class="n">f</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="write_metadata"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.write_metadata.html#pyarrow.parquet.write_metadata">[docs]</a><span class="k">def</span> <span class="nf">write_metadata</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s1">'1.0'</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
- <span class="n">coerce_timestamps</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Write metadata-only Parquet file from schema</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> schema : pyarrow.Schema</span>
-<span class="sd"> where: string or pyarrow.NativeFile</span>
-<span class="sd"> version : {"1.0", "2.0"}, default "1.0"</span>
-<span class="sd"> The Parquet format version, defaults to 1.0</span>
-<span class="sd"> use_deprecated_int96_timestamps : boolean, default False</span>
-<span class="sd"> Write nanosecond resolution timestamps to INT96 Parquet format</span>
-<span class="sd"> coerce_timestamps : string, default None</span>
-<span class="sd"> Cast timestamps a particular resolution.</span>
-<span class="sd"> Valid values: {None, 'ms', 'us'}</span>
-<span class="sd"> filesystem : FileSystem, default None</span>
-<span class="sd"> If nothing passed, paths assumed to be found in the local on-disk</span>
-<span class="sd"> filesystem</span>
-<span class="sd"> """</span>
- <span class="n">writer</span> <span class="o">=</span> <span class="n">ParquetWriter</span><span class="p">(</span>
- <span class="n">where</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="n">version</span><span class="p">,</span>
- <span class="n">use_deprecated_int96_timestamps</span><span class="o">=</span><span class="n">use_deprecated_int96_timestamps</span><span class="p">,</span>
- <span class="n">coerce_timestamps</span><span class="o">=</span><span class="n">coerce_timestamps</span><span class="p">)</span>
- <span class="n">writer</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
-
-
-<div class="viewcode-block" id="read_metadata"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.read_metadata.html#pyarrow.parquet.read_metadata">[docs]</a><span class="k">def</span> <span class="nf">read_metadata</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read FileMetadata from footer of a single Parquet file</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> where : string (filepath) or file-like object</span>
-<span class="sd"> memory_map : boolean, default False</span>
-<span class="sd"> Create memory map when the source is a file path</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> metadata : FileMetadata</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="n">memory_map</span><span class="p">)</span><span class="o">.</span><span class="n">metadata</span></div>
-
-
-<div class="viewcode-block" id="read_schema"><a class="viewcode-back" href="../../python/generated/pyarrow.parquet.read_schema.html#pyarrow.parquet.read_schema">[docs]</a><span class="k">def</span> <span class="nf">read_schema</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Read effective Arrow schema from Parquet file metadata</span>
-
-<span class="sd"> Parameters</span>
-<span class="sd"> ----------</span>
-<span class="sd"> where : string (filepath) or file-like object</span>
-<span class="sd"> memory_map : boolean, default False</span>
-<span class="sd"> Create memory map when the source is a file path</span>
-
-<span class="sd"> Returns</span>
-<span class="sd"> -------</span>
-<span class="sd"> schema : pyarrow.Schema</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">ParquetFile</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="n">memory_map</span><span class="o">=</span><span class="n">memory_map</span><span class="p">)</span><span class="o">.</span><span class="n">schema</span><span class="o">.</span><span class="n">to_arrow_schema</span><span class="p">()</span></div>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_modules/pyarrow/types.html b/docs/latest/_modules/pyarrow/types.html
deleted file mode 100644
index 8abad45..0000000
--- a/docs/latest/_modules/pyarrow/types.html
+++ /dev/null
@@ -1,516 +0,0 @@
-
-
-
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
- <meta charset="utf-8">
-
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
- <title>pyarrow.types — Apache Arrow v0.11.1.dev473+g6ed02454</title>
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
- <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
- <link rel="index" title="Index" href="../../genindex.html" />
- <link rel="search" title="Search" href="../../search.html" />
-
-
- <script src="../../_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav">
-
-
- <div class="wy-grid-for-nav">
-
-
- <nav data-toggle="wy-nav-shift" class="wy-nav-side">
- <div class="wy-side-scroll">
- <div class="wy-side-nav-search">
-
-
-
- <a href="../../index.html" class="icon icon-home"> Apache Arrow
-
-
-
- </a>
-
-
-
-
- <div class="version">
- 0.11.1.dev473+g6ed02454
- </div>
-
-
-
-
-<div role="search">
- <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
- <input type="text" name="q" placeholder="Search docs" />
- <input type="hidden" name="check_keywords" value="yes" />
- <input type="hidden" name="area" value="default" />
- </form>
-</div>
-
-
- </div>
-
- <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-
-
-
-
-
-
- <p class="caption"><span class="caption-text">Memory Format</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../format/README.html">Arrow specification documents</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Guidelines.html">Implementation guidelines</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Layout.html">Physical memory layout</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/Metadata.html">Metadata: Logical types, schemas, data headers</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../format/IPC.html">Interprocess messaging / communication (IPC)</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Languages</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../../cpp/index.html">C++ Implementation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../python/index.html">Python bindings</a></li>
-</ul>
-
-
-
- </div>
- </div>
- </nav>
-
- <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-
- <nav class="wy-nav-top" aria-label="top navigation">
-
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../index.html">Apache Arrow</a>
-
- </nav>
-
-
- <div class="wy-nav-content">
-
- <div class="rst-content">
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
- <ul class="wy-breadcrumbs">
-
- <li><a href="../../index.html">Docs</a> »</li>
-
- <li><a href="../index.html">Module code</a> »</li>
-
- <li><a href="../pyarrow.html">pyarrow</a> »</li>
-
- <li>pyarrow.types</li>
-
-
- <li class="wy-breadcrumbs-aside">
-
- </li>
-
- </ul>
-
-
- <hr/>
-</div>
- <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
- <div itemprop="articleBody">
-
- <h1>Source code for pyarrow.types</h1><div class="highlight"><pre>
-<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
-<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
-<span class="c1"># distributed with this work for additional information</span>
-<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
-<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
-<span class="c1"># "License"); you may not use this file except in compliance</span>
-<span class="c1"># with the License. You may obtain a copy of the License at</span>
-<span class="c1">#</span>
-<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
-<span class="c1">#</span>
-<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
-<span class="c1"># software distributed under the License is distributed on an</span>
-<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
-<span class="c1"># KIND, either express or implied. See the License for the</span>
-<span class="c1"># specific language governing permissions and limitations</span>
-<span class="c1"># under the License.</span>
-
-<span class="c1"># Tools for dealing with Arrow type metadata in Python</span>
-
-<span class="kn">from</span> <span class="nn">pyarrow.lib</span> <span class="k">import</span> <span class="p">(</span><span class="n">is_boolean_value</span><span class="p">,</span> <span class="c1"># noqa</span>
- <span class="n">is_integer_value</span><span class="p">,</span>
- <span class="n">is_float_value</span><span class="p">)</span>
-
-<span class="kn">import</span> <span class="nn">pyarrow.lib</span> <span class="k">as</span> <span class="nn">lib</span>
-
-
-<span class="n">_SIGNED_INTEGER_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_INT8</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT16</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT32</span><span class="p">,</span>
- <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT64</span><span class="p">}</span>
-<span class="n">_UNSIGNED_INTEGER_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT8</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT16</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT32</span><span class="p">,</span>
- <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT64</span><span class="p">}</span>
-<span class="n">_INTEGER_TYPES</span> <span class="o">=</span> <span class="n">_SIGNED_INTEGER_TYPES</span> <span class="o">|</span> <span class="n">_UNSIGNED_INTEGER_TYPES</span>
-<span class="n">_FLOATING_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_HALF_FLOAT</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_FLOAT</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DOUBLE</span><span class="p">}</span>
-<span class="n">_DATE_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_DATE32</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DATE64</span><span class="p">}</span>
-<span class="n">_TIME_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_TIME32</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_TIME64</span><span class="p">}</span>
-<span class="n">_TEMPORAL_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_TIMESTAMP</span><span class="p">}</span> <span class="o">|</span> <span class="n">_TIME_TYPES</span> <span class="o">|</span> <span class="n">_DATE_TYPES</span>
-<span class="n">_NESTED_TYPES</span> <span class="o">=</span> <span class="p">{</span><span class="n">lib</span><span class="o">.</span><span class="n">Type_LIST</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_STRUCT</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UNION</span><span class="p">,</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_MAP</span>< [...]
-
-
-<div class="viewcode-block" id="is_null"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_null.html#pyarrow.types.is_null">[docs]</a><span class="k">def</span> <span class="nf">is_null</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a null type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_NA</span></div>
-
-
-<div class="viewcode-block" id="is_boolean"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_boolean.html#pyarrow.types.is_boolean">[docs]</a><span class="k">def</span> <span class="nf">is_boolean</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a boolean type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_BOOL</span></div>
-
-
-<div class="viewcode-block" id="is_integer"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_integer.html#pyarrow.types.is_integer">[docs]</a><span class="k">def</span> <span class="nf">is_integer</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of any integer type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_INTEGER_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_signed_integer"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_signed_integer.html#pyarrow.types.is_signed_integer">[docs]</a><span class="k">def</span> <span class="nf">is_signed_integer</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of any signed integer type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_SIGNED_INTEGER_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_unsigned_integer"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_unsigned_integer.html#pyarrow.types.is_unsigned_integer">[docs]</a><span class="k">def</span> <span class="nf">is_unsigned_integer</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of any unsigned integer type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_UNSIGNED_INTEGER_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_int8"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_int8.html#pyarrow.types.is_int8">[docs]</a><span class="k">def</span> <span class="nf">is_int8</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an int8 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT8</span></div>
-
-
-<div class="viewcode-block" id="is_int16"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_int16.html#pyarrow.types.is_int16">[docs]</a><span class="k">def</span> <span class="nf">is_int16</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an int16 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT16</span></div>
-
-
-<div class="viewcode-block" id="is_int32"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_int32.html#pyarrow.types.is_int32">[docs]</a><span class="k">def</span> <span class="nf">is_int32</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an int32 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT32</span></div>
-
-
-<div class="viewcode-block" id="is_int64"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_int64.html#pyarrow.types.is_int64">[docs]</a><span class="k">def</span> <span class="nf">is_int64</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an int64 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_INT64</span></div>
-
-
-<div class="viewcode-block" id="is_uint8"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_uint8.html#pyarrow.types.is_uint8">[docs]</a><span class="k">def</span> <span class="nf">is_uint8</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an uint8 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT8</span></div>
-
-
-<div class="viewcode-block" id="is_uint16"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_uint16.html#pyarrow.types.is_uint16">[docs]</a><span class="k">def</span> <span class="nf">is_uint16</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an uint16 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT16</span></div>
-
-
-<div class="viewcode-block" id="is_uint32"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_uint32.html#pyarrow.types.is_uint32">[docs]</a><span class="k">def</span> <span class="nf">is_uint32</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an uint32 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT32</span></div>
-
-
-<div class="viewcode-block" id="is_uint64"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_uint64.html#pyarrow.types.is_uint64">[docs]</a><span class="k">def</span> <span class="nf">is_uint64</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an uint64 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UINT64</span></div>
-
-
-<div class="viewcode-block" id="is_floating"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_floating.html#pyarrow.types.is_floating">[docs]</a><span class="k">def</span> <span class="nf">is_floating</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a floating point numeric type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_FLOATING_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_float16"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_float16.html#pyarrow.types.is_float16">[docs]</a><span class="k">def</span> <span class="nf">is_float16</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an float16 (half-precision) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_HALF_FLOAT</span></div>
-
-
-<div class="viewcode-block" id="is_float32"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_float32.html#pyarrow.types.is_float32">[docs]</a><span class="k">def</span> <span class="nf">is_float32</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an float32 (single precision) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_FLOAT</span></div>
-
-
-<div class="viewcode-block" id="is_float64"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_float64.html#pyarrow.types.is_float64">[docs]</a><span class="k">def</span> <span class="nf">is_float64</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of an float64 (double precision) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DOUBLE</span></div>
-
-
-<div class="viewcode-block" id="is_list"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_list.html#pyarrow.types.is_list">[docs]</a><span class="k">def</span> <span class="nf">is_list</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a list type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_LIST</span></div>
-
-
-<div class="viewcode-block" id="is_struct"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_struct.html#pyarrow.types.is_struct">[docs]</a><span class="k">def</span> <span class="nf">is_struct</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a struct type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_STRUCT</span></div>
-
-
-<div class="viewcode-block" id="is_union"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_union.html#pyarrow.types.is_union">[docs]</a><span class="k">def</span> <span class="nf">is_union</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a union type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_UNION</span></div>
-
-
-<div class="viewcode-block" id="is_nested"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_nested.html#pyarrow.types.is_nested">[docs]</a><span class="k">def</span> <span class="nf">is_nested</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a nested type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_NESTED_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_temporal"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_temporal.html#pyarrow.types.is_temporal">[docs]</a><span class="k">def</span> <span class="nf">is_temporal</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a temporal (date, time, timestamp)</span>
-<span class="sd"> type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_TEMPORAL_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_timestamp"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_timestamp.html#pyarrow.types.is_timestamp">[docs]</a><span class="k">def</span> <span class="nf">is_timestamp</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a timestamp type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_TIMESTAMP</span></div>
-
-
-<div class="viewcode-block" id="is_time"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_time.html#pyarrow.types.is_time">[docs]</a><span class="k">def</span> <span class="nf">is_time</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a time type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_TIME_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_time32"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_time32.html#pyarrow.types.is_time32">[docs]</a><span class="k">def</span> <span class="nf">is_time32</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a time32 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_TIME32</span></div>
-
-
-<div class="viewcode-block" id="is_time64"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_time64.html#pyarrow.types.is_time64">[docs]</a><span class="k">def</span> <span class="nf">is_time64</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a time64 type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_TIME64</span></div>
-
-
-<div class="viewcode-block" id="is_binary"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_binary.html#pyarrow.types.is_binary">[docs]</a><span class="k">def</span> <span class="nf">is_binary</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a variable-length binary type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_BINARY</span></div>
-
-
-<div class="viewcode-block" id="is_unicode"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_unicode.html#pyarrow.types.is_unicode">[docs]</a><span class="k">def</span> <span class="nf">is_unicode</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Alias for is_string</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">is_string</span><span class="p">(</span><span class="n">t</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="is_string"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_string.html#pyarrow.types.is_string">[docs]</a><span class="k">def</span> <span class="nf">is_string</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of string (utf8 unicode) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_STRING</span></div>
-
-
-<div class="viewcode-block" id="is_fixed_size_binary"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_fixed_size_binary.html#pyarrow.types.is_fixed_size_binary">[docs]</a><span class="k">def</span> <span class="nf">is_fixed_size_binary</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a fixed size binary type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_FIXED_SIZE_BINARY</span></div>
-
-
-<div class="viewcode-block" id="is_date"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_date.html#pyarrow.types.is_date">[docs]</a><span class="k">def</span> <span class="nf">is_date</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a date type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="ow">in</span> <span class="n">_DATE_TYPES</span></div>
-
-
-<div class="viewcode-block" id="is_date32"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_date32.html#pyarrow.types.is_date32">[docs]</a><span class="k">def</span> <span class="nf">is_date32</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a date32 (days) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DATE32</span></div>
-
-
-<div class="viewcode-block" id="is_date64"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_date64.html#pyarrow.types.is_date64">[docs]</a><span class="k">def</span> <span class="nf">is_date64</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a date64 (milliseconds) type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DATE64</span></div>
-
-
-<div class="viewcode-block" id="is_map"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_map.html#pyarrow.types.is_map">[docs]</a><span class="k">def</span> <span class="nf">is_map</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a map logical type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_MAP</span></div>
-
-
-<div class="viewcode-block" id="is_decimal"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_decimal.html#pyarrow.types.is_decimal">[docs]</a><span class="k">def</span> <span class="nf">is_decimal</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a decimal type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DECIMAL</span></div>
-
-
-<div class="viewcode-block" id="is_dictionary"><a class="viewcode-back" href="../../python/generated/pyarrow.types.is_dictionary.html#pyarrow.types.is_dictionary">[docs]</a><span class="k">def</span> <span class="nf">is_dictionary</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if value is an instance of a dictionary-encoded type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">t</span><span class="o">.</span><span class="n">id</span> <span class="o">==</span> <span class="n">lib</span><span class="o">.</span><span class="n">Type_DICTIONARY</span></div>
-
-
-<span class="k">def</span> <span class="nf">is_primitive</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Return True if the value is an instance of a primitive type</span>
-<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">lib</span><span class="o">.</span><span class="n">_is_primitive</span><span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
-</pre></div>
-
- </div>
-
- </div>
- <footer>
-
-
- <hr/>
-
- <div role="contentinfo">
- <p>
- © Copyright 2016-2018 Apache Software Foundation
-
- </p>
- </div>
- Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
-
-</footer>
-
- </div>
- </div>
-
- </section>
-
- </div>
-
-
-
-
-
-
-
- <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
- <script type="text/javascript" src="../../_static/jquery.js"></script>
- <script type="text/javascript" src="../../_static/underscore.js"></script>
- <script type="text/javascript" src="../../_static/doctools.js"></script>
-
-
-
-
- <script type="text/javascript" src="../../_static/js/theme.js"></script>
-
- <script type="text/javascript">
- jQuery(function () {
- SphinxRtdTheme.Navigation.enable(true);
- });
- </script>
-<script async src="https://www.googletagmanager.com/gtag/js?id=UA-107500873-1"></script>
-<script>
- window.dataLayer = window.dataLayer || [];
- function gtag(){dataLayer.push(arguments);}
- gtag('js', new Date());
-
- gtag('config', 'UA-107500873-1');
-</script>
-
-
-</body>
-</html>
\ No newline at end of file
diff --git a/docs/latest/_sources/cpp/api.rst.txt b/docs/latest/_sources/cpp/api.rst.txt
deleted file mode 100644
index f6c0418..0000000
--- a/docs/latest/_sources/cpp/api.rst.txt
+++ /dev/null
@@ -1,30 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-*************
-API Reference
-*************
-
-.. toctree::
- :maxdepth: 3
-
- api/support
- api/memory
- api/datatype
- api/array
- api/builder
- api/table
diff --git a/docs/latest/_sources/cpp/api/array.rst.txt b/docs/latest/_sources/cpp/api/array.rst.txt
deleted file mode 100644
index bb981d1..0000000
--- a/docs/latest/_sources/cpp/api/array.rst.txt
+++ /dev/null
@@ -1,92 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-======
-Arrays
-======
-
-.. doxygenclass:: arrow::Array
- :project: arrow_cpp
- :members:
-
-Concrete array subclasses
-=========================
-
-.. doxygenclass:: arrow::DictionaryArray
- :project: arrow_cpp
- :members:
-
-Non-nested
-----------
-
-.. doxygenclass:: arrow::FlatArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::NullArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::BinaryArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::StringArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::PrimitiveArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::BooleanArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::FixedSizeBinaryArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::Decimal128Array
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::NumericArray
- :project: arrow_cpp
- :members:
-
-Nested
-------
-
-.. doxygenclass:: arrow::UnionArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::ListArray
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::StructArray
- :project: arrow_cpp
- :members:
-
-Chunked Arrays
-==============
-
-.. doxygenclass:: arrow::ChunkedArray
- :project: arrow_cpp
- :members:
diff --git a/docs/latest/_sources/cpp/api/builder.rst.txt b/docs/latest/_sources/cpp/api/builder.rst.txt
deleted file mode 100644
index 9e6540a..0000000
--- a/docs/latest/_sources/cpp/api/builder.rst.txt
+++ /dev/null
@@ -1,56 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-==============
-Array Builders
-==============
-
-.. doxygenclass:: arrow::ArrayBuilder
- :members:
-
-Concrete builder subclasses
-===========================
-
-.. doxygenclass:: arrow::NullBuilder
- :members:
-
-.. doxygenclass:: arrow::BooleanBuilder
- :members:
-
-.. doxygenclass:: arrow::NumericBuilder
- :members:
-
-.. doxygenclass:: arrow::BinaryBuilder
- :members:
-
-.. doxygenclass:: arrow::StringBuilder
- :members:
-
-.. doxygenclass:: arrow::FixedSizeBinaryBuilder
- :members:
-
-.. doxygenclass:: arrow::Decimal128Builder
- :members:
-
-.. doxygenclass:: arrow::ListBuilder
- :members:
-
-.. doxygenclass:: arrow::StructBuilder
- :members:
-
-.. doxygenclass:: arrow::DictionaryBuilder
- :members:
diff --git a/docs/latest/_sources/cpp/api/datatype.rst.txt b/docs/latest/_sources/cpp/api/datatype.rst.txt
deleted file mode 100644
index adfc6e4..0000000
--- a/docs/latest/_sources/cpp/api/datatype.rst.txt
+++ /dev/null
@@ -1,148 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-==========
-Data Types
-==========
-
-.. doxygenenum:: arrow::Type::type
-
-.. doxygenclass:: arrow::DataType
- :members:
-
-.. _api-type-factories:
-
-Factory functions
-=================
-
-These functions are recommended for creating data types. They may return
-new objects or existing singletons, depending on the type requested.
-
-.. doxygengroup:: type-factories
- :project: arrow_cpp
- :content-only:
-
-Concrete type subclasses
-========================
-
-Primitive
----------
-
-.. doxygenclass:: arrow::NullType
- :members:
-
-.. doxygenclass:: arrow::BooleanType
- :members:
-
-.. doxygenclass:: arrow::Int8Type
- :members:
-
-.. doxygenclass:: arrow::Int16Type
- :members:
-
-.. doxygenclass:: arrow::Int32Type
- :members:
-
-.. doxygenclass:: arrow::Int64Type
- :members:
-
-.. doxygenclass:: arrow::UInt8Type
- :members:
-
-.. doxygenclass:: arrow::UInt16Type
- :members:
-
-.. doxygenclass:: arrow::UInt32Type
- :members:
-
-.. doxygenclass:: arrow::UInt64Type
- :members:
-
-.. doxygenclass:: arrow::HalfFloatType
- :members:
-
-.. doxygenclass:: arrow::FloatType
- :members:
-
-.. doxygenclass:: arrow::DoubleType
- :members:
-
-Time-related
-------------
-
-.. doxygenenum:: arrow::TimeUnit::type
-
-.. doxygenclass:: arrow::Date32Type
- :members:
-
-.. doxygenclass:: arrow::Date64Type
- :members:
-
-.. doxygenclass:: arrow::Time32Type
- :members:
-
-.. doxygenclass:: arrow::Time64Type
- :members:
-
-.. doxygenclass:: arrow::TimestampType
- :members:
-
-Binary-like
------------
-
-.. doxygenclass:: arrow::BinaryType
- :members:
-
-.. doxygenclass:: arrow::StringType
- :members:
-
-.. doxygenclass:: arrow::FixedSizeBinaryType
- :members:
-
-.. doxygenclass:: arrow::Decimal128Type
- :members:
-
-Nested
-------
-
-.. doxygenclass:: arrow::ListType
- :members:
-
-.. doxygenclass:: arrow::StructType
- :members:
-
-.. doxygenclass:: arrow::UnionType
- :members:
-
-Dictionary-encoded
-------------------
-
-.. doxygenclass:: arrow::DictionaryType
- :members:
-
-Fields and Schemas
-==================
-
-.. doxygengroup:: schema-factories
- :project: arrow_cpp
- :content-only:
-
-.. doxygenclass:: arrow::Field
- :members:
-
-.. doxygenclass:: arrow::Schema
- :members:
diff --git a/docs/latest/_sources/cpp/api/memory.rst.txt b/docs/latest/_sources/cpp/api/memory.rst.txt
deleted file mode 100644
index c921229..0000000
--- a/docs/latest/_sources/cpp/api/memory.rst.txt
+++ /dev/null
@@ -1,90 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Memory (management)
-===================
-
-Buffers
--------
-
-.. doxygenclass:: arrow::Buffer
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::MutableBuffer
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::ResizableBuffer
- :project: arrow_cpp
- :members:
-
-Memory Pools
-------------
-
-.. doxygenfunction:: arrow::default_memory_pool
- :project: arrow_cpp
-
-.. doxygenclass:: arrow::MemoryPool
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::LoggingMemoryPool
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::ProxyMemoryPool
- :project: arrow_cpp
- :members:
-
-Allocation Functions
---------------------
-
-These functions allocate a buffer from a particular memory pool.
-
-.. doxygengroup:: buffer-allocation-functions
- :project: arrow_cpp
- :content-only:
-
-Slicing
--------
-
-.. doxygengroup:: buffer-slicing-functions
- :project: arrow_cpp
- :content-only:
-
-Buffer Builders
----------------
-
-.. doxygenclass:: arrow::BufferBuilder
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::TypedBufferBuilder
- :project: arrow_cpp
- :members:
-
-STL Integration
----------------
-
-.. doxygenclass:: arrow::stl_allocator
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::STLMemoryPool
- :project: arrow_cpp
- :members:
diff --git a/docs/latest/_sources/cpp/api/support.rst.txt b/docs/latest/_sources/cpp/api/support.rst.txt
deleted file mode 100644
index b165a99..0000000
--- a/docs/latest/_sources/cpp/api/support.rst.txt
+++ /dev/null
@@ -1,29 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Programming Support
-===================
-
-Error return and reporting
---------------------------
-
-.. doxygenclass:: arrow::Status
- :project: arrow_cpp
- :members:
-
-.. doxygendefine:: ARROW_RETURN_NOT_OK
-
diff --git a/docs/latest/_sources/cpp/api/table.rst.txt b/docs/latest/_sources/cpp/api/table.rst.txt
deleted file mode 100644
index e8b4f8e..0000000
--- a/docs/latest/_sources/cpp/api/table.rst.txt
+++ /dev/null
@@ -1,52 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-========================
-Two-dimensional Datasets
-========================
-
-Columns
-=======
-
-.. doxygenclass:: arrow::Column
- :project: arrow_cpp
- :members:
-
-Tables
-======
-
-.. doxygenclass:: arrow::Table
- :project: arrow_cpp
- :members:
-
-.. doxygenfunction:: arrow::ConcatenateTables
- :project: arrow_cpp
-
-Record Batches
-==============
-
-.. doxygenclass:: arrow::RecordBatch
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::RecordBatchReader
- :project: arrow_cpp
- :members:
-
-.. doxygenclass:: arrow::TableBatchReader
- :project: arrow_cpp
- :members:
diff --git a/docs/latest/_sources/cpp/arrays.rst.txt b/docs/latest/_sources/cpp/arrays.rst.txt
deleted file mode 100644
index 0c5272d..0000000
--- a/docs/latest/_sources/cpp/arrays.rst.txt
+++ /dev/null
@@ -1,211 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-======
-Arrays
-======
-
-The central type in Arrow is the class :class:`arrow::Array`. An array
-represents a known-length sequence of values all having the same type.
-Internally, those values are represented by one or several buffers, the
-number and meaning of which depend on the array's data type, as documented
-in :doc:`the Arrow data layout specification <../format/Layout>`.
-
-Those buffers consist of the value data itself and an optional bitmap buffer
-that indicates which array entries are null values. The bitmap buffer
-can be entirely omitted if the array is known to have zero null values.
-
-There are concrete subclasses of :class:`arrow::Array` for each data type,
-that help you access individual values of the array.
-
-Building an array
-=================
-
-As Arrow objects are immutable, there are classes provided that help you
-build these objects incrementally from third-party data. These classes
-are organized in a hierarchy around the :class:`arrow::ArrayBuilder` base class,
-with concrete subclasses tailored for each particular data type.
-
-For example, to build an array of ``int64_t`` elements, we can use the
-:class:`arrow::Int64Builder` class. In the following example, we build an array
-of the range 1 to 8 where the element that should hold the value 4 is nulled::
-
- arrow::Int64Builder builder;
- builder.Append(1);
- builder.Append(2);
- builder.Append(3);
- builder.AppendNull();
- builder.Append(5);
- builder.Append(6);
- builder.Append(7);
- builder.Append(8);
-
- std::shared_ptr<arrow::Array> array;
- arrow::Status st = builder.Finish(&array);
- if (!st.ok()) {
- // ... do something on array building failure
- }
-
-The resulting Array (which can be casted to the concrete :class:`arrow::Int64Array`
-subclass if you want to access its values) then consists of two
-:class:`arrow::Buffer`\s.
-The first buffer holds the null bitmap, which consists here of a single byte with
-the bits ``0|0|0|0|1|0|0|0``. As we use `least-significant bit (LSB) numbering`_.
-this indicates that the fourth entry in the array is null. The second
-buffer is simply an ``int64_t`` array containing all the above values.
-As the fourth entry is null, the value at that position in the buffer is
-undefined.
-
-Here is how you could access the concrete array's contents::
-
- // Cast the Array to its actual type to access its data
- auto int64_array = std::static_pointer_cast<arrow::Int64Array>(array);
-
- // Get the pointer to the null bitmap.
- const uint8_t* null_bitmap = int64_array->null_bitmap_data();
-
- // Get the pointer to the actual data
- const int64_t* data = int64_array->raw_values();
-
- // Alternatively, given an array index, query its null bit and value directly
- int64_t index = 2;
- if (!int64_array->IsNull(index)) {
- int64_t value = int64_array->Value(index);
- }
-
-.. note::
- :class:`arrow::Int64Array` (respectively :class:`arrow::Int64Builder`) is
- just a ``typedef``, provided for convenience, of ``arrow::NumericArray<Int64Type>``
- (respectively ``arrow::NumericBuilder<Int64Type>``).
-
-.. _least-significant bit (LSB) numbering: https://en.wikipedia.org/wiki/Bit_numbering
-
-Performance
------------
-
-While it is possible to build an array value-by-value as in the example above,
-to attain highest performance it is recommended to use the bulk appending
-methods (usually named ``AppendValues``) in the concrete :class:`arrow::ArrayBuilder`
-subclasses.
-
-If you know the number of elements in advance, it is also recommended to
-presize the working area by calling the :func:`~arrow::ArrayBuilder::Resize`
-or :func:`~arrow::ArrayBuilder::Reserve` methods.
-
-Here is how one could rewrite the above example to take advantage of those
-APIs::
-
- arrow::Int64Builder builder;
- // Make place for 8 values in total
- builder.Resize(8);
- // Bulk append the given values (with a null in 4th place as indicated by the
- // validity vector)
- std::vector<bool> validity = {true, true, true, false, true, true, true, true};
- std::vector<int64_t> values = {1, 2, 3, 0, 5, 6, 7, 8};
- builder.AppendValues(values, validity);
-
- std::shared_ptr<arrow::Array> array;
- arrow::Status st = builder.Finish(&array);
-
-If you still must append values one by one, some concrete builder subclasses
-have methods marked "Unsafe" that assume the working area has been correctly
-presized, and offer higher performance in exchange::
-
- arrow::Int64Builder builder;
- // Make place for 8 values in total
- builder.Resize(8);
- builder.UnsafeAppend(1);
- builder.UnsafeAppend(2);
- builder.UnsafeAppend(3);
- builder.UnsafeAppendNull();
- builder.UnsafeAppend(5);
- builder.UnsafeAppend(6);
- builder.UnsafeAppend(7);
- builder.UnsafeAppend(8);
-
- std::shared_ptr<arrow::Array> array;
- arrow::Status st = builder.Finish(&array);
-
-
-Size Limitations and Recommendations
-====================================
-
-Some array types are structurally limited to 32-bit sizes. This is the case
-for list arrays (which can hold up to 2^31 elements), string arrays and binary
-arrays (which can hold up to 2GB of binary data), at least. Some other array
-types can hold up to 2^63 elements in the C++ implementation, but other Arrow
-implementations can have a 32-bit size limitation for those array types as well.
-
-For these reasons, it is recommended that huge data be chunked in subsets of
-more reasonable size.
-
-Chunked Arrays
-==============
-
-A :class:`arrow::ChunkedArray` is, like an array, a logical sequence of values;
-but unlike a simple array, a chunked array does not require the entire sequence
-to be physically contiguous in memory. Also, the constituents of a chunked array
-need not have the same size, but they must all have the same data type.
-
-A chunked array is constructed by agregating any number of arrays. Here we'll
-build a chunked array with the same logical values as in the example above,
-but in two separate chunks::
-
- std::vector<std::shared_ptr<arrow::Array>> chunks;
- std::shared_ptr<arrow::Array> array;
-
- // Build first chunk
- arrow::Int64Builder builder;
- builder.Append(1);
- builder.Append(2);
- builder.Append(3);
- if (!builder.Finish(&array).ok()) {
- // ... do something on array building failure
- }
- chunks.push_back(std::move(array));
-
- // Build second chunk
- builder.Reset();
- builder.AppendNull();
- builder.Append(5);
- builder.Append(6);
- builder.Append(7);
- builder.Append(8);
- if (!builder.Finish(&array).ok()) {
- // ... do something on array building failure
- }
- chunks.push_back(std::move(array));
-
- auto chunked_array = std::make_shared<arrow::ChunkedArray>(std::move(chunks));
-
- assert(chunked_array->num_chunks() == 2);
- // Logical length in number of values
- assert(chunked_array->length() == 8);
- assert(chunked_array->null_count() == 1);
-
-Slicing
-=======
-
-Like for physical memory buffers, it is possible to make zero-copy slices
-of arrays and chunked arrays, to obtain an array or chunked array referring
-to some logical subsequence of the data. This is done by calling the
-:func:`arrow::Array::Slice` and :func:`arrow::ChunkedArray::Slice` methods,
-respectively.
-
diff --git a/docs/latest/_sources/cpp/conventions.rst.txt b/docs/latest/_sources/cpp/conventions.rst.txt
deleted file mode 100644
index b042435..0000000
--- a/docs/latest/_sources/cpp/conventions.rst.txt
+++ /dev/null
@@ -1,91 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-Conventions
-===========
-
-The Arrow C++ API follows a few simple guidelines. As with many rules,
-there may be exceptions.
-
-Language version
-----------------
-
-Arrow is C++11-compatible. A few backports are used for newer functionality,
-for example the :class:`std::string_view` class.
-
-Namespacing
------------
-
-All the Arrow API (except macros) is namespaced inside a ``arrow`` namespace,
-and nested namespaces thereof.
-
-Safe pointers
--------------
-
-Arrow objects are usually passed and stored using safe pointers -- most of
-the time :class:`std::shared_ptr` but sometimes also :class:`std::unique_ptr`.
-
-Immutability
-------------
-
-Many Arrow objects are immutable: once constructed, their logical properties
-cannot change anymore. This makes it possible to use them in multi-threaded
-scenarios without requiring tedious and error-prone synchronization.
-
-There are obvious exceptions to this, such as IO objects or mutable data buffers.
-
-Error reporting
----------------
-
-Most APIs indicate a successful or erroneous outcome by returning a
-:class:`arrow::Status` instance. Arrow doesn't throw exceptions of its
-own, but third-party exceptions might propagate through, especially
-:class:`std::bad_alloc` (but Arrow doesn't use the standard allocators for
-large data).
-
-As a consequence, the result value of a function is generally passed as an
-out-pointer parameter, rather than as a function return value.
-
-(however, functions which always determiniscally succeed may eschew this
-convention and return their result directly)
-
-Here is an example of checking the outcome of an operation::
-
- const int64_t buffer_size = 4096;
- std::shared_ptr<arrow::Buffer> buffer;
-
- auto status = arrow::AllocateBuffer(buffer_size, &buffer);
- if (!status.ok()) {
- // ... handle error
- }
-
-If the caller function itself returns a :class:`arrow::Status` and wants
-to propagate any non-successful outcomes, a convenience macro
-:cpp:func:`ARROW_RETURN_NON_OK` is available::
-
- arrow::Status DoSomething() {
- const int64_t buffer_size = 4096;
- std::shared_ptr<arrow::Buffer> buffer;
- ARROW_RETURN_NON_OK(arrow::AllocateBuffer(buffer_size, &buffer));
- // ... allocation successful, do something with buffer below
-
- // return success at the end
- return Status::OK();
- }
diff --git a/docs/latest/_sources/cpp/datatypes.rst.txt b/docs/latest/_sources/cpp/datatypes.rst.txt
deleted file mode 100644
index 117c05b..0000000
--- a/docs/latest/_sources/cpp/datatypes.rst.txt
+++ /dev/null
@@ -1,65 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-Data Types
-==========
-
-Data types govern how physical data is interpreted. Their :ref:`specification
-<spec-logical-types>` allows binary interoperability between different Arrow
-implementations, including from different programming languages and runtimes
-(for example it is possible to access the same data, without copying, from
-both Python and Java using the :py:mod:`pyarrow.jvm` bridge module).
-
-Information about a data type in C++ can be represented in three ways:
-
-1. Using a :class:`arrow::DataType` instance (e.g. as a function argument)
-2. Using a :class:`arrow::DataType` concrete subclass (e.g. as a template
- parameter)
-3. Using a :type:`arrow::Type::type` enum value (e.g. as the condition of
- a switch statement)
-
-The first form (using a :class:`arrow::DataType` instance) is the most idiomatic
-and flexible. Runtime-parametric types can only be fully represented with
-a DataType instance. For example, a :class:`arrow::TimestampType` needs to be
-constructed at runtime with a :type:`arrow::TimeUnit::type` parameter; a
-:class:`arrow::Decimal128Type` with *scale* and *precision* parameters;
-a :class:`arrow::ListType` with a full child type (itself a
-:class:`arrow::DataType` instance).
-
-The two other forms can be used where performance is critical, in order to
-avoid paying the price of dynamic typing and polymorphism. However, some
-amount of runtime switching can still be required for parametric types.
-It is not possible to reify all possible types at compile time, since Arrow
-data types allows arbitrary nesting.
-
-Creating data types
--------------------
-
-To instantiate data types, it is recommended to call the provided
-:ref:`factory functions <api-type-factories>`::
-
- std::shared_ptr<arrow::DataType> type;
-
- // A 16-bit integer type
- type = arrow::int16();
- // A 64-bit timestamp type (with microsecond granularity)
- type = arrow::timestamp(arrow::TimeUnit::MICRO);
- // A list type of single-precision floating-point values
- type = arrow::list(arrow::float32());
diff --git a/docs/latest/_sources/cpp/examples.rst.txt b/docs/latest/_sources/cpp/examples.rst.txt
deleted file mode 100644
index 5f4372f..0000000
--- a/docs/latest/_sources/cpp/examples.rst.txt
+++ /dev/null
@@ -1,30 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-Examples
-========
-
-Row to columnar conversion
---------------------------
-
-The following example converts an array of structs to a :class:`arrow::Table`
-instance, and then converts it back to the original array of structs.
-
-.. literalinclude:: ../../../cpp/examples/arrow/row-wise-conversion-example.cc
diff --git a/docs/latest/_sources/cpp/getting_started.rst.txt b/docs/latest/_sources/cpp/getting_started.rst.txt
deleted file mode 100644
index 7c55b76..0000000
--- a/docs/latest/_sources/cpp/getting_started.rst.txt
+++ /dev/null
@@ -1,31 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-Getting Started
-===============
-
-.. toctree::
-
- overview
- conventions
- memory
- arrays
- datatypes
- tables
diff --git a/docs/latest/_sources/cpp/index.rst.txt b/docs/latest/_sources/cpp/index.rst.txt
deleted file mode 100644
index 1d70e6a..0000000
--- a/docs/latest/_sources/cpp/index.rst.txt
+++ /dev/null
@@ -1,32 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-C++ Implementation
-==================
-
-.. toctree::
- :maxdepth: 2
-
- getting_started
- examples
- api
-
-.. TODO add "topics" chapter
-.. - nested arrays
-.. - dictionary encoding
-
-.. TODO add "building" or "development" chapter
diff --git a/docs/latest/_sources/cpp/memory.rst.txt b/docs/latest/_sources/cpp/memory.rst.txt
deleted file mode 100644
index 23b4725..0000000
--- a/docs/latest/_sources/cpp/memory.rst.txt
+++ /dev/null
@@ -1,127 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-=================
-Memory Management
-=================
-
-Buffers
-=======
-
-To avoid passing around raw data pointers with varying and non-obvious
-lifetime rules, Arrow provides a generic abstraction called :class:`arrow::Buffer`.
-A Buffer encapsulates a pointer and data size, and generally also ties its
-lifetime to that of an underlying provider (in other words, a Buffer should
-*always* point to valid memory till its destruction). Buffers are untyped:
-they simply denote a physical memory area regardless of its intended meaning
-or interpretation.
-
-Buffers may be allocated by Arrow itself , or by third-party routines.
-For example, it is possible to pass the data of a Python bytestring as a Arrow
-buffer, keeping the Python object alive as necessary.
-
-In addition, buffers come in various flavours: mutable or not, resizable or
-not. Generally, you will hold a mutable buffer when building up a piece
-of data, then it will be frozen as an immutable container such as an
-:doc:`array <arrays>`.
-
-.. note::
- Some buffers may point to non-CPU memory, such as GPU-backed memory
- provided by a CUDA context. If you're writing a GPU-aware application,
- you will need to be careful not to interpret a GPU memory pointer as
- a CPU-reachable pointer, or vice-versa.
-
-Accessing Buffer Memory
------------------------
-
-Buffers provide fast access to the underlying memory using the
-:func:`~arrow::Buffer::size` and :func:`~arrow::Buffer::data` accessors
-(or :func:`~arrow::Buffer::mutable_data` for writable access to a mutable
-buffer).
-
-Slicing
--------
-
-It is possible to make zero-copy slices of buffers, to obtain a buffer
-referring to some contiguous subset of the underlying data. This is done
-by calling the :func:`arrow::SliceBuffer` and :func:`arrow::SliceMutableBuffer`
-functions.
-
-Allocating a Buffer
--------------------
-
-You can allocate a buffer yourself by calling one of the
-:func:`arrow::AllocateBuffer` or :func:`arrow::AllocateResizableBuffer`
-overloads::
-
- std::shared_ptr<arrow::Buffer> buffer;
-
- if (!arrow::AllocateBuffer(4096, &buffer).ok()) {
- // ... handle allocation error
- }
- uint8_t* buffer_data = buffer->mutable_data();
- memcpy(buffer_data, "hello world", 11);
-
-Allocating a buffer this way ensures it is 64-bytes aligned and padded
-as recommended by the :doc:`Arrow memory specification <../format/Layout>`.
-
-Building a Buffer
------------------
-
-You can also allocate *and* build a Buffer incrementally, using the
-:class:`arrow::BufferBuilder` API::
-
- BufferBuilder builder;
- builder.Resize(11);
- builder.Append("hello ", 6);
- builder.Append("world", 5);
-
- std::shared_ptr<arrow::Buffer> buffer;
- if (!builder.Finish(&buffer).ok()) {
- // ... handle buffer allocation error
- }
-
-Memory Pools
-============
-
-When allocating a Buffer using the Arrow C++ API, the buffer's underlying
-memory is allocated by a :class:`arrow::MemoryPool` instance. Usually this
-will be the process-wide *default memory pool*, but many Arrow APIs allow
-you to pass another MemoryPool instance for their internal allocations.
-
-Memory pools are used for large long-lived data such as array buffers.
-Other data, such as small C++ objects and temporary workspaces, usually
-goes through the regular C++ allocators.
-
-Default Memory Pool
--------------------
-
-Depending on how Arrow was compiled, the default memory pool may use the
-standard C ``malloc`` allocator, or a `jemalloc <http://jemalloc.net/>`_ heap.
-
-STL Integration
----------------
-
-If you wish to use a Arrow memory pool to allocate the data of STL containers,
-you can do so using the :class:`arrow::stl_allocator` wrapper.
-
-Conversely, you can also use a STL allocator to allocate Arrow memory,
-using the :class:`arrow::STLMemoryPool` class. However, this may be less
-performant, as STL allocators don't provide a resizing operation.
diff --git a/docs/latest/_sources/cpp/overview.rst.txt b/docs/latest/_sources/cpp/overview.rst.txt
deleted file mode 100644
index 490efc1..0000000
--- a/docs/latest/_sources/cpp/overview.rst.txt
+++ /dev/null
@@ -1,93 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-High-Level Overview
-===================
-
-The Arrow C++ library is comprised of different parts, each of which serves
-a specific purpose.
-
-The physical layer
-------------------
-
-**Memory management** abstractions provide a uniform API over memory that
-may be allocated through various means, such as heap allocation, the memory
-mapping of a file or a static memory area. In particular, the **buffer**
-abstraction represents a contiguous area of physical data.
-
-The one-dimensional layer
--------------------------
-
-**Data types** govern the *logical* interpretation of *physical* data.
-Many operations in Arrow are parametered, at compile-time or at runtime,
-by a data type.
-
-**Arrays** assemble one or several buffers with a data type, allowing to
-view them as a logical contiguous sequence of values (possibly nested).
-
-**Chunked arrays** are a generalization of arrays, comprising several same-type
-arrays into a longer logical sequence of values.
-
-The two-dimensional layer
--------------------------
-
-**Schemas** describe a logical collection of several pieces of data,
-each with a distinct name and type, and optional metadata.
-
-**Columns** are like chunked arrays, but with optional metadata.
-
-**Tables** are collections of columns in accordance to a schema. They are
-the most capable dataset-providing abstraction in Arrow.
-
-**Record batches** are collections of contiguous arrays, described
-by a schema. They allow incremental construction or serialization of tables.
-
-The compute layer
------------------
-
-**Datums** are flexible dataset references, able to hold for example an array or table
-reference.
-
-**Kernels** are specialized computation functions running in a loop over a
-given set of datums representing input and output parameters to the functions.
-
-The IO layer
-------------
-
-**Streams** allow untyped sequential or seekable access over external data
-of various kinds (for example compressed or memory-mapped).
-
-The Inter-Process Communication (IPC) layer
--------------------------------------------
-
-A **messaging format** allows interchange of Arrow data between processes, using
-as few copies as possible.
-
-The file formats layer
-----------------------
-
-Reading and writing Arrow data from/to various file formats is possible, for
-example **Parquet**, **CSV**, **Orc** or the Arrow-specific **Feather** format.
-
-The devices layer
------------------
-
-Basic **CUDA** integration is provided, allowing to describe Arrow data backed
-by GPU-allocated memory.
diff --git a/docs/latest/_sources/cpp/tables.rst.txt b/docs/latest/_sources/cpp/tables.rst.txt
deleted file mode 100644
index d42f0c6..0000000
--- a/docs/latest/_sources/cpp/tables.rst.txt
+++ /dev/null
@@ -1,87 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. default-domain:: cpp
-.. highlight:: cpp
-
-========================
-Two-dimensional Datasets
-========================
-
-While arrays and chunked arrays represent a one-dimensional sequence of
-homogenous values, data often comes in the form of two-dimensional sets of
-heterogenous data (such as database tables, CSV files...). Arrow provides
-several abstractions to handle such data conveniently and efficiently.
-
-Fields
-======
-
-Fields are used to denote the particular columns of a table (and also
-the particular members of a nested data type such as :class:`arrow::StructType`).
-A field, i.e. an instance of :class:`arrow::Field`, holds together a data
-type, a field name and some optional metadata.
-
-The recommended way to create a field is to call the :func:`arrow::field`
-factory function.
-
-Schemas
-=======
-
-A schema describes the overall structure of a two-dimensional dataset such
-as a table. It holds a sequence of fields together with some optional
-schema-wide metadata (in addition to per-field metadata). The recommended
-way to create a schema is to call one the :func:`arrow::schema` factory
-function overloads::
-
- // Create a schema describing datasets with two columns:
- // a int32 column "A" and a utf8-encoded string column "B"
- std::shared_ptr<arrow::Field> field_a, field_b;
- std::shared_ptr<arrow::Schema> schema;
-
- field_a = arrow::field("A", arrow::int32());
- field_b = arrow::field("B", arrow::utf8());
- schema = arrow::schema({field_a, field_b});
-
-Columns
-=======
-
-A :class:`arrow::Column` is a chunked array tied together with a field.
-The field describes the column's name (for lookup in a larger dataset)
-and its metadata.
-
-Tables
-======
-
-A :class:`arrow::Table` is a two-dimensional dataset of a number of columns,
-together with a schema. The columns' names and types must match the schema.
-Also, each column must have the same logical length in number of elements
-(although each column can be chunked in a different way).
-
-Record Batches
-==============
-
-A :class:`arrow::RecordBatch` is a two-dimensional dataset of a number of
-contiguous arrays, each the same length. Like a table, a record batch also
-has a schema which must match its arrays' datatypes.
-
-Record batches are a convenient unit of work for various serialization
-and computation functions, possibly incremental.
-
-A table can be streamed as an arbitrary number of record batches using
-a :class:`arrow::TableBatchReader`. Conversely, a logical sequence of
-record batches can be assembled to form a table using one of the
-:func:`arrow::Table::FromRecordBatches` factory function overloads.
diff --git a/docs/latest/_sources/format/Guidelines.rst.txt b/docs/latest/_sources/format/Guidelines.rst.txt
deleted file mode 100644
index 5b03220..0000000
--- a/docs/latest/_sources/format/Guidelines.rst.txt
+++ /dev/null
@@ -1,43 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Implementation guidelines
-=========================
-
-An execution engine (or framework, or UDF executor, or storage engine, etc) can implements only a subset of the Arrow spec and/or extend it given the following constraints:
-
-Implementing a subset the spec
-------------------------------
-
-If only producing (and not consuming) arrow vectors.
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Any subset of the vector spec and the corresponding metadata can be implemented.
-
-If consuming and producing vectors
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There is a minimal subset of vectors to be supported.
-Production of a subset of vectors and their corresponding metadata is always fine.
-Consumption of vectors should at least convert the unsupported input vectors to the supported subset (for example Timestamp.millis to timestamp.micros or int32 to int64)
-
-Extensibility
--------------
-
-An execution engine implementor can also extend their memory representation with their own vectors internally as long as they are never exposed. Before sending data to another system expecting Arrow data these custom vectors should be converted to a type that exist in the Arrow spec.
-An example of this is operating on compressed data.
-These custom vectors are not exchanged externally and there is no support for custom metadata.
diff --git a/docs/latest/_sources/format/IPC.rst.txt b/docs/latest/_sources/format/IPC.rst.txt
deleted file mode 100644
index 8cb74b8..0000000
--- a/docs/latest/_sources/format/IPC.rst.txt
+++ /dev/null
@@ -1,237 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Interprocess messaging / communication (IPC)
-============================================
-
-Encapsulated message format
----------------------------
-
-Data components in the stream and file formats are represented as encapsulated
-*messages* consisting of:
-
-* A length prefix indicating the metadata size
-* The message metadata as a `Flatbuffer`_
-* Padding bytes to an 8-byte boundary
-* The message body, which must be a multiple of 8 bytes
-
-Schematically, we have: ::
-
- <metadata_size: int32>
- <metadata_flatbuffer: bytes>
- <padding>
- <message body>
-
-The complete serialized message must be a multiple of 8 bytes so that messages
-can be relocated between streams. Otherwise the amount of padding between the
-metadata and the message body could be non-deterministic.
-
-The ``metadata_size`` includes the size of the flatbuffer plus padding. The
-``Message`` flatbuffer includes a version number, the particular message (as a
-flatbuffer union), and the size of the message body: ::
-
- table Message {
- version: org.apache.arrow.flatbuf.MetadataVersion;
- header: MessageHeader;
- bodyLength: long;
- }
-
-Currently, we support 4 types of messages:
-
-* Schema
-* RecordBatch
-* DictionaryBatch
-* Tensor
-
-Streaming format
-----------------
-
-We provide a streaming format for record batches. It is presented as a sequence
-of encapsulated messages, each of which follows the format above. The schema
-comes first in the stream, and it is the same for all of the record batches
-that follow. If any fields in the schema are dictionary-encoded, one or more
-``DictionaryBatch`` messages will be included. ``DictionaryBatch`` and
-``RecordBatch`` messages may be interleaved, but before any dictionary key is used
-in a ``RecordBatch`` it should be defined in a ``DictionaryBatch``. ::
-
- <SCHEMA>
- <DICTIONARY 0>
- ...
- <DICTIONARY k - 1>
- <RECORD BATCH 0>
- ...
- <DICTIONARY x DELTA>
- ...
- <DICTIONARY y DELTA>
- ...
- <RECORD BATCH n - 1>
- <EOS [optional]: int32>
-
-When a stream reader implementation is reading a stream, after each message, it
-may read the next 4 bytes to know how large the message metadata that follows
-is. Once the message flatbuffer is read, you can then read the message body.
-
-The stream writer can signal end-of-stream (EOS) either by writing a 0 length
-as an ``int32`` or simply closing the stream interface.
-
-File format
------------
-
-We define a "file format" supporting random access in a very similar format to
-the streaming format. The file starts and ends with a magic string ``ARROW1``
-(plus padding). What follows in the file is identical to the stream format. At
-the end of the file, we write a *footer* containing a redundant copy of the
-schema (which is a part of the streaming format) plus memory offsets and sizes
-for each of the data blocks in the file. This enables random access any record
-batch in the file. See ``File.fbs`` for the precise details of the file
-footer.
-
-Schematically we have: ::
-
- <magic number "ARROW1">
- <empty padding bytes [to 8 byte boundary]>
- <STREAMING FORMAT>
- <FOOTER>
- <FOOTER SIZE: int32>
- <magic number "ARROW1">
-
-In the file format, there is no requirement that dictionary keys should be
-defined in a ``DictionaryBatch`` before they are used in a ``RecordBatch``, as long
-as the keys are defined somewhere in the file.
-
-RecordBatch body structure
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``RecordBatch`` metadata contains a depth-first (pre-order) flattened set of
-field metadata and physical memory buffers (some comments from ``Message.fbs``
-have been shortened / removed): ::
-
- table RecordBatch {
- length: long;
- nodes: [FieldNode];
- buffers: [Buffer];
- }
-
- struct FieldNode {
- length: long;
- null_count: long;
- }
-
- struct Buffer {
- /// The relative offset into the shared memory page where the bytes for this
- /// buffer starts
- offset: long;
-
- /// The absolute length (in bytes) of the memory buffer. The memory is found
- /// from offset (inclusive) to offset + length (non-inclusive).
- length: long;
- }
-
-In the context of a file, the ``page`` is not used, and the ``Buffer`` offsets use
-as a frame of reference the start of the message body. So, while in a general
-IPC setting these offsets may be anyplace in one or more shared memory regions,
-in the file format the offsets start from 0.
-
-The location of a record batch and the size of the metadata block as well as
-the body of buffers is stored in the file footer: ::
-
- struct Block {
- offset: long;
- metaDataLength: int;
- bodyLength: long;
- }
-
-The ``metaDataLength`` here includes the metadata length prefix, serialized
-metadata, and any additional padding bytes, and by construction must be a
-multiple of 8 bytes.
-
-Some notes about this
-
-* The ``Block`` offset indicates the starting byte of the record batch.
-* The metadata length includes the flatbuffer size, the record batch metadata
- flatbuffer, and any padding bytes
-
-Dictionary Batches
-~~~~~~~~~~~~~~~~~~
-
-Dictionaries are written in the stream and file formats as a sequence of record
-batches, each having a single field. The complete semantic schema for a
-sequence of record batches, therefore, consists of the schema along with all of
-the dictionaries. The dictionary types are found in the schema, so it is
-necessary to read the schema to first determine the dictionary types so that
-the dictionaries can be properly interpreted. ::
-
- table DictionaryBatch {
- id: long;
- data: RecordBatch;
- isDelta: boolean = false;
- }
-
-The dictionary ``id`` in the message metadata can be referenced one or more times
-in the schema, so that dictionaries can even be used for multiple fields. See
-the :doc:`Layout` document for more about the semantics of
-dictionary-encoded data.
-
-The dictionary ``isDelta`` flag allows dictionary batches to be modified
-mid-stream. A dictionary batch with ``isDelta`` set indicates that its vector
-should be concatenated with those of any previous batches with the same ``id``. A
-stream which encodes one column, the list of strings
-``["A", "B", "C", "B", "D", "C", "E", "A"]``, with a delta dictionary batch could
-take the form: ::
-
- <SCHEMA>
- <DICTIONARY 0>
- (0) "A"
- (1) "B"
- (2) "C"
-
- <RECORD BATCH 0>
- 0
- 1
- 2
- 1
-
- <DICTIONARY 0 DELTA>
- (3) "D"
- (4) "E"
-
- <RECORD BATCH 1>
- 3
- 2
- 4
- 0
- EOS
-
-Tensor (Multi-dimensional Array) Message Format
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``Tensor`` message types provides a way to write a multidimensional array of
-fixed-size values (such as a NumPy ndarray) using Arrow's shared memory
-tools. Arrow implementations in general are not required to implement this data
-format, though we provide a reference implementation in C++.
-
-When writing a standalone encapsulated tensor message, we use the format as
-indicated above, but additionally align the starting offset of the metadata as
-well as the starting offset of the tensor body (if writing to a shared memory
-region) to be multiples of 64 bytes: ::
-
- <PADDING>
- <metadata size: int32>
- <metadata>
- <tensor body>
-
-.. _Flatbuffer: https://github.com/google/flatbuffers
diff --git a/docs/latest/_sources/format/Layout.rst.txt b/docs/latest/_sources/format/Layout.rst.txt
deleted file mode 100644
index 868a99b..0000000
--- a/docs/latest/_sources/format/Layout.rst.txt
+++ /dev/null
@@ -1,664 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Physical memory layout
-======================
-
-Definitions / Terminology
--------------------------
-
-Since different projects have used different words to describe various
-concepts, here is a small glossary to help disambiguate.
-
-* Array: a sequence of values with known length all having the same type.
-* Slot or array slot: a single logical value in an array of some particular data type
-* Contiguous memory region: a sequential virtual address space with a given
- length. Any byte can be reached via a single pointer offset less than the
- region's length.
-* Contiguous memory buffer: A contiguous memory region that stores
- a multi-value component of an Array. Sometimes referred to as just "buffer".
-* Primitive type: a data type that occupies a fixed-size memory slot specified
- in bit width or byte width
-* Nested or parametric type: a data type whose full structure depends on one or
- more other child relative types. Two fully-specified nested types are equal
- if and only if their child types are equal. For example, ``List<U>`` is distinct
- from ``List<V>`` iff U and V are different relative types.
-* Relative type or simply type (unqualified): either a specific primitive type
- or a fully-specified nested type. When we say slot we mean a relative type
- value, not necessarily any physical storage region.
-* Logical type: A data type that is implemented using some relative (physical)
- type. For example, Decimal values are stored as 16 bytes in a fixed byte
- size array. Similarly, strings can be stored as ``List<1-byte>``.
-* Parent and child arrays: names to express relationships between physical
- value arrays in a nested type structure. For example, a ``List<T>``-type parent
- array has a T-type array as its child (see more on lists below).
-* Leaf node or leaf: A primitive value array that may or may not be a child
- array of some array with a nested type.
-
-Requirements, goals, and non-goals
-----------------------------------
-
-Base requirements
-
-* A physical memory layout enabling zero-deserialization data interchange
- amongst a variety of systems handling flat and nested columnar data, including
- such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
- proprietary systems that utilize the open source components.
-* All array slots are accessible in constant time, with complexity growing
- linearly in the nesting level
-* Capable of representing fully-materialized and decoded / decompressed `Parquet`_
- data
-* It is required to have all the contiguous memory buffers in an IPC payload
- aligned at 8-byte boundaries. In other words, each buffer must start at
- an aligned 8-byte offset.
-* The general recommendation is to align the buffers at 64-byte boundary, but
- this is not absolutely necessary.
-* Any relative type can have null slots
-* Arrays are immutable once created. Implementations can provide APIs to mutate
- an array, but applying mutations will require a new array data structure to
- be built.
-* Arrays are relocatable (e.g. for RPC/transient storage) without pointer
- swizzling. Another way of putting this is that contiguous memory regions can
- be migrated to a different address space (e.g. via a memcpy-type of
- operation) without altering their contents.
-
-Goals (for this document)
--------------------------
-
-* To describe relative types (physical value types and a preliminary set of
- nested types) sufficient for an unambiguous implementation
-* Memory layout and random access patterns for each relative type
-* Null value representation
-
-Non-goals (for this document)
------------------------------
-
-* To enumerate or specify logical types that can be implemented as primitive
- (fixed-width) value types. For example: signed and unsigned integers,
- floating point numbers, boolean, exact decimals, date and time types,
- CHAR(K), VARCHAR(K), etc.
-* To specify standardized metadata or a data layout for RPC or transient file
- storage.
-* To define a selection or masking vector construct
-* Implementation-specific details
-* Details of a user or developer C/C++/Java API.
-* Any "table" structure composed of named arrays each having their own type or
- any other structure that composes arrays.
-* Any memory management or reference counting subsystem
-* To enumerate or specify types of encodings or compression support
-
-Byte Order (`Endianness`_)
----------------------------
-
-The Arrow format is little endian by default.
-The Schema metadata has an endianness field indicating endianness of RecordBatches.
-Typically this is the endianness of the system where the RecordBatch was generated.
-The main use case is exchanging RecordBatches between systems with the same Endianness.
-At first we will return an error when trying to read a Schema with an endianness
-that does not match the underlying system. The reference implementation is focused on
-Little Endian and provides tests for it. Eventually we may provide automatic conversion
-via byte swapping.
-
-Alignment and Padding
----------------------
-
-As noted above, all buffers must be aligned in memory at 8-byte boundaries and padded
-to a length that is a multiple of 8 bytes. The alignment requirement follows best
-practices for optimized memory access:
-
-* Elements in numeric arrays will be guaranteed to be retrieved via aligned access.
-* On some architectures alignment can help limit partially used cache lines.
-* 64 byte alignment is recommended by the `Intel performance guide`_ for
- data-structures over 64 bytes (which will be a common case for Arrow Arrays).
-
-Recommending padding to a multiple of 64 bytes allows for using `SIMD`_ instructions
-consistently in loops without additional conditional checks.
-This should allow for simpler, efficient and CPU cache-friendly code.
-The specific padding length was chosen because it matches the largest known
-SIMD instruction registers available as of April 2016 (Intel AVX-512). In other
-words, we can load the entire 64-byte buffer into a 512-bit wide SIMD register
-and get data-level parallelism on all the columnar values packed into the 64-byte
-buffer. Guaranteed padding can also allow certain compilers
-to generate more optimized code directly (e.g. One can safely use Intel's
-``-qopt-assume-safe-padding``).
-
-Unless otherwise noted, padded bytes do not need to have a specific value.
-
-Array lengths
--------------
-
-Array lengths are represented in the Arrow metadata as a 64-bit signed
-integer. An implementation of Arrow is considered valid even if it only
-supports lengths up to the maximum 32-bit signed integer, though. If using
-Arrow in a multi-language environment, we recommend limiting lengths to
-2 :sup:`31` - 1 elements or less. Larger data sets can be represented using
-multiple array chunks.
-
-Null count
-----------
-
-The number of null value slots is a property of the physical array and
-considered part of the data structure. The null count is represented in the
-Arrow metadata as a 64-bit signed integer, as it may be as large as the array
-length.
-
-Null bitmaps
-------------
-
-Any relative type can have null value slots, whether primitive or nested type.
-
-An array with nulls must have a contiguous memory buffer, known as the null (or
-validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
-and large enough to have at least 1 bit for each array
-slot.
-
-Whether any array slot is valid (non-null) is encoded in the respective bits of
-this bitmap. A 1 (set bit) for index ``j`` indicates that the value is not null,
-while a 0 (bit not set) indicates that it is null. Bitmaps are to be
-initialized to be all unset at allocation time (this includes padding).::
-
- is_valid[j] -> bitmap[j / 8] & (1 << (j % 8))
-
-We use `least-significant bit (LSB) numbering`_ (also known as
-bit-endianness). This means that within a group of 8 bits, we read
-right-to-left: ::
-
- values = [0, 1, null, 2, null, 3]
-
- bitmap
- j mod 8 7 6 5 4 3 2 1 0
- 0 0 1 0 1 0 1 1
-
-Arrays having a 0 null count may choose to not allocate the null
-bitmap. Implementations may choose to always allocate one anyway as a matter of
-convenience, but this should be noted when memory is being shared.
-
-Nested type arrays have their own null bitmap and null count regardless of
-the null count and null bits of their child arrays.
-
-Primitive value arrays
-----------------------
-
-A primitive value array represents a fixed-length array of values each having
-the same physical slot width typically measured in bytes, though the spec also
-provides for bit-packed types (e.g. boolean values encoded in bits).
-
-Internally, the array contains a contiguous memory buffer whose total size is
-equal to the slot width multiplied by the array length. For bit-packed types,
-the size is rounded up to the nearest byte.
-
-The associated null bitmap is contiguously allocated (as described above) but
-does not need to be adjacent in memory to the values buffer.
-
-
-Example Layout: Int32 Array
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For example a primitive array of int32s: ::
-
- [1, null, 2, 4, 8]
-
-Would look like: ::
-
- * Length: 5, Null count: 1
- * Null bitmap buffer:
-
- |Byte 0 (validity bitmap) | Bytes 1-63 |
- |-------------------------|-----------------------|
- | 00011101 | 0 (padding) |
-
- * Value Buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-63 |
- |------------|-------------|-------------|-------------|-------------|-------------|
- | 1 | unspecified | 2 | 4 | 8 | unspecified |
-
-Example Layout: Non-null int32 Array
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-``[1, 2, 3, 4, 8]`` has two possible layouts: ::
-
- * Length: 5, Null count: 0
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-----------------------|
- | 00011111 | 0 (padding) |
-
- * Value Buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | bytes 12-15 | bytes 16-19 | Bytes 20-63 |
- |------------|-------------|-------------|-------------|-------------|-------------|
- | 1 | 2 | 3 | 4 | 8 | unspecified |
-
-or with the bitmap elided: ::
-
- * Length 5, Null count: 0
- * Null bitmap buffer: Not required
- * Value Buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | bytes 12-15 | bytes 16-19 | Bytes 20-63 |
- |------------|-------------|-------------|-------------|-------------|-------------|
- | 1 | 2 | 3 | 4 | 8 | unspecified |
-
-List type
----------
-
-List is a nested type in which each array slot contains a variable-size
-sequence of values all having the same relative type (heterogeneity can be
-achieved through unions, described later).
-
-A list type is specified like ``List<T>``, where ``T`` is any relative type
-(primitive or nested).
-
-A list-array is represented by the combination of the following:
-
-* A values array, a child array of type T. T may also be a nested type.
-* An offsets buffer containing 32-bit signed integers with length equal to the
- length of the top-level array plus one. Note that this limits the size of the
- values array to 2 :sup:`31` -1.
-
-The offsets array encodes a start position in the values array, and the length
-of the value in each slot is computed using the first difference with the next
-element in the offsets array. For example, the position and length of slot j is
-computed as: ::
-
- slot_position = offsets[j]
- slot_length = offsets[j + 1] - offsets[j] // (for 0 <= j < length)
-
-The first value in the offsets array is 0, and the last element is the length
-of the values array.
-
-Example Layout: ``List<Char>`` Array
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Let's consider an example, the type ``List<Char>``, where Char is a 1-byte
-logical type.
-
-For an array of length 4 with respective values: ::
-
- [['j', 'o', 'e'], null, ['m', 'a', 'r', 'k'], []]
-
-will have the following representation: ::
-
- * Length: 4, Null count: 1
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-----------------------|
- | 00001101 | 0 (padding) |
-
- * Offsets buffer (int32)
-
- | Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-63 |
- |------------|-------------|-------------|-------------|-------------|-------------|
- | 0 | 3 | 3 | 7 | 7 | unspecified |
-
- * Values array (char array):
- * Length: 7, Null count: 0
- * Null bitmap buffer: Not required
-
- | Bytes 0-6 | Bytes 7-63 |
- |------------|-------------|
- | joemark | unspecified |
-
-Example Layout: ``List<List<byte>>``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-``[[[1, 2], [3, 4]], [[5, 6, 7], null, [8]], [[9, 10]]]``
-
-will be be represented as follows: ::
-
- * Length 3
- * Nulls count: 0
- * Null bitmap buffer: Not required
- * Offsets buffer (int32)
-
- | Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-63 |
- |------------|------------|------------|-------------|-------------|
- | 0 | 2 | 5 | 6 | unspecified |
-
- * Values array (`List<byte>`)
- * Length: 6, Null count: 1
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-------------|
- | 00110111 | 0 (padding) |
-
- * Offsets buffer (int32)
-
- | Bytes 0-27 | Bytes 28-63 |
- |----------------------|-------------|
- | 0, 2, 4, 7, 7, 8, 10 | unspecified |
-
- * Values array (bytes):
- * Length: 10, Null count: 0
- * Null bitmap buffer: Not required
-
- | Bytes 0-9 | Bytes 10-63 |
- |-------------------------------|-------------|
- | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 | unspecified |
-
-Struct type
------------
-
-A struct is a nested type parameterized by an ordered sequence of relative
-types (which can all be distinct), called its fields.
-
-Typically the fields have names, but the names and their types are part of the
-type metadata, not the physical memory layout.
-
-A struct array does not have any additional allocated physical storage for its values.
-A struct array must still have an allocated null bitmap, if it has one or more null values.
-
-Physically, a struct type has one child array for each field. The child arrays are independent and need not be adjacent to each other in memory.
-
-For example, the struct (field names shown here as strings for illustration
-purposes)::
-
- Struct <
- name: String (= List<char>),
- age: Int32
- >
-
-has two child arrays, one ``List<char>`` array (layout as above) and one 4-byte
-primitive value array having ``Int32`` logical type.
-
-Example Layout: ``Struct<List<char>, Int32>``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The layout for ``[{'joe', 1}, {null, 2}, null, {'mark', 4}]`` would be: ::
-
- * Length: 4, Null count: 1
- * Null bitmap buffer:
-
- |Byte 0 (validity bitmap) | Bytes 1-63 |
- |-------------------------|-----------------------|
- | 00001011 | 0 (padding) |
-
- * Children arrays:
- * field-0 array (`List<char>`):
- * Length: 4, Null count: 2
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-----------------------|
- | 00001001 | 0 (padding) |
-
- * Offsets buffer:
-
- | Bytes 0-19 |
- |----------------|
- | 0, 3, 3, 3, 7 |
-
- * Values array:
- * Length: 7, Null count: 0
- * Null bitmap buffer: Not required
-
- * Value buffer:
-
- | Bytes 0-6 |
- |----------------|
- | joemark |
-
- * field-1 array (int32 array):
- * Length: 4, Null count: 1
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-----------------------|
- | 00001011 | 0 (padding) |
-
- * Value Buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-63 |
- |------------|-------------|-------------|-------------|-------------|
- | 1 | 2 | unspecified | 4 | unspecified |
-
-While a struct does not have physical storage for each of its semantic slots
-(i.e. each scalar C-like struct), an entire struct slot can be set to null via
-the null bitmap. Any of the child field arrays can have null values according
-to their respective independent null bitmaps.
-This implies that for a particular struct slot the null bitmap for the struct
-array might indicate a null slot when one or more of its child arrays has a
-non-null value in their corresponding slot. When reading the struct array the
-parent null bitmap is authoritative.
-This is illustrated in the example above, the child arrays have valid entries
-for the null struct but are 'hidden' from the consumer by the parent array's
-null bitmap. However, when treated independently corresponding
-values of the children array will be non-null.
-
-Dense union type
-----------------
-
-A dense union is semantically similar to a struct, and contains an ordered
-sequence of relative types. While a struct contains multiple arrays, a union is
-semantically a single array in which each slot can have a different type.
-
-The union types may be named, but like structs this will be a matter of the
-metadata and will not affect the physical memory layout.
-
-We define two distinct union types that are optimized for different use
-cases. This first, the dense union, represents a mixed-type array with 5 bytes
-of overhead for each value. Its physical layout is as follows:
-
-* One child array for each relative type
-* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
- to each type. A union with more then 127 possible types can be modeled as a
- union of unions.
-* Offsets buffer: A buffer of signed int32 values indicating the relative offset
- into the respective child array for the type in a given slot. The respective
- offsets for each child value array must be in order / increasing.
-
-Critically, the dense union allows for minimal overhead in the ubiquitous
-union-of-structs with non-overlapping-fields use case (``Union<s1: Struct1, s2:
-Struct2, s3: Struct3, ...>``)
-
-Example Layout: Dense union
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An example layout for logical union of:
-``Union<f: float, i: int32>`` having the values:
-``[{f=1.2}, null, {f=3.4}, {i=5}]``::
-
- * Length: 4, Null count: 1
- * Null bitmap buffer:
- |Byte 0 (validity bitmap) | Bytes 1-63 |
- |-------------------------|-----------------------|
- |00001101 | 0 (padding) |
-
- * Types buffer:
-
- |Byte 0 | Byte 1 | Byte 2 | Byte 3 | Bytes 4-63 |
- |---------|-------------|----------|----------|-------------|
- | 0 | unspecified | 0 | 1 | unspecified |
-
- * Offset buffer:
-
- |Byte 0-3 | Byte 4-7 | Byte 8-11 | Byte 12-15 | Bytes 16-63 |
- |---------|-------------|-----------|------------|-------------|
- | 0 | unspecified | 1 | 0 | unspecified |
-
- * Children arrays:
- * Field-0 array (f: float):
- * Length: 2, nulls: 0
- * Null bitmap buffer: Not required
-
- * Value Buffer:
-
- | Bytes 0-7 | Bytes 8-63 |
- |-----------|-------------|
- | 1.2, 3.4 | unspecified |
-
-
- * Field-1 array (i: int32):
- * Length: 1, nulls: 0
- * Null bitmap buffer: Not required
-
- * Value Buffer:
-
- | Bytes 0-3 | Bytes 4-63 |
- |-----------|-------------|
- | 5 | unspecified |
-
-Sparse union type
------------------
-
-A sparse union has the same structure as a dense union, with the omission of
-the offsets array. In this case, the child arrays are each equal in length to
-the length of the union.
-
-While a sparse union may use significantly more space compared with a dense
-union, it has some advantages that may be desirable in certain use cases:
-
-* A sparse union is more amenable to vectorized expression evaluation in some use cases.
-* Equal-length arrays can be interpreted as a union by only defining the types array.
-
-Example layout: ``SparseUnion<u0: Int32, u1: Float, u2: List<Char>>``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For the union array: ::
-
- [{u0=5}, {u1=1.2}, {u2='joe'}, {u1=3.4}, {u0=4}, {u2='mark'}]
-
-will have the following layout: ::
-
- * Length: 6, Null count: 0
- * Null bitmap buffer: Not required
-
- * Types buffer:
-
- | Byte 0 | Byte 1 | Byte 2 | Byte 3 | Byte 4 | Byte 5 | Bytes 6-63 |
- |------------|-------------|-------------|-------------|-------------|--------------|-----------------------|
- | 0 | 1 | 2 | 1 | 0 | 2 | unspecified (padding) |
-
- * Children arrays:
-
- * u0 (Int32):
- * Length: 6, Null count: 4
- * Null bitmap buffer:
-
- |Byte 0 (validity bitmap) | Bytes 1-63 |
- |-------------------------|-----------------------|
- |00010001 | 0 (padding) |
-
- * Value buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-23 | Bytes 24-63 |
- |------------|-------------|-------------|-------------|-------------|--------------|-----------------------|
- | 5 | unspecified | unspecified | unspecified | 4 | unspecified | unspecified (padding) |
-
- * u1 (float):
- * Length: 6, Null count: 4
- * Null bitmap buffer:
-
- |Byte 0 (validity bitmap) | Bytes 1-63 |
- |-------------------------|-----------------------|
- | 00001010 | 0 (padding) |
-
- * Value buffer:
-
- |Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-23 | Bytes 24-63 |
- |-------------|-------------|-------------|-------------|-------------|--------------|-----------------------|
- | unspecified | 1.2 | unspecified | 3.4 | unspecified | unspecified | unspecified (padding) |
-
- * u2 (`List<char>`)
- * Length: 6, Null count: 4
- * Null bitmap buffer:
-
- | Byte 0 (validity bitmap) | Bytes 1-63 |
- |--------------------------|-----------------------|
- | 00100100 | 0 (padding) |
-
- * Offsets buffer (int32)
-
- | Bytes 0-3 | Bytes 4-7 | Bytes 8-11 | Bytes 12-15 | Bytes 16-19 | Bytes 20-23 | Bytes 24-27 | Bytes 28-63 |
- |------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
- | 0 | 0 | 0 | 3 | 3 | 3 | 7 | unspecified |
-
- * Values array (char array):
- * Length: 7, Null count: 0
- * Null bitmap buffer: Not required
-
- | Bytes 0-7 | Bytes 8-63 |
- |------------|-----------------------|
- | joemark | unspecified (padding) |
-
-Note that nested types in a sparse union must be internally consistent
-(e.g. see the List in the diagram), i.e. random access at any index j
-on any child array will not cause an error.
-In other words, the array for the nested type must be valid if it is
-reinterpreted as a non-nested array.
-
-Similar to structs, a particular child array may have a non-null slot
-even if the null bitmap of the parent union array indicates the slot is
-null. Additionally, a child array may have a non-null slot even if
-the types array indicates that a slot contains a different type at the index.
-
-Dictionary encoding
--------------------
-
-When a field is dictionary encoded, the values are represented by an array of
-Int32 representing the index of the value in the dictionary. The Dictionary is
-received as one or more DictionaryBatches with the id referenced by a
-dictionary attribute defined in the metadata (Message.fbs) in the Field
-table. The dictionary has the same layout as the type of the field would
-dictate. Each entry in the dictionary can be accessed by its index in the
-DictionaryBatches. When a Schema references a Dictionary id, it must send at
-least one DictionaryBatch for this id.
-
-As an example, you could have the following data: ::
-
- type: List<String>
-
- [
- ['a', 'b'],
- ['a', 'b'],
- ['a', 'b'],
- ['c', 'd', 'e'],
- ['c', 'd', 'e'],
- ['c', 'd', 'e'],
- ['c', 'd', 'e'],
- ['a', 'b']
- ]
-
-In dictionary-encoded form, this could appear as: ::
-
- data List<String> (dictionary-encoded, dictionary id i)
- indices: [0, 0, 0, 1, 1, 1, 0]
-
- dictionary i
-
- type: List<String>
-
- [
- ['a', 'b'],
- ['c', 'd', 'e'],
- ]
-
-References
-----------
-
-Apache Drill Documentation - `Value Vectors`_
-
-.. _least-significant bit (LSB) numbering: https://en.wikipedia.org/wiki/Bit_numbering
-.. _Intel performance guide: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
-.. _Endianness: https://en.wikipedia.org/wiki/Endianness
-.. _SIMD: https://software.intel.com/en-us/node/600110
-.. _Parquet: https://parquet.apache.org/documentation/latest/
-.. _Value Vectors: https://drill.apache.org/docs/value-vectors/
diff --git a/docs/latest/_sources/format/Metadata.rst.txt b/docs/latest/_sources/format/Metadata.rst.txt
deleted file mode 100644
index 293d011..0000000
--- a/docs/latest/_sources/format/Metadata.rst.txt
+++ /dev/null
@@ -1,396 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Metadata: Logical types, schemas, data headers
-==============================================
-
-This is documentation for the Arrow metadata specification, which enables
-systems to communicate the
-
-* Logical array types (which are implemented using the physical memory layouts
- specified in :doc:`Layout`)
-
-* Schemas for table-like collections of Arrow data structures
-
-* "Data headers" indicating the physical locations of memory buffers sufficient
- to reconstruct a Arrow data structures without copying memory.
-
-Canonical implementation
-------------------------
-
-We are using `Flatbuffers`_ for low-overhead reading and writing of the Arrow
-metadata. See ``Message.fbs``.
-
-Schemas
--------
-
-The ``Schema`` type describes a table-like structure consisting of any number of
-Arrow arrays, each of which can be interpreted as a column in the table. A
-schema by itself does not describe the physical structure of any particular set
-of data.
-
-A schema consists of a sequence of **fields**, which are metadata describing
-the columns. The Flatbuffers IDL for a field is: ::
-
- table Field {
- // Name is not required, in i.e. a List
- name: string;
- nullable: bool;
- type: Type;
-
- // Present only if the field is dictionary encoded
- dictionary: DictionaryEncoding;
-
- // children apply only to Nested data types like Struct, List and Union
- children: [Field];
-
- // User-defined metadata
- custom_metadata: [ KeyValue ];
- }
-
-The ``type`` is the logical type of the field. Nested types, such as List,
-Struct, and Union, have a sequence of child fields.
-
-A JSON representation of the schema is also provided:
-
-Field: ::
-
- {
- "name" : "name_of_the_field",
- "nullable" : false,
- "type" : /* Type */,
- "children" : [ /* Field */ ],
- }
-
-Type: ::
-
- {
- "name" : "null|struct|list|union|int|floatingpoint|utf8|binary|fixedsizebinary|bool|decimal|date|time|timestamp|interval"
- // fields as defined in the Flatbuffer depending on the type name
- }
-
-Union: ::
-
- {
- "name" : "union",
- "mode" : "Sparse|Dense",
- "typeIds" : [ /* integer */ ]
- }
-
-The ``typeIds`` field in the Union are the codes used to denote each type, which
-may be different from the index of the child array. This is so that the union
-type ids do not have to be enumerated from 0.
-
-Int: ::
-
- {
- "name" : "int",
- "bitWidth" : /* integer */,
- "isSigned" : /* boolean */
- }
-
-FloatingPoint: ::
-
- {
- "name" : "floatingpoint",
- "precision" : "HALF|SINGLE|DOUBLE"
- }
-
-Decimal: ::
-
- {
- "name" : "decimal",
- "precision" : /* integer */,
- "scale" : /* integer */
- }
-
-Timestamp: ::
-
- {
- "name" : "timestamp",
- "unit" : "SECOND|MILLISECOND|MICROSECOND|NANOSECOND"
- }
-
-Date: ::
-
- {
- "name" : "date",
- "unit" : "DAY|MILLISECOND"
- }
-
-Time: ::
-
- {
- "name" : "time",
- "unit" : "SECOND|MILLISECOND|MICROSECOND|NANOSECOND",
- "bitWidth": /* integer: 32 or 64 */
- }
-
-Interval: ::
-
- {
- "name" : "interval",
- "unit" : "YEAR_MONTH|DAY_TIME"
- }
-
-Schema: ::
-
- {
- "fields" : [
- /* Field */
- ]
- }
-
-Record data headers
--------------------
-
-A record batch is a collection of top-level named, equal length Arrow arrays
-(or vectors). If one of the arrays contains nested data, its child arrays are
-not required to be the same length as the top-level arrays.
-
-One can be thought of as a realization of a particular schema. The metadata
-describing a particular record batch is called a "data header". Here is the
-Flatbuffers IDL for a record batch data header: ::
-
- table RecordBatch {
- length: long;
- nodes: [FieldNode];
- buffers: [Buffer];
- }
-
-The ``RecordBatch`` metadata provides for record batches with length exceeding
-2 :sup:`31` - 1, but Arrow implementations are not required to implement support
-beyond this size.
-
-The ``nodes`` and ``buffers`` fields are produced by a depth-first traversal /
-flattening of a schema (possibly containing nested types) for a given in-memory
-data set.
-
-Buffers
-~~~~~~~
-
-A buffer is metadata describing a contiguous memory region relative to some
-virtual address space. This may include:
-
-* Shared memory, e.g. a memory-mapped file
-* An RPC message received in-memory
-* Data in a file
-
-The key form of the Buffer type is: ::
-
- struct Buffer {
- offset: long;
- length: long;
- }
-
-In the context of a record batch, each field has some number of buffers
-associated with it, which are derived from their physical memory layout.
-
-Each logical type (separate from its children, if it is a nested type) has a
-deterministic number of buffers associated with it. These will be specified in
-the logical types section.
-
-Field metadata
-~~~~~~~~~~~~~~
-
-The ``FieldNode`` values contain metadata about each level in a nested type
-hierarchy. ::
-
- struct FieldNode {
- /// The number of value slots in the Arrow array at this level of a nested
- /// tree
- length: long;
-
- /// The number of observed nulls.
- null_count: lohng;
- }
-
-The ``FieldNode`` metadata provides for fields with length exceeding 2 :sup:`31` - 1,
-but Arrow implementations are not required to implement support for large
-arrays.
-
-Flattening of nested data
--------------------------
-
-Nested types are flattened in the record batch in depth-first order. When
-visiting each field in the nested type tree, the metadata is appended to the
-top-level ``fields`` array and the buffers associated with that field (but not
-its children) are appended to the ``buffers`` array.
-
-For example, let's consider the schema ::
-
- col1: Struct<a: Int32, b: List<Int64>, c: Float64>
- col2: Utf8
-
-The flattened version of this is: ::
-
- FieldNode 0: Struct name='col1'
- FieldNode 1: Int32 name=a'
- FieldNode 2: List name='b'
- FieldNode 3: Int64 name='item' # arbitrary
- FieldNode 4: Float64 name='c'
- FieldNode 5: Utf8 name='col2'
-
-For the buffers produced, we would have the following (as described in more
-detail for each type below): ::
-
- buffer 0: field 0 validity bitmap
-
- buffer 1: field 1 validity bitmap
- buffer 2: field 1 values <int32_t*>
-
- buffer 3: field 2 validity bitmap
- buffer 4: field 2 list offsets <int32_t*>
-
- buffer 5: field 3 validity bitmap
- buffer 6: field 3 values <int64_t*>
-
- buffer 7: field 4 validity bitmap
- buffer 8: field 4 values <double*>
-
- buffer 9: field 5 validity bitmap
- buffer 10: field 5 offsets <int32_t*>
- buffer 11: field 5 data <uint8_t*>
-
-.. _spec-logical-types:
-
-Logical types
--------------
-
-A logical type consists of a type name and metadata along with an explicit
-mapping to a physical memory representation. These may fall into some different
-categories:
-
-* Types represented as fixed-width primitive arrays (for example: C-style
- integers and floating point numbers)
-* Types having equivalent memory layout to a physical nested type (e.g. strings
- use the list representation, but logically are not nested types)
-
-Integers
-~~~~~~~~
-
-In the first version of Arrow we provide the standard 8-bit through 64-bit size
-standard C integer types, both signed and unsigned:
-
-* Signed types: Int8, Int16, Int32, Int64
-* Unsigned types: UInt8, UInt16, UInt32, UInt64
-
-The IDL looks like: ::
-
- table Int {
- bitWidth: int;
- is_signed: bool;
- }
-
-The integer endianness is currently set globally at the schema level. If a
-schema is set to be little-endian, then all integer types occurring within must
-be little-endian. Integers that are part of other data representations, such as
-list offsets and union types, must have the same endianness as the entire
-record batch.
-
-Floating point numbers
-~~~~~~~~~~~~~~~~~~~~~~
-
-We provide 3 types of floating point numbers as fixed bit-width primitive array
-
-- Half precision, 16-bit width
-- Single precision, 32-bit width
-- Double precision, 64-bit width
-
-The IDL looks like: ::
-
- enum Precision:int {HALF, SINGLE, DOUBLE}
-
- table FloatingPoint {
- precision: Precision;
- }
-
-Boolean
-~~~~~~~
-
-The Boolean logical type is represented as a 1-bit wide primitive physical
-type. The bits are numbered using least-significant bit (LSB) ordering.
-
-Like other fixed bit-width primitive types, boolean data appears as 2 buffers
-in the data header (one bitmap for the validity vector and one for the values).
-
-List
-~~~~
-
-The ``List`` logical type is the logical (and identically-named) counterpart to
-the List physical type.
-
-In data header form, the list field node contains 2 buffers:
-
-* Validity bitmap
-* List offsets
-
-The buffers associated with a list's child field are handled recursively
-according to the child logical type (e.g. ``List<Utf8>`` vs. ``List<Boolean>``).
-
-Utf8 and Binary
-~~~~~~~~~~~~~~~
-
-We specify two logical types for variable length bytes:
-
-* ``Utf8`` data is Unicode values with UTF-8 encoding
-* ``Binary`` is any other variable length bytes
-
-These types both have the same memory layout as the nested type ``List<UInt8>``,
-with the constraint that the inner bytes can contain no null values. From a
-logical type perspective they are primitive, not nested types.
-
-In data header form, while ``List<UInt8>`` would appear as 2 field nodes (``List``
-and ``UInt8``) and 4 buffers (2 for each of the nodes, as per above), these types
-have a simplified representation single field node (of ``Utf8`` or ``Binary``
-logical type, which have no children) and 3 buffers:
-
-* Validity bitmap
-* List offsets
-* Byte data
-
-Decimal
-~~~~~~~
-
-Decimals are represented as a 2's complement 128-bit (16 byte) signed integer
-in little-endian byte order.
-
-Timestamp
-~~~~~~~~~
-
-All timestamps are stored as a 64-bit integer, with one of four unit
-resolutions: second, millisecond, microsecond, and nanosecond.
-
-Date
-~~~~
-
-We support two different date types:
-
-* Days since the UNIX epoch as a 32-bit integer
-* Milliseconds since the UNIX epoch as a 64-bit integer
-
-Time
-~~~~
-
-Time supports the same unit resolutions: second, millisecond, microsecond, and
-nanosecond. We represent time as the smallest integer accommodating the
-indicated unit. For second and millisecond: 32-bit, for the others 64-bit.
-
-Dictionary encoding
--------------------
-
-.. _Flatbuffers: http://github.com/google/flatbuffers
diff --git a/docs/latest/_sources/format/README.rst.txt b/docs/latest/_sources/format/README.rst.txt
deleted file mode 100644
index f2f770b..0000000
--- a/docs/latest/_sources/format/README.rst.txt
+++ /dev/null
@@ -1,53 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Arrow specification documents
-=============================
-
-Currently, the Arrow specification consists of these pieces:
-
-- Metadata specification (see :doc:`Metadata`)
-- Physical memory layout specification (see :doc:`Layout`)
-- Logical Types, Schemas, and Record Batch Metadata (see Schema.fbs)
-- Encapsulated Messages (see Message.fbs)
-- Mechanics of messaging between Arrow systems (IPC, RPC, etc.) (see :doc:`IPC`)
-- Tensor (Multi-dimensional array) Metadata (see Tensor.fbs)
-
-The metadata currently uses Google's `flatbuffers library`_ for serializing a
-couple related pieces of information:
-
-- Schemas for tables or record (row) batches. This contains the logical types,
- field names, and other metadata. Schemas do not contain any information about
- actual data.
-- *Data headers* for record (row) batches. These must correspond to a known
- schema, and enable a system to send and receive Arrow row batches in a form
- that can be precisely disassembled or reconstructed.
-
-Arrow Format Maturity and Stability
------------------------------------
-
-We have made significant progress hardening the Arrow in-memory format and
-Flatbuffer metadata since the project started in February 2016. We have
-integration tests which verify binary compatibility between the Java and C++
-implementations, for example.
-
-Major versions may still include breaking changes to the memory format or
-metadata, so it is recommended to use the same released version of all
-libraries in your applications for maximum compatibility. Data stored in the
-Arrow IPC formats should not be used for long term storage.
-
-.. _flatbuffers library: http://github.com/google/flatbuffers
diff --git a/docs/latest/_sources/index.rst.txt b/docs/latest/_sources/index.rst.txt
deleted file mode 100644
index fa6c683..0000000
--- a/docs/latest/_sources/index.rst.txt
+++ /dev/null
@@ -1,42 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Apache Arrow
-============
-
-Apache Arrow is a cross-language development platform for in-memory data. It
-specifies a standardized language-independent columnar memory format for flat
-and hierarchical data, organized for efficient analytic operations on modern
-hardware. It also provides computational libraries and zero-copy streaming
-messaging and interprocess communication.
-
-.. toctree::
- :maxdepth: 1
- :caption: Memory Format
-
- format/README
- format/Guidelines
- format/Layout
- format/Metadata
- format/IPC
-
-.. toctree::
- :maxdepth: 2
- :caption: Languages
-
- cpp/index
- python/index
diff --git a/docs/latest/_sources/python/api.rst.txt b/docs/latest/_sources/python/api.rst.txt
deleted file mode 100644
index 0bad76f..0000000
--- a/docs/latest/_sources/python/api.rst.txt
+++ /dev/null
@@ -1,399 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. currentmodule:: pyarrow
-.. _api:
-
-*************
-API Reference
-*************
-
-.. _api.types:
-
-Type and Schema Factory Functions
----------------------------------
-
-.. autosummary::
- :toctree: generated/
-
- null
- bool_
- int8
- int16
- int32
- int64
- uint8
- uint16
- uint32
- uint64
- float16
- float32
- float64
- time32
- time64
- timestamp
- date32
- date64
- binary
- string
- utf8
- decimal128
- list_
- struct
- dictionary
- field
- schema
- from_numpy_dtype
-
-.. currentmodule:: pyarrow.types
-.. _api.types.checking:
-
-Type checking functions
------------------------
-
-.. autosummary::
- :toctree: generated/
-
- is_boolean
- is_integer
- is_signed_integer
- is_unsigned_integer
- is_int8
- is_int16
- is_int32
- is_int64
- is_uint8
- is_uint16
- is_uint32
- is_uint64
- is_floating
- is_float16
- is_float32
- is_float64
- is_decimal
- is_list
- is_struct
- is_union
- is_nested
- is_temporal
- is_timestamp
- is_date
- is_date32
- is_date64
- is_time
- is_time32
- is_time64
- is_null
- is_binary
- is_unicode
- is_string
- is_fixed_size_binary
- is_map
- is_dictionary
-
-.. currentmodule:: pyarrow
-
-.. _api.value:
-
-Scalar Value Types
-------------------
-
-.. autosummary::
- :toctree: generated/
-
- NA
- Scalar
- ArrayValue
- BooleanValue
- Int8Value
- Int16Value
- Int32Value
- Int64Value
- UInt8Value
- UInt16Value
- UInt32Value
- UInt64Value
- FloatValue
- DoubleValue
- ListValue
- BinaryValue
- StringValue
- FixedSizeBinaryValue
- Date32Value
- Date64Value
- TimestampValue
- DecimalValue
-
-.. _api.array:
-
-.. currentmodule:: pyarrow
-
-Array Types
------------
-
-.. autosummary::
- :toctree: generated/
-
- array
- Array
- BooleanArray
- DictionaryArray
- FloatingPointArray
- IntegerArray
- Int8Array
- Int16Array
- Int32Array
- Int64Array
- NullArray
- NumericArray
- UInt8Array
- UInt16Array
- UInt32Array
- UInt64Array
- BinaryArray
- FixedSizeBinaryArray
- StringArray
- Time32Array
- Time64Array
- Date32Array
- Date64Array
- TimestampArray
- Decimal128Array
- ListArray
-
-.. _api.table:
-
-.. currentmodule:: pyarrow
-
-Tables and Record Batches
--------------------------
-
-.. autosummary::
- :toctree: generated/
-
- column
- chunked_array
- concat_tables
- ChunkedArray
- Column
- RecordBatch
- Table
-
-.. _api.tensor:
-
-Tensor type and Functions
--------------------------
-
-.. autosummary::
- :toctree: generated/
-
- Tensor
-
-.. _api.io:
-
-In-Memory Buffers
------------------
-
-.. autosummary::
- :toctree: generated/
-
- allocate_buffer
- compress
- decompress
- py_buffer
- foreign_buffer
- Buffer
- ResizableBuffer
-
-Input / Output and Shared Memory
---------------------------------
-
-.. autosummary::
- :toctree: generated/
-
- input_stream
- output_stream
- BufferReader
- BufferOutputStream
- FixedSizeBufferWriter
- NativeFile
- OSFile
- MemoryMappedFile
- CompressedInputStream
- CompressedOutputStream
- memory_map
- create_memory_map
- PythonFile
-
-File Systems
-------------
-
-.. autosummary::
- :toctree: generated/
-
- hdfs.connect
- LocalFileSystem
-
-.. class:: HadoopFileSystem
- :noindex:
-
-.. _api.ipc:
-
-Serialization and IPC
----------------------
-
-.. autosummary::
- :toctree: generated/
-
- ipc.open_file
- ipc.open_stream
- Message
- MessageReader
- RecordBatchFileReader
- RecordBatchFileWriter
- RecordBatchStreamReader
- RecordBatchStreamWriter
- read_message
- read_record_batch
- get_record_batch_size
- read_tensor
- write_tensor
- get_tensor_size
- serialize
- serialize_to
- deserialize
- deserialize_components
- deserialize_from
- read_serialized
- SerializedPyObject
- SerializationContext
-
-.. _api.memory_pool:
-
-Memory Pools
-------------
-
-.. currentmodule:: pyarrow
-
-.. autosummary::
- :toctree: generated/
-
- MemoryPool
- default_memory_pool
- total_allocated_bytes
- set_memory_pool
- log_memory_allocations
-
-.. _api.type_classes:
-
-.. currentmodule:: pyarrow
-
-Type Classes
-------------
-
-.. autosummary::
- :toctree: generated/
-
- DataType
- Field
- Schema
-
-.. currentmodule:: pyarrow.plasma
-
-.. _api.plasma:
-
-Plasma In-Memory Object Store
------------------------------
-
-.. autosummary::
- :toctree: generated/
-
- ObjectID
- PlasmaClient
- PlasmaBuffer
-
-.. currentmodule:: pyarrow.csv
-
-.. _api.csv:
-
-CSV Files
----------
-
-.. autosummary::
- :toctree: generated/
-
- ReadOptions
- ParseOptions
- ConvertOptions
- read_csv
-
-.. _api.feather:
-
-Feather Files
--------------
-
-.. currentmodule:: pyarrow.feather
-
-.. autosummary::
- :toctree: generated/
-
- read_feather
- write_feather
-
-.. currentmodule:: pyarrow
-
-.. _api.parquet:
-
-Parquet Files
--------------
-
-.. currentmodule:: pyarrow.parquet
-
-.. autosummary::
- :toctree: generated/
-
- ParquetDataset
- ParquetFile
- ParquetWriter
- read_table
- read_metadata
- read_pandas
- read_schema
- write_metadata
- write_table
- write_to_dataset
-
-.. currentmodule:: pyarrow
-
-Multi-Threading
----------------
-
-.. autosummary::
- :toctree: generated/
-
- cpu_count
- set_cpu_count
-
-Using with C extensions
------------------------
-
-.. autosummary::
- :toctree: generated/
-
- get_include
- get_libraries
- get_library_dirs
diff --git a/docs/latest/_sources/python/benchmarks.rst.txt b/docs/latest/_sources/python/benchmarks.rst.txt
deleted file mode 100644
index 6c3144a..0000000
--- a/docs/latest/_sources/python/benchmarks.rst.txt
+++ /dev/null
@@ -1,53 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Benchmarks
-==========
-
-The ``pyarrow`` package comes with a suite of benchmarks meant to
-run with `asv`_. You'll need to install the ``asv`` package first
-(``pip install asv`` or ``conda install -c conda-forge asv``).
-
-The benchmarks are run using `asv`_ which is also their only requirement.
-
-Running the benchmarks
-----------------------
-
-To run the benchmarks, call ``asv run --python=same``. You cannot use the
-plain ``asv run`` command at the moment as asv cannot handle python packages
-in subdirectories of a repository.
-
-Running with arbitrary revisions
---------------------------------
-
-ASV allows to store results and generate graphs of the benchmarks over
-the project's evolution. For this you have the latest development version of ASV:
-
-.. code::
-
- pip install git+https://github.com/airspeed-velocity/asv
-
-Now you should be ready to run ``asv run`` or whatever other command
-suits your needs.
-
-Compatibility
--------------
-
-We only expect the benchmarking setup to work with Python 3.6 or later,
-on a Unix-like system.
-
-.. asv:: https://asv.readthedocs.org/
diff --git a/docs/latest/_sources/python/csv.rst.txt b/docs/latest/_sources/python/csv.rst.txt
deleted file mode 100644
index 17023b1..0000000
--- a/docs/latest/_sources/python/csv.rst.txt
+++ /dev/null
@@ -1,92 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-.. currentmodule:: pyarrow.csv
-.. _csv:
-
-Reading CSV files
-=================
-
-Arrow provides preliminary support for reading data from CSV files.
-The features currently offered are the following:
-
-* multi-threaded or single-threaded reading
-* automatic decompression of input files (based on the filename extension,
- such as ``my_data.csv.gz``)
-* fetching column names from the first row in the CSV file
-* column-wise type inference and conversion to one of ``null``, ``int64``,
- ``float64``, ``timestamp[s]``, ``string`` or ``binary`` data
-* detecting various spellings of null values such as ``NaN`` or ``#N/A``
-
-Usage
------
-
-CSV reading functionality is available through the :mod:`pyarrow.csv` module.
-In many cases, you will simply call the :func:`read_csv` function
-with the file path you want to read from::
-
- >>> from pyarrow import csv
- >>> fn = 'tips.csv.gz'
- >>> table = csv.read_csv(fn)
- >>> table
- pyarrow.Table
- total_bill: double
- tip: double
- sex: string
- smoker: string
- day: string
- time: string
- size: int64
- >>> len(table)
- 244
- >>> df = table.to_pandas()
- >>> df.head()
- total_bill tip sex smoker day time size
- 0 16.99 1.01 Female No Sun Dinner 2
- 1 10.34 1.66 Male No Sun Dinner 3
- 2 21.01 3.50 Male No Sun Dinner 3
- 3 23.68 3.31 Male No Sun Dinner 2
- 4 24.59 3.61 Female No Sun Dinner 4
-
-Customized parsing
-------------------
-
-To alter the default parsing settings in case of reading CSV files with an
-unusual structure, you should create a :class:`ParseOptions` instance
-and pass it to :func:`read_csv`.
-
-Customized conversion
----------------------
-
-To alter how CSV data is converted to Arrow types and data, you should create
-a :class:`ConvertOptions` instance and pass it to :func:`read_csv`.
-
-Performance
------------
-
-Due to the structure of CSV files, one cannot expect the same levels of
-performance as when reading dedicated binary formats like
-:ref:`Parquet <Parquet>`. Nevertheless, Arrow strives to reduce the
-overhead of reading CSV files.
-
-Performance options can be controlled through the :class:`ReadOptions` class.
-Multi-threaded reading is the default for highest performance, distributing
-the workload efficiently over all available cores.
-
-.. note::
- The number of threads to use concurrently is automatically inferred by Arrow
- and can be inspected using the :func:`~pyarrow.cpu_count()` function.
diff --git a/docs/latest/_sources/python/data.rst.txt b/docs/latest/_sources/python/data.rst.txt
deleted file mode 100644
index 3260f6d..0000000
--- a/docs/latest/_sources/python/data.rst.txt
+++ /dev/null
@@ -1,434 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
... 132588 lines suppressed ...