Apache Arrow (Python)¶

Conda¶

To install the latest version of PyArrow from conda-forge using conda:

conda install -c conda-forge pyarrow

Pip¶

Install the latest version from PyPI:

pip install pyarrow

Installing from source¶

See Development.

Developing with conda¶

$ gcc --version

$ sudo apt-get install g++-4.9

export CC=gcc-4.9
export CXX=g++-4.9

conda create -y -q -n pyarrow-dev \
      python=3.6 numpy six setuptools cython pandas pytest \
      cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
      brotli jemalloc -c conda-forge
source activate pyarrow-dev

mkdir repos
cd repos
git clone https://github.com/apache/arrow.git
git clone https://github.com/apache/parquet-cpp.git

$ ls -l
total 8
drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 arrow/
drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 parquet-cpp/

export ARROW_BUILD_TYPE=release

export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
export PARQUET_BUILD_TOOLCHAIN=$CONDA_PREFIX
export ARROW_HOME=$CONDA_PREFIX
export PARQUET_HOME=$CONDA_PREFIX

mkdir arrow/cpp/build
pushd arrow/cpp/build

cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DARROW_PYTHON=on \
      -DARROW_BUILD_TESTS=OFF \
      ..
make -j4
make install
popd

mkdir parquet-cpp/build
pushd parquet-cpp/build

cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
      -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
      -DPARQUET_BUILD_BENCHMARKS=off \
      -DPARQUET_BUILD_EXECUTABLES=off \
      -DPARQUET_ZLIB_VENDORED=off \
      -DPARQUET_BUILD_TESTS=off \
      ..

make -j4
make install
popd

cd arrow/python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
       --with-parquet --with-jemalloc --inplace

$ py.test pyarrow
================================ test session starts ====================
platform linux -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /home/wesm/arrow-clone/python, inifile:
collected 198 items

pyarrow/tests/test_array.py ...........
pyarrow/tests/test_convert_builtin.py .....................
pyarrow/tests/test_convert_pandas.py .............................
pyarrow/tests/test_feather.py ..........................
pyarrow/tests/test_hdfs.py sssssssssssssss
pyarrow/tests/test_io.py ..................
pyarrow/tests/test_ipc.py ........
pyarrow/tests/test_jemalloc.py ss
pyarrow/tests/test_parquet.py ....................
pyarrow/tests/test_scalars.py ..........
pyarrow/tests/test_schema.py .........
pyarrow/tests/test_table.py .............
pyarrow/tests/test_tensor.py ................

====================== 181 passed, 17 skipped in 0.98 seconds ===========

Windows¶

First, make sure you can build the C++ library.

Now, we need to build and install the C++ libraries someplace.

mkdir cpp\build
cd cpp\build
set ARROW_HOME=C:\thirdparty
cmake -G "Visual Studio 14 2015 Win64" ^
      -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^
      -DCMAKE_BUILD_TYPE=Release ^
      -DARROW_BUILD_TESTS=off ^
      -DARROW_PYTHON=on ..
cmake --build . --target INSTALL --config Release
cd ..\..

After that, we must put the install directory’s bin path in our %PATH%:

set PATH=%ARROW_HOME%\bin;%PATH%

Now, we can build pyarrow:

cd python
python setup.py build_ext --inplace

set CONDA_ENV=C:\Users\wesm\Miniconda\envs\arrow-test
set PYTHONPATH=%CONDA_ENV%\Lib;%CONDA_ENV%\Lib\site-packages;%CONDA_ENV%\python35.zip;%CONDA_ENV%\DLLs;%CONDA_ENV%

DataFrames¶

The equivalent to a Pandas DataFrame in Arrow is a pyarrow.table.Table. Both consist of a set of named columns of equal length. While Pandas only supports flat columns, the Table also provides nested columns, thus it can represent more data than a DataFrame, so a full conversion is not always possible.

Conversion from a Table to a DataFrame is done by calling pyarrow.table.Table.to_pandas(). The inverse is then achieved by using pyarrow.Table.from_pandas(). This conversion routine provides the convience parameter timestamps_to_ms. Although Arrow supports timestamps of different resolutions, Pandas only supports nanosecond timestamps and most other systems (e.g. Parquet) only work on millisecond timestamps. This parameter can be used to already do the time conversion during the Pandas to Arrow conversion.

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})
# Convert from Pandas to Arrow
table = pa.Table.from_pandas(df)
# Convert back to Pandas
df_new = table.to_pandas()

Series¶

In Arrow, the most similar structure to a Pandas Series is an Array. It is a vector that contains data of the same type as linear memory. You can convert a Pandas Series to an Arrow Array using pyarrow.array.from_pandas_series(). As Arrow Arrays are always nullable, you can supply an optional mask using the mask parameter to mark all null-entries.

Type differences¶

With the current design of Pandas and Arrow, it is not possible to convert all column types unmodified. One of the main issues here is that Pandas has no support for nullable columns of arbitrary type. Also datetime64 is currently fixed to nanosecond resolution. On the other side, Arrow might be still missing support for some types.

Hadoop File System (HDFS)¶

PyArrow comes with bindings to a C++-based interface to the Hadoop File System. You connect like so:

import pyarrow as pa
hdfs = pa.HdfsClient(host, port, user=user, kerb_ticket=ticket_cache_path)

By default, pyarrow.HdfsClient uses libhdfs, a JNI-based interface to the Java Hadoop client. This library is loaded at runtime (rather than at link / library load time, since the library may not be in your LD_LIBRARY_PATH), and relies on some environment variables.

• HADOOP_HOME: the root of your installed Hadoop distribution. Often has lib/native/libhdfs.so.
• JAVA_HOME: the location of your Java SDK installation.
• ARROW_LIBHDFS_DIR (optional): explicit location of libhdfs.so if it is installed somewhere other than $HADOOP_HOME/lib/native.
• CLASSPATH: must contain the Hadoop jars. You can set these using:

export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`

You can also use libhdfs3, a thirdparty C++ library for HDFS from Pivotal Labs:

hdfs3 = pa.HdfsClient(host, port, user=user, kerb_ticket=ticket_cache_path,
                      driver='libhdfs3')

Reading Parquet¶

To read a Parquet file into Arrow memory, you can use the following code snippet. It will read the whole Parquet file into memory as an Table.

import pyarrow.parquet as pq

table = pq.read_table('<filename>')

As DataFrames stored as Parquet are often stored in multiple files, a convenience method read_multiple_files() is provided.

If you already have the Parquet available in memory or get it via non-file source, you can utilize pyarrow.io.BufferReader to read it from memory. As input to the BufferReader you can either supply a Python bytes object or a pyarrow.io.Buffer.

import pyarrow.io as paio
import pyarrow.parquet as pq

buf = ... # either bytes or paio.Buffer
reader = paio.BufferReader(buf)
table = pq.read_table(reader)

Writing Parquet¶

Given an instance of pyarrow.table.Table, the most simple way to persist it to Parquet is by using the pyarrow.parquet.write_table() method.

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.Table(..)
pq.write_table(table, '<filename>')

By default this will write the Table as a single RowGroup using DICTIONARY encoding. To increase the potential of parallelism a query engine can process a Parquet file, set the chunk_size to a fraction of the total number of rows.

If you also want to compress the columns, you can select a compression method using the compression argument. Typically, GZIP is the choice if you want to minimize size and SNAPPY for performance.

Instead of writing to a file, you can also write to Python bytes by utilizing an pyarrow.io.InMemoryOutputStream():

import pyarrow.io as paio
import pyarrow.parquet as pq

table = ...
output = paio.InMemoryOutputStream()
pq.write_table(table, output)
pybytes = output.get_result().to_pybytes()

Type and Schema Factory Functions¶

Scalar Value Types¶

Array Types and Constructors¶

Tables and Record Batches¶

Tensor type and Functions¶

Input / Output and Shared Memory¶

Interprocess Communication and Messaging¶

Memory Pools¶

Type Classes¶

Apache Parquet¶