macOS Python3 misreports Terminal width

Description

  • On TravisCI macOS images, when running Python3 via the language: Bash command, the Travis terminal misreports the size of the terminal window.
  • This can be replicated using macOS system Python3 and Conda installed versions
  • The issue cannot be replicated on a locally run macOS with the same build parameters
  • This incorrect terminal window size reporting behaviour is inconsistent across Travis’ Windows, Linux and macOS build environments

Lightweight example

When running TPUT (Unix standard for getting terminal size):

tput cols

The output is correct:

80

When running python shutil.get_terminal_size() on a Travis macOS environment, from a script called terminal_size.py:

import shutil
print(shutil.get_terminal_size())

and calling this like this:

python3 -m terminal_size

The output is incorrect:

os.terminal_size(columns=0, lines=0)

:point_up_2: this doesn’t occur on Travis CI Windows builds or Linux builds, where the output is as expected:

os.terminal_size(columns=80, lines=40)

When running on a local macOS MacBook Pro, with the same config as the Travis Mac Pipeline, the output is the same as the Travis CI Windows and Linux Builds. This confirms my understanding that the error is to do with the macOS Image Configurations on Travis.

Why this is important

This error can result in inconsistent behaviours across OS Images and can break doctest.

Consider a Sphinx Doctest example where the below is inside an index.rst file in the documentation:

.. testcode:: quick_start

    import pandas as pd

    df = pd.DataFrame({
        "column1": [1, 4, 0, 10, 9],
        "column2": [-1.3, -1.4, -2.9, -10.1, -20.4],
        "column3": ["value_1", "value_2", "value_3", "value_2", "value_1"],
    })

    print(df)

.. testoutput:: quick_start

       column1  column2  column3
    0        1     -1.3  value_1
    1        4     -1.4  value_2
    2        0     -2.9  value_3
    3       10    -10.1  value_2
    4        9    -20.4  value_1

:point_up_2:this will cause a doctest failure only on macOS only because when a terminal window is too small, pandas will only show the first and last columns, while the central column is swapped for ellipsis inside pandasrepr:

python -m sphinx -E -W -b=doctest "docs/source" "docs/_build"
Running Sphinx v2.3.0
...
Document: index
---------------
**********************************************************************
File "index.rst", line 83, in quick_start
...
print(df)
Expected:
       column1  column2  column3
    0        1     -1.3     -0.3
    1        4     -1.4      2.6
    2        0     -2.9     -2.9
    3       10    -10.1     -0.1
    4        9    -20.4    -11.4
Got:
       column1  ...  column3
    0        1  ...     -0.3
    1        4  ...      2.6
    2        0  ...     -2.9
    3       10  ...     -0.1
    4        9  ...    -11.4

i.e. the incorrect reporting of terminal size causes pandas to print incorrect ellipsis, which fails doctest. This is a false positive failure.

Detailed Examples

To help provide a further example of the problem I have provided code samples and links to Travis Pipelines.

Please note: both examples cover macOS, Linux and Windows because the point is to show the inconsistency across systems.

Detailed Example 1 - Using macOS System Python3

language: python
# ===== Linux ======
dist: xenial
python:
  - 3.5
  - 3.6
  - 3.7
matrix:
  include:
    # ======= OSX ========
    - name: "Python 3.6.5 on macOS 10.13"
      os: osx
      osx_image: xcode9.4  # Python 3.6.5 running on macOS 10.13
      language: shell       # 'language: python' is an error on Travis CI macOS
      before_install:
        - python3 --version
        - pip3 install -U pip
        - pip3 install pandas sphinx
      script: |
        echo "terminal columns reported by TPUT:"
        tput cols
        echo "terminal columns reported by python shutil get_terminal_size:"
        python3 -m terminal_size
        python3 -m sphinx -E -W -b=doctest "docs/source" "docs/_build"

You can see :point_up_2: in action here and the overall pipeline showing Linux and Windows successes is here.

Detailed Example 2 - Using macOS with conda installed

language: bash

cache:
  directories:
    - $HOME/miniconda3

before_cache:
  - rm -rf $CONDA_DIR/pkgs/cache
  - rm -rf $CONDA_DIR/envs/hosts
  - rm -rf $CONDA_DIR/conda-meta/history
  - touch $CONDA_DIR/conda-meta/history

os:
  - linux
  - osx
  - windows

env:
  - PYTHON_VERSION="3.5"
  - PYTHON_VERSION="3.6"
  - PYTHON_VERSION="3.7"

before_install:
  - |
    export CONDA_DIR="$HOME/miniconda3"
    if [ "${TRAVIS_OS_NAME}" == "windows" ]; then
        export CONDA_DIR=`cygpath -w $CONDA_DIR`
        export CONDA_BIN_DIR="$CONDA_DIR\scripts"
    else
        export CONDA_BIN_DIR="$CONDA_DIR/bin"
    fi
    export PATH="$CONDA_BIN_DIR:$PATH"

install:
  # Install Conda
  - |
    if [ "${TRAVIS_OS_NAME}" == "windows" ]; then
        choco install openssl.light
    fi

    if [ -d "$CONDA_DIR" ] && [ -e "$CONDA_BIN_DIR/conda" ]; then
        echo "Miniconda install already present from cache: $CONDA_DIR"
        rm -rf $CONDA_DIR/envs/hosts  # Just in case...
    else
        echo "Installing Miniconda..."
        rm -rf $CONDA_DIR  # Just in case...

        if [ "${TRAVIS_OS_NAME}" == "windows" ]; then
            wget https://repo.continuum.io/miniconda/Miniconda3-latest-Windows-x86_64.exe -O miniconda.exe || exit 1
            cmd //c "start /wait "" miniconda.exe /S /D=$CONDA_DIR"
        else
            if [ "${TRAVIS_OS_NAME}" == "osx" ]; then
                wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh || exit 1
            elif [ "${TRAVIS_OS_NAME}" == "linux" ]; then
                wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh || exit 1
            fi
            bash miniconda.sh -b -p "$CONDA_DIR" || exit 1
        fi
    fi

    echo "Configuring Miniconda..."
    conda config --set ssl_verify false || exit 1
    conda config --set always_yes true --set changeps1 false || exit 1

    echo "Updating Miniconda..."
    conda update conda
    conda update --all
    conda info -a || exit 1
  # Setup Conda Env
  - |
    echo "Creating a Python $PYTHON_VERSION environment..."
    conda create -n hosts python=$PYTHON_VERSION || exit 1
    source activate hosts

    conda install pandas sphinx
    echo "terminal columns reported by TPUT:"
    tput cols
    echo "terminal columns reported by python shutil get_terminal_size:"
    python -m terminal_size
    python -m sphinx -E -W -b=doctest "docs/source" "docs/_build"

Travis Example

Github Example

Conclusion

Please let me know if you need more information, or if you think there’s a mistake in the above. I’ve tested thoroughly in multiple different setups/environments and believe that the terminal width is being reported incorrectly on Travis macOS alone.

Looking at https://github.com/python/cpython/blob/master/Lib/shutil.py#L1303, a workaround is to add COLUMNS and LINES envvars.

Imprint