Setting up Emacs Python LSP with PyEnv and stuff

There are many documents describing how to set up Python language servers for autocompletion in Emacs. The ones I read however were all missing the following features:

• Switching between multiple Python versions
• Handling Python virtual environments I have a pretty simple workflow that I am now happy with, so I figured I would share it with all of you!

Setting up the Environment

The tools I'm using:

• pyenv: manage multiple python versions
• direnv: handle project specific enviornment configuration
• venv: switching between Python virtual environments

Setting up pyenv:

There is an official autoinstaller. I sort of hate the idea because of the security risks but it's not like I am going to be checking the install script anyways:

curl https://pyenv.run | bash

For those of you who are more security conscious, follow the other installation instructions.

Now, I'm using zsh as my shell. Let's set up the pyenv configuration in the ~/.zshenv:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Note that I'm doing this in ~/.zshenv instead of ~/.zshrc because ~/.zshrc is for interactive shells. ~/.zprofile will work as well. You can read about the different roles of zsh startup files in this great stackoverflow post.

Now, if we open a new terminal and type whereis pyenv you should get something similar to what I get below:

kota@kota-ThinkPad-P1> whereis pyenv                                                                       ~
pyenv: /home/kota/.pyenv/bin/pyenv

:)

Installing direnv

Wel, direnv is also easy to install as it is prepackaged for many Linux distributions. On Ubuntu:

sudo apt install direnv

Then, we want to install the hooks into your shell. In my case, this is zsh, so in my .zshrc:

eval "$(direnv hook zsh)"

Let's also add the pyenv and venv configuration to ~/.config/direnv/direnvrc (stolen from the direnv documentation):

use_pyenv() {
  unset PYENV_VERSION
  # Because each python version is prepended to the PATH, add them in reverse order
  for ((j = $#; j >= 1; j--)); do
    local python_version=${!j}
    local pyenv_python=$(pyenv root)/versions/${python_version}/bin/python
    if [[ ! -x "$pyenv_python" ]]; then
      log_error "Error: $pyenv_python can't be executed."
      return 1
    fi

    unset PYTHONHOME
    local ve=$($pyenv_python -c "import pkgutil; print('venv' if pkgutil.find_loader('venv') else ('virtualenv' if pkgutil.find_loader('virtualenv') else ''))")

    case $ve in
      "venv")
        VIRTUAL_ENV=$(direnv_layout_dir)/python-$python_version
        export VIRTUAL_ENV
        if [[ ! -d $VIRTUAL_ENV ]]; then
          $pyenv_python -m venv "$VIRTUAL_ENV"
        fi
        PATH_add "$VIRTUAL_ENV"/bin
        ;;
      "virtualenv")
        layout_python "$pyenv_python"
        ;;
      *)
        log_error "Error: neither venv nor virtualenv are available to ${pyenv_python}."
        return 1
        ;;
    esac

    # e.g. Given "use pyenv 3.6.9 2.7.16", PYENV_VERSION becomes "3.6.9:2.7.16"
    [[ -z "$PYENV_VERSION" ]] && PYENV_VERSION=$python_version || PYENV_VERSION="${python_version}:$PYENV_VERSION"
  done

  export PYENV_VERSION
}

Setting up the Project

I created a very simple project structured as follows:

├── .envrc
├── foo
│   ├── app.py
│   ├── __init__.py
│   └── __main__.py
└── setup.py

All this application does is find the length of vector (1, 2, 3):

import numpy as np

def run():
    arr = np.array([1, 2, 3])
    print(np.linalg.norm(arr))

The key thing here is it uses an external dependency.

Let's also take a look at the setup.py:

from setuptools import setup, find_packages
from foo import __version__
setup(
    name="foo",
    version=__version__,
    packages=find_packages(exclude=["tests"]),
    author="Kota Weaver",
    install_requires=[
        'numpy'
    ],
    extras_require={
        'dev': [
            'python-language-server[all]'
        ],
        'test': [
            'pytest', 'pyflakes'
        ]
    }
)

Note that I have the Python LSP server listed in the dev dependencies.

Now let's also set up the Python development enviornment! First my .envrc:

export SIMENV_PYTHON=3.8.1

use pyenv $SIMENV_PYTHON

Now, if we go into the project directory, and do a direnv allow, we should be able to install the correct version of Python using:

pyenv install $SIMENV_PYTHON

(NOTE: you may need to install some of the following dependencies: libffi-dev libssl-dev libreadline-dev libsqlite3-dev libbz2-dev)

Then, leave the directory and enter again, and install the dependencies using:

pip install -e .['dev','test']

(depending on your shell you may need to escape the [ and ] with \)

Make sure the which python shows the correct location! This allows us to run the program using:

kota@kota-ThinkPad-P1> python -m foo                                                                   ~/foo
3.7416573867739413

Yay!

Setting up Emacs

Now we get into the meat of it all… Let's configure our Emacs to do smart things with this!

I'm using use-package, which I bootstrap if it is not installed. I have this set to be a very simple ~/.emacs.d/init.el so you can take what you want:

(setq package-archives '(("gnu" . "https://elpa.gnu.org/packages/")
                         ("marmalade" . "https://marmalade-repo.org/packages/")
                         ("melpa" . "https://melpa.org/packages/")))
(package-initialize)

(when (not (package-installed-p 'use-package)) (package-refresh-contents) (package-install 'use-package))
(require 'use-package)

; direnv mode allows automatic loading of direnv variables
(use-package direnv
  :ensure t
  :config
  (direnv-mode))

; setup Emacs path from our ~/.zshenv
(use-package exec-path-from-shell
  :ensure t
  :config
  (when (memq window-system '(mac ns x))
    (exec-path-from-shell-initialize)))

					; we also should make sure we have flycheck installed
(use-package flycheck
  :ensure t)

; Let's set up company! perhaps not necessary but this is what i like to use
(use-package company
  :ensure t
  :config
  (setq company-idle-delay 0)
  (setq company-minimum-prefix-length 1))

; install lsp mode
(use-package lsp-mode
  :ensure t
  :hook (python-mode . lsp-deferred)
  :commands (lsp lsp-deferred))

; let's add the lsp company backend
(use-package company-lsp
  :ensure t
  :config
  (push 'company-lsp company-backends))

; also installs lsp as a dependency
(use-package lsp-ui
  :ensure t
  :hook (lsp-mode . lsp-ui-mode))

And the necessary screenshot from my machine (this is my normal setup, rather than the one above, but the same functionality should be there):