Using Emacs As A Python Ide with basic Emacs features

Overview

This is a collection of modes, recipes and keybindings useful when working with Python projects in Emacs.

The Major mode for editing Python files in Emacs is python-mode.

Every buffer possesses a major mode, which determines the editing behavior of Emacs while that buffer is current

Packages tree:

• Fundamental mode
  • Major mode: prog-mode
    • Minor mode: python-mode
  • comint-mode
    • Python shell comint buffer. (Inferior mode)

A programming language[^progmode] mode typically specifies the: - syntax of expressions, - the customary rules for indentation, - how to do syntax highlighting for the language, and - how to find the beginning or end of a function definition. - features for compiling and - debugging programs

We will see how to do each of them efficiently in Emacs.

Major mode - Programming mode

This major mode is mostly intended as a parent of other programming modes. All major modes for programming languages should derive from this mode so that users can put generic customization should be on prog-mode-hook.

The prog-mode-hook is the “normal hook run when entering programming modes.”

Useful functions and keybinds:

• C-M-q prog-indent-sexp: Indent the expression after point.

Minor modes

Python mode

python-mode is part of Emacs.

Major mode for editing Python files with some fontification and indentation bits extracted from original Dave Love’s python.el found in GNU Emacs.

Basic functions:

• C-c Prefix Command
• ESC Prefix Command
• DEL python-indent-dedent-line-backspace
• <backtab> python-indent-dedent-line

Implements:

Editing

• C-c C-d python-describe-at-point

• C-c C-f python-eldoc-at-point

• C-c C-j imenu

• C-c C-t Prefix Command

Syntax highlighting

Fontification of code is provided and supports python’s triple quoted strings properly.

Indentation

Automatic indentation with indentation cycling is provided

• TAB: navigate different available levels of indentation by hitting TAB several times.

Movement

Navigating through code.

• C-M-<home>, C-M-a, ESC C-<home>. beginning-of-defun: Move backward to the beginning of a defun.
• C-M-<end>, C-M-e, ESC C-<end>. end-of-defun: Move forward to next end of defun.
• M-a, <remap> <backward-sentence> python-nav-backward-block: Move backward to start of sentence.
• M-e, <remap> <forward-sentence> python-nav-forward-block: Move forward to next end of sentence.
  • The variable ‘sentence-end’ is a regular expression that matches ends of sentences. Also, every paragraph boundary terminates sentences as well.
• C-M-<up>, C-M-u, ESC C-<up>.<remap> <backward-up-list>: python-nav-backward-up-list
• C-M-h <remap> <mark-defun> python-mark-defun: Put mark at end of this defun, point at beginning.

Extra functions without key bounds:

python-nav-forward-statement python-nav-backward-statement python-nav-beginning-of-statement python-nav-end-of-statement python-nav-beginning-of-block python-nav-end-of-block python-nav-if-name-main

Jump through code

Xref find definition

Jump to the definition of the function where the cursor is located.

• M-. xref-find-definitions: Find the definition of the identifier at point.
  • Use M-, to return back to where you invoked this command.
  • xref.el package

For this to work we need a TAGS table.

Generate TAGS table

Create a TAGS file that include Python’s library definitions, your current project identifiers and site packages definitions.

For example to create them in a Makefile:

PYTHONPATH:=/usr/lib/python3.9/
VIRTUALENV_PATH:=~/.virtualenvs/myproject
SITEPACKAGES:=$(VIRTUALENV_PATH)/lib/python3.6/site-packages/
tags:
	rm -f TAGS
	find $(PYTHONPATH) -type f -name '*.py' -print0 | xargs -0 etags --append=yes --output-format=etags
	find . -type f -name '*.py' -print0 | xargs -0 etags --append=yes --output-format=etags
	find $(SITEPACKAGES) -type f -name '*.py' -print0 | xargs -0 etags --append=yes --output-format=etags

Then use xref to jump to function definitions with ALT+. and select the above TAGS file as the tags table, or set it manually with: M-x visit-tags-table which sets the global value of tags-file-name.

• visit-tags-table: Tell tags commands to use tags table file FILE.
  • FILE should be the name of a file created with the etags program.
    • A directory name is ok too; it means file TAGS in that directory.
    • Currently active ctags implementation: Universal ctags: sudo apt install universal-ctags
    • Should be used with --output-format=etags to enable etags mode, which will create a tag file for use with the Emacs editor.

Normally M-x visit-tags-table sets the global value of ‘tags-file-name’.

FFAP: Find File At Point.

For finding the filename for a given module ffap can be used, it needs an inferior python shell running.

• ffap: Find FILENAME, guessing a default from text around point.
  • e.g.: if cursor over import os, running the above function would prompt to open the file: /usr/lib/python3.8/os.py.

Shell

When editing a buffer of Python code, a shell should be started to interact with the code:

• C-c C-p run-python: Run an inferior Python process.
  • Runs the hook inferior-python-mode-hook after comint-mode-hook is run. (Type h in the process buffer for a list of commands.)
• C-c C-z python-shell-switch-to-shell

COMINT

Command interpreter (comint) in a window, defines a general command-interpreter-in-a-buffer package (comint mode).

Ppecific process-in-a-buffer modes can be built on top of comint mode – e.g., shell, Python shell

All these specific packages share a common base functionality, and a common set of bindings, which makes them easier to use (and saves code, implementation time, etc., etc.).

Keybindings from lisp/comint.el:

;; Comint Mode Commands: (common to all derived modes, like shell & cmulisp
;; mode)
;;
;; M-p	   comint-previous-input	   Cycle backwards in input history
;; M-n	   comint-next-input		   Cycle forwards
;; M-r     comint-history-isearch-backward-regexp  Isearch input regexp backward
;; M-C-l   comint-show-output		   Show last batch of process output
;; RET	   comint-send-input
;; C-d	   comint-delchar-or-maybe-eof     Delete char unless at end of buff
;; C-c C-a comint-bol-or-process-mark      First time, move point to bol;
;;					    second time, move to process-mark.
;; C-c C-u comint-kill-input		    ^u
;; C-c C-w backward-kill-word		    ^w
;; C-c C-c comint-interrupt-subjob	    ^c
;; C-c C-z comint-stop-subjob		    ^z
;; C-c C-\ comint-quit-subjob		    ^\
;; C-c C-o comint-delete-output		    Delete last batch of process output
;; C-c C-r comint-show-output		    Show last batch of process output
;; C-c C-l comint-dynamic-list-input-ring  List input history
;;
;; Not bound by default in comint-mode (some are in shell mode)
;; comint-run				Run a program under comint-mode
;; comint-send-invisible		Read a line w/o echo, and send to proc
;; comint-dynamic-complete-filename	Complete filename at point.
;; comint-dynamic-list-filename-completions List completions in help buffer.
;; comint-replace-by-expanded-filename	Expand and complete filename at point;
;;					replace with expanded/completed name.
;; comint-replace-by-expanded-history	Expand history at point;
;;					replace with expanded name.
;; comint-magic-space                  Expand history and add (a) space(s).
;; comint-kill-subjob			No mercy.
;; comint-show-maximum-output          Show as much output as possible.
;; comint-continue-subjob		Send CONT signal to buffer's process
;;					group.  Useful if you accidentally
;;					suspend your process (with C-c C-z).
;; comint-get-next-from-history        Fetch successive input history lines
;; comint-accumulate		       Combine lines to send them together
;;					as input.
;; comint-goto-process-mark	       Move point to where process-mark is.
;; comint-set-process-mark	       Set process-mark to point.

;; comint-mode-hook is the Comint mode hook.  Basically for your keybindings.

Django Shell

For having a shell running in a Django project manage.py shell, before running it set the following parameters:

(setq python-shell-interpreter "python"
      python-shell-interpreter-args "-i /path/to/manage.py shell")

Shell interaction

• C-M-x python-shell-send-defun: Send the current defun to inferior Python process.
• C-c C-c python-shell-send-buffer: Send the entire buffer to inferior Python process.
• C-c C-e python-shell-send-statement: Send the statement at point to inferior Python process.
  • The statement is delimited by python-nav-beginning-of-statement and python-nav-end-of-statement, but if the region is active, the text in the region is sent instead via python-shell-send-region.
• C-c C-l python-shell-send-file: Send FILE-NAME to inferior Python PROCESS.
• C-c C-r python-shell-send-region: Send the region delimited by START and END to inferior Python process.
• C-c C-s python-shell-send-string: Send STRING to inferior Python process.

It allows opening Python shells inside Emacs and executing any block of code of your current buffer in that inferior Python process.

Besides that only the standard CPython (2.x and 3.x) shell and IPython are officially supported out of the box, the interaction should support any other readline based Python shells as well (e.g. Jython and PyPy have been reported to work). You can change your default interpreter and commandline arguments by setting the python-shell-interpreter and python-shell-interpreter-args variables. This example enables IPython globally:

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "-i")

The interaction relies upon having prompts for input (e.g. “»> " and “… " in standard Python shell) and output (e.g. “Out[1]: " in IPython) detected properly. Failing that Emacs may hang but, in the case that happens, you can recover with \[keyboard-quit]. To avoid this issue, a two-step prompt autodetection mechanism is provided: the first step is manual and consists of a collection of regular expressions matching common prompts for Python shells stored in python-shell-prompt-input-regexps and python-shell-prompt-output-regexps, and dir-local friendly vars python-shell-prompt-regexp, python-shell-prompt-block-regexp, python-shell-prompt-output-regexp which are appended to the former automatically when a shell spawns; the second step is automatic and depends on the python-shell-prompt-detect helper function.

Shell completion

• C-M-i completion-at-point: Perform completion on the text around point.

  • lisp/minibuffer.el
  • “Function for completion-at-point-functions in python-mode. For this to work as best as possible you should call python-shell-send-buffer from time to time so context in inferior Python process is updated properly.”
  • Completion works for the functions and variables of the current buffer, so if it the buffer is not sent before, it won’t be able to display any suggestions.

• In an inferior Python shell, Hitting TAB will try to complete the current word.

Shell virtualenv support:

The shell also contains support for virtualenvs and other special environment modifications thanks to python-shell-process-environment and python-shell-exec-path. These two variables allows you to modify execution paths and environment variables to make easy for you to setup virtualenv rules or behavior modifications when running shells.

The variable python-shell-virtualenv-root is provided to set with the path of the virtualenv to use, then process-environment and exec-path get proper values in order to run shells inside the specified virtualenv.

(setq python-shell-virtualenv-root "/path/to/env/")

Also the python-shell-extra-pythonpaths variable have been introduced as simple way of adding paths to the PYTHONPATH without affecting existing values.

Shell package support:

you can enable a package in the current shell so that relative imports work properly using the python-shell-package-enable command.

Shell remote support:

remote Python shells are started with the correct environment for files opened remotely through tramp, also respecting dir-local variables provided enable-remote-dir-locals is non-nil. The logic for this is transparently handled by the python-shell-with-environment macro.

Shell syntax highlighting:

when enabled current input in shell is highlighted. The variable python-shell-font-lock-enable controls activation of this feature globally when shells are started. Activation/deactivation can be also controlled on the fly via the python-shell-font-lock-toggle command.

Completion

Incremental looking for all definitions with helm-tags.el.

• C-x c e helm-etags-select: Preconfigured helm for etags.
  • if any of the tag files have been modified, reinitialize cache.
  • This function aggregates three sources of tag files:
    • 1. An automatically located file in the parent directories, by helm-etags-get-tag-file.
    • 2. tags-file-name, which is commonly set by find-tag command.
    • 3. tags-table-list which is commonly set by visit-tags-table command.

Pdb tracking

when you execute a block of code that contains some call to pdb (or ipdb) it will prompt the block of code and will follow the execution of pdb marking the current line with an arrow.

import pdb; pdb.set_trace()

Symbol completion

Skeletons

skeletons are provided for simple inserting of things like class, def, for, import, if, try, and while.

These skeletons are integrated with abbrev.

If you have abbrev-mode activated and python-skeleton-autoinsert is set to t, then whenever you type the name of any of those defined and hit SPC, they will be automatically expanded. As an alternative you can use the defined skeleton commands: python-skeleton-<foo>.

Yasnippets

Code completion with yasnippet-snippets.

• yas-describe-tables: Display snippets for each table.
  • If being in python-mode shows all Python related snippets
• company-yasnippet: incrementally type snippet names for the current mode

Company

By default — having company-mode enabled (see Initial Setup) — a tooltip with completion candidates is shown when a user types in a few characters.

• C-? company-complete: To initiate completion manually, use the command.

To select next or previous of the shown completion candidates, use respectively key bindings C-n and C-p, then do one of the following:

Hit RET to choose a selected candidate for completion. Hit TAB to complete with the common part: characters present at the beginning of all the candidates. Hit C-g to stop activity of Company.

;; Company autocomplete
;; http://company-mode.github.io/
;; Completion will start automatically after you type a few letters.
;; Use M-n and M-p to select, <return> to complete or <tab> to complete
;; the common part. Search through the completions with C-s, C-r and C-o.
;; Press M-(digit) to quickly complete with one of the first 10 candidates.
;; Type M-x company-complete to initiate completion manually
;; Third party packages
;; https://github.com/company-mode/company-mode/wiki/Third-Party-Packages
(use-package company
  :bind (("C-?" . company-complete))
  :config
  (global-company-mode t)
  ;(add-to-list 'company-backends '(company-anaconda :with company-capf))
  (add-to-list 'company-backends 'company-emoji)
  (add-to-list 'company-backends 'company-ghc)
  (add-to-list 'company-backends 'company-ghci)
  (add-to-list 'company-backends 'company-go)
  (add-to-list 'company-backends 'company-ac-php-backend) ;; provided by company-php
  (add-to-list 'company-backends '(company-shell company-fish-shell))
  (add-to-list 'company-backends 'company-web-html))

Code Check

Check the current file for errors with python-check using the program defined in python-check-command.

By default, python-check will try to execute ``pyflakesand if not found,epylint(pylint), but it can also be used any custom checker with settingpython-check-command`.

So install pylint or pylint-django before using this command.

• C-c C-v python-check: Check a Python file (default current buffer’s file).
  • Compile the program including the current buffer.

For Django

From pylin-django usage instructions:

Ensure pylint-django is installed and on your path. In order to access some of the internal Django features to improve pylint inspections, you should also provide a Django settings module appropriate to your project. This can be done either with an environment variable:

DJANGO_SETTINGS_MODULE=your.app.settings pylint --load-plugins pylint_django [..other options..] <path_to_your_sources>

Alternatively, this can be passed in as a commandline flag:

pylint --load-plugins pylint_django --django-settings-module=your.app.settings [..other options..] <path_to_your_sources>

Flycheck

On-the-fly syntax checking for Emacs.

Install and enabling it globally.

(use-package flycheck
  :ensure t
  :init
  (global-flycheck-mode)
  (setq flycheck-python-pylint-executable "/home/marcanuy/.virtualenvs/wikigod/bin/pylint")
  ;;(setq flycheck-python-flake8-executable "python3")
  :config
  (add-hook 'after-init-hook #'global-flycheck-mode))

Install syntax checker programs

pip install pylint

Check setup in a Python buffer:

• C-c ! v flycheck-verify-setup

To use pylint from the virtual environment, override the executable with flycheck-python-pylint-executable.

(setq flycheck-python-pylint-executable "/home/marcanuy/.virtualenvs/project/bin/pylint")
(setq python-check-command "/home/marcanuy/.virtualenvs/project/bin/pylint")

Navigate and list errors

• C-c ! n and C-c ! p jump back and forth between erroneous places.

  • If you keep on such a place for a little while Flycheck will show the corresponding error message in the each area. Likewise, if you hover such a place with the mouse cursor Flycheck will show the error message in a tooltip.

• C-c ! l to pop up a list of all errors in the current buffer.

  • This list automatically updates itself when you fix errors or introduce new ones, and follows the currently selected buffer. If the error list is selected you can type n and p to move up and down between errors and jump to their corresponding location in the buffer.

Code format

Reformat python buffers using the “black” formatter.

• blacken uses black. Needs to have black installed pip install black

• blacken-mode: Automatically run black before saving.

The whole buffer can be reformatted with blacken-buffer. If you want to format every time you save, enable blacken-mode in relevant python buffers. Note that if blacken-only-if-project-is-blackened is non-nil, then blacken will only run if your pyproject.toml contains the [tool.black] setting. This setting is off by default.

To automatically format all Python buffers before saving, add the function blacken-mode to python-mode-hook:

(add-hook 'python-mode-hook 'blacken-mode)

To choose the right black executable:

(set-variable 'blacken-executable "/home/user/.virtualenvs/project/bin/black")

ElDoc:

returns documentation for object at point by using the inferior python subprocess to inspect its documentation. As you might guessed you should run python-shell-send-buffer from time to time to get better results too.

menu:

• F10 menu-bar-open: Start key navigation of the menu bar in FRAME.

modes:

• flyspell-prog-mode
  • Turn on flyspell-mode for comments and strings.
• abbrev-mode
  • In Abbrev mode, inserting an abbreviation causes it to expand and be replaced by its expansion.
  • list-abbrevs

• display-line-numbers-mode
  • Toggle display of line numbers in the buffer.
• prettify-symbols-mode
  • When Prettify Symbols mode and font-locking are enabled, symbols are prettified (displayed as composed characters)

Syntax highlighting:

Fontification of code is provided and supports python’s triple quoted strings properly.

Common workflow

Set up

Python process

Set the current virtual environment:

(setq python-shell-virtualenv-root "/home/user/.virtualenvs/project/")

Run an inferior Python process to send buffers and defuns.

(run-python)

Testing

Tests can be executed through the compile command or the shell directly. This has the advantage of being able to browse errors directly from Emacs with the errors output buffer.

• compile:

  • Output buffer works for M-x compile and M-x grep commands. In the test results output buffer:

    • C-x </kbd>, <kbd>M-g n</kbd>, <kbd>M-g M-n</kbd> next-error`: Visit next ‘next-error’ message and corresponding source code.
      • lisp/simple.el
    • M-g p, M-g M-p, previous-error: Visit previous ‘next-error’ message and corresponding source code.

  • recompile:

    • g in the errors buffer.
    • M-x recompile to rerun the same test again.

  • M-n compilation-next-error

  • M-p compilation-previous-error

  • M-{ compilation-previous-file

  • M-} compilation-next-file

  • C-c C-c compile-goto-error

  • C-c C-f next-error-follow-minor-mode

  • C-c C-k kill-compilation

• shell: When running tests from a common shell in Emacs, if Compilation Shell minor mode is enabled, all the error-parsing commands of the Compilation major mode are available but bound to keys that don’t collide with Shell mode.

(add-hook 'shell-mode-hook 'compilation-shell-minor-mode)

Advanced testing

More powerful testing using pytest which also support Django, etc: https://docs.pytest.org/en/7.1.x/

• python-pytest https://github.com/wbolster/emacs-python-pytest helpers to run pytest

To install:

(use-package python-pytest)

To test:

M-x python-pytest-dispatch

Version Control

Using git with Magit.

C-x g magit-status: When invoked from within an existing Git repository, then this command shows the status of that repository in a buffer.

Django

• Django autocompletion in Shell https://simpleit.rocks/python/django/add-bash-manage-autocompletion-to-command-tab-in-linux/

References

• Emacs documentation: https://www.gnu.org/software/emacs/manual/html_node/emacs/
  • Major modes https://www.gnu.org/software/emacs/manual/html_node/emacs/Major-Modes.html