← Back to Blog Home

Bash Completion, Part 2: Programmable Completion

DevOps & System Admin.
by: Mike English

Article summary

Don’t miss the previous post in this series: Bash Tab Completion

With Bash’s programmable completion functionality, we can create scripts that allow us to tab-complete arguments for specific commands. We can even include logic to handle deeply nested arguments for subcommands.

Programmable completion is a feature I’ve been aware of for some time, but I only recently took the time to figure out how it works. I’ll provide some links to more in-depth treatments at the end of this post, but for now, I want to share what I learned about using these other resources.

Completion Specifications

First, let’s take a look at what “completion specifications” (or “compspecs”) we have in our shell already. This list of compspecs essentially acts as a registry of handlers that offer completion options for different starting words. We can print a list of compspecs for our current shell using complete -p. The complete built-in is also used to register new compspecs, but let’s not get ahead of ourselves.

Here’s a sampling of compspecs from my shell:

$ complete -p
complete -o nospace -F _python_argcomplete gsutil
complete -o filenames -o nospace -F _pass pass
complete -o default -o nospace -F _python_argcomplete gcloud
complete -F _opam opam
complete -o default -F _bq_completer bq
complete -F _rbenv rbenv
complete -C aws_completer aws

Here, we have some rules for completing the arguments to the following commands:

  • gsutil
  • pass
  • gcloud
  • opam
  • bq
  • rbenv
  • aws

If I type any one of those commands into my shell followed by <TAB><TAB>, these rules will be used to determine the options Bash offers for completion.

OK, so, what are we looking at? Each of the compspecs in our list starts with complete and ends with the name of the command where it will provide programmable completion. Some of the compspecs here include some -o options, and we’ll get to those later. Each of these compspecs includes either -C or -F.

Completion Commands

The compspec for aws uses -C to specify a “completion command,” which is a command somewhere in our $PATH that will output completion options.

As input, the command will receive from Bash two environment variables: COMP_LINE and COMP_POINT. These represent the current line being completed, and the point at which completion is taking place.

As output, the completion command is expected to produce a list of completion options (one per line). I won’t go into the details of this approach, but if you’re curious, you can read the source for the aws_completer command provided by Amazon’s aws-cli project.

Completion Functions

A more common approach to completion is the use of custom completion functions. Each of the compspecs containing -F registers a completion function. These are simply Bash functions that make use of environment variables to provide completion options. By convention, completion functions begin with an underscore character (_), but there’s nothing magical about the function names.

Like the completion commands, completion functions receive the COMP_LINE and COMP_POINT environment variables. However, rather than providing line-based text output, completion functions are expected to set the COMPREPLY environment variable to an array of completion options. In addition to COMP_LINE and COMP_POINT, completion functions also receive the COMP_WORDS and COMP_CWORD environment variables.

Let’s look at some of these completion functions to see how they work. We can use the Bash built-in type command to print out these function definitions (even before we know where they came from).

$ type _rbenv
_rbenv is a function
_rbenv ()
{
    COMPREPLY=();
    local word="${COMP_WORDS[COMP_CWORD]}";
    if [ "$COMP_CWORD" -eq 1 ]; then
        COMPREPLY=($(compgen -W "$(rbenv commands)" -- "$word"));
    else
        local words=("${COMP_WORDS[@]}");
        unset words[0];
        unset words[$COMP_CWORD];
        local completions=$(rbenv completions "${words[@]}");
        COMPREPLY=($(compgen -W "$completions" -- "$word"));
    fi
}

This example demonstrates a few common patterns. We see that COMP_CWORD can be used to index into COMP_WORDS to get the current word being completed. We also see that COMPREPLY can be set in one of two ways, both using some external helpers and a built-in command we haven’t seen yet: compgen. Let’s run through some possible input to see how this might work.

If we type:

$ rbenv h<TAB><TAB>

We’ll see:

$ rbenv h
help hooks

In this case, COMPREPLY comes from the first branch of (COMP_CWORD is 1). The local variable word is set to h, and this is passed to compgen along with a list of possible commands generated by rbenv commands. The compgen built-in returns only those options from a given wordlist (-W) that start with the current word of the user’s input, $word. We can perform similar filtering with grep:

$ rbenv commands | grep '^h'
help
hooks

The second branch provides completion options for subcommands. Let’s walk through another example:

$ rbenv hooks <TAB><TAB>

Will give us:

$ rbenv hooks
exec    rehash  which

Each of these options simply comes from rbenv completions:

$ rbenv completions hooks
exec
rehash
which

And since we haven’t provided another word yet, compgen is filtering with an empty string, analogous to:

$ rbenv completions hooks | grep '^'
exec
rehash
which

If we instead provide the start of a word, we’ll have it completed for us:

$ rbenv hooks e<TAB>

Will give us:

$ rbenv hooks exec

In this case, our compgen invocation might be something like:

$ compgen -W "$(rbenv completions hooks)" -- "e"
exec

Or we can imagine with grep:

$ rbenv completions hooks | grep '^e'
exec

With just a single result in COMPREPLY, readline is happy to complete the rest of the word exec for us.

Registering Custom Completion Functions

Now that we know what it’s doing, let’s use Bash’s extended debugging option to find out where this _rbenv function came from:

$ shopt -s extdebug && declare -F _rbenv && shopt -u extdebug
_rbenv 1 /usr/local/Cellar/rbenv/0.4.0/libexec/../completions/rbenv.bash

If we look in this rbenv.bash file, we’ll see:

$ cat /usr/local/Cellar/rbenv/0.4.0/libexec/../completions/rbenv.bash
_rbenv() {
  COMPREPLY=()
  local word="${COMP_WORDS[COMP_CWORD]}"

  if [ "$COMP_CWORD" -eq 1 ]; then
    COMPREPLY=( $(compgen -W "$(rbenv commands)" -- "$word") )
  else
    local words=("${COMP_WORDS[@]}")
    unset words[0]
    unset words[$COMP_CWORD]
    local completions=$(rbenv completions "${words[@]}")
    COMPREPLY=( $(compgen -W "$completions" -- "$word") )
  fi
}

complete -F _rbenv rbenv

We’ve already seen all of this! This file simply declares a new function and then registers a corresponding completion specification using complete. For this completion to be available, this file only needs to be sourced at some point. I haven’t dug into how rbenv does it, but I suspect that something in the eval "$(rbenv init -)" line included in our Bash profile ends up sourcing that completion script.

Parting Thoughts

Readline

The unsung hero of Bash’s programmable completion is really the readline library. This library is responsible for turning your <TAB> key-presses into calls to compspecs, as well as displaying or completing the resulting options those compspecs provide.

Some functionality of the readline library is configurable. One interesting option that can be set tells readline to immediately display ambiguous options after just one <TAB> key-press instead of two. With this option set, our above examples would look a little different. For example:

$ rbenv h<TAB><TAB>
help hooks

would only need to be:

$ rbenv h<TAB>
help hooks

If this sounds appealing, put the following in your ~/.inputrc:

set show-all-if-ambiguous on

To find out about other readline variables we could set in our ~/.inputrc (and to see their current values), we can use the Bash built-in command bind, with a -v flag.

$ bind -v
set bind-tty-special-chars on
set blink-matching-paren on
set byte-oriented off
set completion-ignore-case off
set convert-meta off
set disable-completion off
set enable-keypad off
set expand-tilde off
set history-preserve-point off
set horizontal-scroll-mode off
set input-meta on
set mark-directories on
set mark-modified-lines off
set mark-symlinked-directories off
set match-hidden-files on
set meta-flag on
set output-meta on
set page-completions on
set prefer-visible-bell on
set print-completions-horizontally off
set show-all-if-ambiguous off
set show-all-if-unmodified off
set visible-stats off
set bell-style audible
set comment-begin #
set completion-query-items 100
set editing-mode emacs
set keymap emacs

For more information, consult the relevant Bash info page node:

$ info -n '(bash)Readline Init File Syntax'

More on Completion

Larger completion scripts often contain multiple compspecs and several helpers. One convention I’ve seen several times is to name the helper functions with two leading underscores. If you find you need to write a large amount of completion logic in Bash, these conventions may be helpful to follow. As we’ve already seen, it’s also possible to handle some, most, or even all of the completion logic in other languages using external commands.

There is a package available from Homebrew called bash-completion that contains a great number of completion scripts for common commands. After installation, it also prompts the user to configure their Bash profile to source all of these scripts. They all live in a bash-completions.d directory under $(brew --prefix)/etc and can be good reading. A similar package should also be available for Linux (and probably originated there).

Speaking of similar features for different platforms, I should also mention that while this post focuses specifically on the programmable completion feature of the Bash shell, other shells have similar functionality. If you’re interested in learning about completion for zsh or fish, please see the links at the end of this post.

Further Reading

This is only the tip of the iceberg of what’s possible with Bash programmable completion. I hope that walking through a couple of examples has helped demystify what happens when tab completion magically provides custom options to commands. For further reading, see the links below.

Related Posts