davos

_Someone once told me that the night is dark and full of terrors. And tonight I am no knight. Tonight I am Davos the

Introduction

The davos library provides Python with an additional keyword: smuggle.

The smuggle statement works just like the built-in import statement, with two major differences:

• You can smuggle a package without installing it first
• You can smuggle a specific version of a package

Table of contents

• Table of contents
• Introduction (↑)

• Installation

• Overview

• User guide

• How It Works: The davos Parser
• Additional Notes

Why would I want an alternative to import?

In many cases, smuggle and import do the same thing—*if you're running code in the same environment you developed it in*. But what if you want to share a Jupyter notebook containing your code with someone else? If the user (i.e., the "someone else" in this example) doesn't have all of the packages your notebook imports, Python will raise an exception and the code won't run. It's not a huge deal, of course, but it's inconvenient (e.g., the user might need to pip-install the missing packages, restart their kernel, re-run the code up to the point it crashed, etc.—possibly going through this cycle multiple times until the thing finally runs).

A second (and more subtle) issue arises when the developer (i.e., the person who wrote the code) used or assumed different versions of the imported packages than what the user has installed in their environment. So maybe the original author was developing and testing their code using pandas 1.3.5, but the user hasn't upgraded their pandas installation since 0.25.0. Python will happily "import pandas" in both cases, but any changes across those versions might change what the developer's code actually does in the user's (different) environment—or cause it to fail altogether.

The problem davos tries to solve is similar to the idea motivating virtual environments, containers, and virtual machines: we want a way of replicating the original developer's environment on the user's machine, to a sufficiently good approximation that we can be "reasonably confident" that the code will continue to behave as expected.

When you smuggle packages instead of importing them, it guarantees (for whatever environment the code is running in) that the packages are importable, even if they hadn't been installed previously. Under the hood, davos figures out whether the package is available, and if not, it uses pip to download and install anything that's missing (including missing dependencies). From that point, after having automatically handled those sorts of dependency issues, smuggle behaves just like import.

The second powerful feature of davos comes from another construct, called "onion comments." These are like standard Python comments, but they appear on the same line(s) as smuggle statements, and they are formatted in a particular way. Onion comments provide a way of precisely controlling how, when, and where packages are installed, how (or if) the system checks for existing installations, and so on. A key feature is the ability to specify exactly which version(s) of each package are imported into the current workspace. When used in this way, davos enables authors to guarantee that the same versions of the packages they developed their code with will also be imported into the user's workspace at the appropriate times.

Why not use virtual environments, containers, and/or virtual machines instead?

Psst-- we'll let you in on a little secret: importing davos automatically creates a virtual environment for your notebook. However, whereas setting up a virtual environment is usually left to the user, davos handles the pesky details for you, without you needing to think about them. Any packages you smuggle via davos that aren't available in the notebook's original runtime environment are installed into a new virtual environment. This ensures that davos will not change the runtime environment (e.g., by installing new packages, changing existing package versions, etc.).

By default, each notebook's virtual environment is stored in a hidden ".davos" folder inside the current user's home directory. The default environment name is computed to uniquely identify each notebook, according to its filename and path. However, a notebook's virtual environment may be customized by setting davos.project to any string that can be used as a valid folder name in the user's operating system. This is useful for multi-notebook projects that share dependencies (without needing to duplicate each package installation for each notebook).

If you prefer, you can also disable davos's virtual environment infrastructure by setting davos.project to None. Doing so will cause any packages installed by davos to affect the notebook's runtime environment. This is generally not recommended, as it can lead to unintended consequences for other code that shares the runtime environment. That said, davos also works great when used inside of (standard) virtual environments, containers, and virtual machines.

There are a few additional specific advantages to davos that go beyond more typical virtual environments, containers, and/or virtual machines. The main advantage is that davos is very lightweight: importing davos into a notebook-based environment unlocks all of its functionality without needed to install, set up, and learn how to use additional stuff. There is none of the typical overhead of setting up a new virtual environment (or container, virtual machine, etc.), installing third-party tools, writing and sharing configuration files, and so on. All of your code and its dependencies may be contained in a single notebook file.

Okay... so how do I use this thing?

To turn a standard Jupyter (IPython) notebook, including a Google Colaboratory notebook, into a davos-enhanced notebook, just add two lines to the first cell:

%pip install davos import davos

This will enable the smuggle keyword in your notebook environment. Then you can do things like:

# pip-install numpy v1.23.1, if needed
smuggle numpy as np    # pip: numpy==1.23.1
the smuggled package is fully imported and usable
arr = np.arange(15).reshape(3, 5)
and the onion comment guarantees the desired version!
assert np.version == '1.23.1'

Interested? Curious? Intrigued? Check out the table of contents for more details! You may also want to check out our paper for more formal descriptions and explanations.

Installation

Latest Stable PyPI Release

pip install davos

Latest GitHub Update

pip install git+https://github.com/ContextLab/davos.git

Installing in Colaboratory

Note: restarting the Colab runtime does not affect installed packages. However, if the runtime is "factory reset" or disconnected due to reaching its idle timeout limit, you'll need to rerun the cell to reinstall davos on the fresh VM instance.

Overview

Smuggling Missing Packages

The smuggle statement, however, can handle missing packages on the fly. If you smuggle a package that isn't installed locally, davos will install it for you, make its contents available to Python's import machinery, and load it into the namespace for immediate use. You can control how davos installs missing packages by adding a special type of inline comment called an "onion" comment next to a smuggle statement.

Smuggling Specific Package Versions

Python doesn't provide a native, viable way to ensure a third-party package imported at runtime matches a specific version or satisfies a particular version constraint. Many packages expose their version info via a top-level version attribute (see PEP 396), and certain tools (such as the standard library's importlib.metadata and setuptools's pkg_resources) attempt to parse version info from installed distributions. However, using these to constrain imported package would require writing extra code to compare version strings and still manually installing the desired version and restarting the interpreter any time an invalid version is caught.

Additionally, for packages installed through a version control system (e.g., git), this would be insensitive to differences between revisions (e.g., commits) within the same semantic version.

davos solves these issues by allowing you to specify a specific version or set of acceptable versions for each smuggled package. To do this, simply provide a version specifier in an onion comment next to the smuggle statement:

smuggle numpy as np              # pip: numpy==1.23.1 from pandas smuggle DataFrame    # pip: pandas>=1.0,<2.0

Similarly, the second line will load the "DataFrame" object from the pandas library, analogously to "from pandas import DataFrame". A local pandas version of 1.2.1 would be used, but a local version of 2.1.1 would cause davos to replace it with a valid pandas version, as if you had manually run pip install pandas>=1.0,<2.0.

In both cases, the imported versions will fit the constraints specified in their onion comments, and the next time numpy or pandas is smuggled with the same constraints, valid local installations will be found.

You can also force the state of a smuggled packages to match a specific VCS ref (branch, revision, tag, release, etc.). For example:

smuggle hypertools as hyp    # pip: git+https://github.com/ContextLab/hypertools.git@98a3d80

And with a few exceptions, smuggling a specific package version will work _even if the package has already been imported_!

Note: davos v0.2.x supports IPython environments (e.g., Jupyter and Colaboratory notebooks) only. v0.3.x will add support for "regular" (i.e., non-interactive) Python scripts.

Use Cases

Simplify sharing reproducible code & Python environments

Instead, davos allows you to share code with one simple instruction: just pip install davos! Replace your import statements with smuggle statements, pin package versions in onion comments, and let davos take care of the rest. Beyond its simplicity, this approach ensures your predetermined package versions are in place every time your code is run.

Guarantee your code always uses the latest version, release, or revision

smuggle mypkg    # pip: mypkg --upgrade

smuggle mypkg    # pip: git+https://username/reponame.git

Compare behavior across package versions

def testmyfunc_unchanged():
    """Regression test for mypkg.my_func()"""
    data = list(range(10))
smuggle mypkg                    # pip: mypkg==0.1     result1 = mypkg.my_func(data)
smuggle mypkg                    # pip: mypkg==0.2     result2 = mypkg.my_func(data)
smuggle mypkg                    # pip: git+https://github.com/MyOrg/mypkg.git     result3 = mypkg.my_func(data)
assert result1 == result2 == result3

Usage

The smuggle Statement

Syntax

smuggle_stmt    ::=  "smuggle" module ["as" identifier] ("," module ["as" identifier])*
                     | "from" relative_module "smuggle" identifier ["as" identifier]
                     ("," identifier ["as" identifier])*
                     | "from" relative_module "smuggle" "(" identifier ["as" identifier]
                     ("," identifier ["as" identifier])* [","] ")"
                     | "from" module "smuggle" "*"
module          ::=  (identifier ".")* identifier
relative_module ::=  "."* module | "."+

In simpler terms, any valid syntax for import is also valid for smuggle.

Rules

• Like import statements, smuggle statements are whitespace-insensitive, unless a lack of whitespace between two

from os.path smuggle dirname, join as opj                       # valid
  from   os   . path   smuggle  dirname    ,join      as   opj    # also valid
  from os.path smuggle dirname, join asopj                        # invalid ("asopj" != "as opj")

• Any context that would cause an import statement not to be executed will have the same effect on a smuggle

# smuggle matplotlib.pyplot as plt           # not executed
  print('smuggle matplotlib.pyplot as plt')    # not executed
  foo = """
  smuggle matplotlib.pyplot as plt"""          # not executed

• Because the davos parser is less complex than the full Python parser, there are two fairly non-disruptive edge

exec('from pathlib import Path')         # executed
     exec('from pathlib smuggle Path')        # raises SyntaxError

if True: import random                   # executed
     if True: smuggle random                  # raises SyntaxError
while True: import math; break           # executed      while True: smuggle math; break          # raises SyntaxError
for _ in range(1): import json           # executed      for _ in range(1): smuggle json          # raises SyntaxError
# etc...

• In IPython environments (e.g., Jupyter &

# example.ipynb   import davos

def import_example():       import datetime
def smuggle_example():       smuggle datetime
import_example()   type(datetime)                               # raises NameError
smuggle_example()   type(datetime)                               # returns

The Onion Comment

• determines whether the smuggled package should be installed
• installs the smuggled package, if necessary

from sklearn.decomposition smuggle pca    # pip: scikit-learn

Syntax

onioncomment   ::=  "#" installer ":" installopt pkgspec installopt
installer       ::=  ("pip" | "conda")
pkgspec        ::=  identifier [versionspec]

where installer is the program used to install the package; install_opt is any option accepted by the installer's "install" command; and version_spec may be a version specifier defined by PEP 440 followed by a version string, or an alternative syntax valid for the given installer program. For example, pip uses specific syntaxes for local, editable, and VCS-based installation.

Less formally, an onion comment simply consists of two parts, separated by a colon:

• the name of the installer program (e.g., pip)
• arguments passed to the program's "install" command

pip install -I --no-cache-dir numpy==1.23.1 -vvv --timeout 30

smuggle numpy    # pip: -I --no-cache-dir numpy==1.23.1 -vvv --timeout 30

In practice, onion comments are identified as matches for the regular expression:

#+ (?:pip|conda) : [^#\n ].+?(?= +#| \n| *$)

Note: support for installing smuggled packages via the conda package manager will be added in v0.2. For v0.1, onion comments should always specify "pip" as the installer program.

Rules

• An onion comment must be placed on the same line as a smuggle statement; otherwise, it is not parsed:

# assuming the dateutil package is not installed...
# pip: python-dateutil                       # <-- has no effect   smuggle dateutil                             # raises InstallerError (no "dateutil" package exists)
smuggle dateutil                             # raises InstallerError (no "dateutil" package exists)   # pip: python-dateutil                       # <-- has no effect
smuggle dateutil    # pip: python-dateutil   # installs "python-dateutil" package, if necessary

• An onion comment may be followed by unrelated inline comments as long as they are separated by at least one space:

smuggle tqdm    # pip: tqdm>=4.46,<4.60 # this comment is ignored   smuggle tqdm    # pip: tqdm>=4.46,<4.60            # so is this one   smuggle tqdm    # pip: tqdm>=4.46,<4.60# but this comment raises OnionArgumentError

• An onion comment must be the first inline comment immediately following a smuggle statement; otherwise, it is not

smuggle numpy    # pip: numpy!=1.19.1        # <-- guarantees smuggled version is not v1.19.1   smuggle numpy    # has no effect -->         # pip: numpy==1.19.1

smuggle numpy    ## pip: numpy!=1.19.1       # <-- has no effect

• Onion comments are generally whitespace-insensitive, but installer arguments must be separated by at least one space:

from umap smuggle UMAP    # pip: umap-learn --user -v --no-clean     # valid   from umap smuggle UMAP#pip:umap-learn --user     -v    --no-clean    # also valid   from umap smuggle UMAP    # pip: umap-learn --user-v--no-clean       # raises OnionArgumentError

• Onion comments have no effect on standard library modules:

smuggle threading    # pip: threading==9999  # <-- has no effect

• When smuggling multiple packages with a single smuggle statement, an onion comment may be used to refer to the

smuggle nilearn, nibabel, nltools    # pip: nilearn==0.7.1

• If multiple separate smuggle statements are placed on a single line, an onion comment may be used to refer to the

smuggle gensim; smuggle spacy; smuggle nltk    # pip: nltk~=3.5 --pre

• For multiline smuggle statements, an onion comment may be placed on the first line:

from scipy.interpolate smuggle (    # pip: scipy==1.6.3       interp1d,       interpn as interp_ndgrid,       LinearNDInterpolator,       NearestNDInterpolator,   )

from scipy.interpolate smuggle (interp1d,                  # this comment has no effect                                   interpn as interp_ndgrid,                                   LinearNDInterpolator,                                   NearestNDInterpolator)     # pip: scipy==1.6.3

from scipy.interpolate smuggle (    # pip: scipy==1.6.3    # <-- this version is installed       interp1d,       interpn as interp_ndgrid,       LinearNDInterpolator,       NearestNDInterpolator,   )    # pip: scipy==1.6.2                                   # <-- this comment is ignored

from scipy.interpolate smuggle (       interp1d,                       # pip: scipy==1.6.3    # <-- ignored       interpn as interp_ndgrid,       LinearNDInterpolator,           # unrelated comment    # <-- ignored       NearestNDInterpolator   )                                   # pip: scipy==1.6.2    # <-- parsed

• The onion comment is intended to describe how a specific smuggled package should be installed if it is not found

The davos Config

Reference

import davos
  davos.configure(active=False, noninteractive=True, pip_executable='/usr/bin/pip3')

import davos
  davos.active = False
  davos.noninteractive = True
  davos.pip_executable = '/usr/bin/pip3'

smuggle numpy as np    # pip: numpy>1.16,<=1.24 -vv

smuggle(name="numpy", as="np", installer="pip", argsstr="""numpy>1.16,<=1.24 -vv""", installer_kwargs={'editable': False, 'spec': 'numpy>1.16,<=1.24', 'verbosity': 2})