04 September 2018 at 20:15 (updated 03 April 2021 at 13:00)
- Python
- best practices
- devel
- guide
- Python
- venv
- virtual environments
- virtualenv
venv
A guide about Python virtual environments for any platform
In Python, virtual environments are used to isolate projects from each other(if they require different versions of the same library, for example). They letyou install and manage packages without administrative privileges, and withoutconflicting with the system package manager. They also allow to quickly createan environment somewhere else with the same dependencies.
Virtual environments are a crucial tool for any Python developer. And at that,a very simple tool to work with.
Let’s get started!
Install
The best tool that can be used to create virtual environments is thevenv module, which is part ofthe standard library since Python 3.3.
venv is built into Python, and most users don’t need to install anything.However, Debian/Ubuntu users will need to run sudo apt-get install python3-venv to make it work (due to Debian not installing some componentsthat venv needs by default). [1]
The alternative (and original, and previously standard) virtual environment tool is virtualenv. It works with Python 2.7, and has a coupleextra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or pip install --user virtualenv.
Which one to use? Probably venv. Both tools achieve the same goal in similarways. And if one of them does not work, you can try the other and it might justwork better.
(Terminology note: most of the time, the names of both tools are usedinterchargeably, “venv” was often used as an abbreviation for “virtualenv”before the stdlib tool was created)
Create
To create a virtual environment named env, you need to run the venvtool with the Python you want to use in that environment.
Linux: $ python3 -m venv envWindows: > py -m venv env
or, if you’re using virtualenv:
$ python3 -m virtualenv env> py -m virtualenv env
Afterwards, you will end up with a folder named env that contains foldersnamed bin (Scripts on Windows — contains executables and scriptsinstalled by packages, includingpython), lib (contains code), and include (contains C headers).
Both tools install pip and setuptools, but venv does not ship withwheel. In addition, the default versions tend to be more-or-less outdated.Let’s upgrade them real quick: [2]
$ env/bin/python -m pip install --upgrade pip setuptools wheel> env\Scripts\python -m pip install --upgrade pip setuptools wheel
Where to store virtual environments?
While the tools allow you to put your virtual environments anywhere in thesystem, it is not a desirable thing to do. There are two options:
Have one global place for them, like
~/virtualenvs.Store them in each project’s directory, like
~/git/foobar/.venv.
The first option can be easier to manage, there are tools that can help managethose (eg. virtualenvwrapper, shell auto-activation scripts, or theworkon functions described below). The second option is equally easy towork with, but comes with one caveat — you must add the venv directory to your.gitignore file (or .git/info/exclude if you don’t want to commitchanges to .gitignore), since you don’t want it in your repository (it’sbinary bloat, and works only on your machine).
If you pick the global virtual environment store option, you can use the following shortfunction (put it in .bashrc / .zshrc / your shell configuration file)to get a simple way to activate an environment (by running workon foo).virtualenvwrapper also has a workon feature, although I don’t thinkvirtualenvwrapper is really necessary and too helpful — the workonfeature is handy though, and so here’s a way to do it withoutvirtualenvwrapper:
And for PowerShell fans, here’s a workon.ps1 script:
And for cmd.exe fans… you should switch to PowerShell, it’s a very nice andfriendly shell (though perhaps requiring some effort to learn how to beproductive with it).
Use
There are three ways of working with virtual environments interactively (in ashell):
activation (run
source env/bin/activateon *nix;env\Scripts\activateon Windows) — it simplifies work and requires lesstyping, although it can sometimes fail to work properly. (After installingscripts,hash -rmay be necessary on *nix to use them.)executing
env/bin/python(env\Scripts\python) and other scripts directly, asactivation only changes$PATHand some helper variables — those variablesare not mandatory for operation, running the correctpythonis, and thatmethod is failsafe.in subshells (IMO, it’s bad UX)
Whichever method you use, you must remember that without doing any of thesethings, you will still be working with the system Python.
For non-interactive work (eg. crontab entries, system services, etc.),activation and subshells are not viable solutions. In these cases, you mustalways use the full path to Python.
Here are some usage examples (paths can be relative, of course):
## *nix, activation ##$ source /path/to/env/bin/activate(env)$ pip install Django(env)$ deactivate## *nix, manual execution ##$ /path/to/env/bin/pip install Django## Windows, activation ##> C:\path\to\env\Scripts\activate(venv)> pip install Django(venv)> deactivate## Windows, manual execution ##> C:\path\to\env\Scripts\pip install Django## Windows, updating pip/setuptools/wheel ##> C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel
The same principle applies to running Python itself, or any other scriptinstalled by a package. (With Django’s manage.py, calling it as./manage.py requires activation, or you can runvenv/bin/python manage.py.)
Moving/renaming/copying environments?
If you try to copy or rename a virtual environment, you will discover that thecopied environment does not work. This is because a virtual environment isclosely tied to both the Python it was created with, and the location it wascreated in. (The “relocatable” option of virtualenv does not work and is deprecated.) [3]
However, this is very easy to fix. Instead of moving/copying, just create a newenvironment in the new location. Then, run pip freeze > requirements.txt inthe old environment to create a list of packages installed in it. With that,you can just run pip install -r requirements.txt in the new environment toinstall packages from the saved list. (Of course, you can copy requirements.txtbetween machines. In many cases, it will just work; sometimes, you might need a fewmodifications to requirements.txt to remove OS-specific stuff.)
$ oldenv/bin/pip freeze > requirements.txt$ python3 -m venv newenv$ newenv/bin/pip install -r requirements.txt(You may rm -rf oldenv now if you desire)
Note that it might also be necessary to re-create your virtual environmentafter a Python upgrade, [4] so it might be handy to keep an up-to-daterequirements.txt for your virtual environments (for many projects, it makessense to put that in the repository).
To manage those requirements.txt files in a more orgnized yet still simpleway, you might be interested in pip-tools.
Frequently Asked Questions
I’m using virtualenv. Do I need to install it for each Python I want to use it with?
In most cases, you can use virtualenv -p pythonX env to specify a differentPython version, but with some Python version combinations, that approach mightbe unsuccessful. (The venv module is tied to the Python version it’sinstalled in.)
I’m the only user on my system. Do I still need virtual environments?
Yes, you do. First, you will still need separation between projects, sooner orlater. Moreover, if you were to install packages system-wide with pip, youmight end up causing conflicts between packages installed by the system packagemanager and by pip. Running sudo pip is never a good idea because of this.
I’m using Docker. Do I still need virtual environments?
They are still a good idea in that case. They protect you against any badsystem-wide Python packages your OS image might have (and one popular base OSis famous for those). They don’t introduce any extra overhead, while allowingto have a clean environment and the ability to re-create it outside of Docker(eg. for local development without Docker)
What about Pipenv?
Pipenv is a dependency management tool. It isn’t compatible with most workflows, and comes with many issues. In my opinion, it’s not worth using (Also, that thing about it being an officially recommended tool? Turns out it’s not true.)
I also wrote a blog post detailing concerns with that tool, titled Pipenv: promises a lot, delivers very little.
Consider using pip-tools instead.
