Virtual Environments Pt. 2A – Virtualenv

This is Part 2A of my series on virtual environments, which covers virtualenv, a Python specific package for managing virtual environments. If you are new to virtual environments, I suggest checking out Part 1 of the series, which covers virtual environment basics.  This guide will cover how to do the following with the virtualenv package:

  1. Installation
  2. Creating new environments
  3. Use and management
  4. Exporting
  5. Importing

This guide ends with a comprehensive example that incorporates most of the steps covered in the article.  Note that the version numbers you see locally may differ from my own; this is normal.  Different versions of python and pip may download different package versions.



Installation

virtualenv is a widely used virtual environment manager for Python.  The latest stable package can be installed through pip via:

$ pip install virtualenv


Creating New 
Environments

Creating a new virtual environment is a simple process.  Open a blank folder in the directory of your choosing and enter the following:

$ virtualenv my_venv

Using this command will create a new folder called my_venv which contains everything needed to run an isolated version of python.  Virtual environment name restrictions following standard file/directory name restrictions.  The word venv is usually used as a suffix for the environment name to clearly indicate purpose.

Using and Managing Current Environments

Virtual environments must be activated before use.  The method for activating virtual environments differs slightly depending on which OS you are using:

$ source my_venv/bin/activate #Linux & Mac
$ my_venv\Scripts\activate #Windows

Note the differences between slash direction, path, and whether or not the source keyword is used.

Activating will alter your command line by inserting (my_venv) to the left of the line itself.  Below are examples of an activated virtual environment for terminal and windows command prompt.

(my_venv) jhart@home$  #Linux $ Mac terminal
(my_venv) C:\Users\jhart #Windows cmd prompt

You can continue to use the terminal normally during this time.  You can deactivate the environment using the following command (same for all operating systems):

$ deactivate

Note that closing the terminal will also deactivate it.

While your virtual environment is active, pip can still be used to install packages.  The packages installed will match versions compatible with whatever version of Python is installed to the activated environment.  All installed packages to the are contained to the your virtual environment’s folder.

Deleting the environment doesn’t require any command at all; simply delete the folder created by the virtualenv command.


Custom Python Version

By default, your virtual environment’s python interpreter is the same version as your main python interpreter.  To change this, you can do the following:

$ virtualenv my_venv -python=python2.7

-python=python2.7 will result in python 2.7 being installed.  You main python install won’t be affected in any way, and any version of Python can be installed to a virtual environment.


Exporting 
Environments

Importing and exporting virtualenv environment is done through pip, not virtualenv.
To export a virtual environment, activate the environment of interest and enter the following into the terminal:

$ pip freeze > requirements.txt

This will create a text document called requirements in whatever directory the terminal is currently active in.  The file name isn’t limited to “requirements”, but it should be something descriptive if it will be shared with others.  The only stipulation is that it must be a .txt file.

If my_venv was the active virtual environment during the export process, this would create a blank document because only the default packages ( pip, setuptools, and wheel) were installed to it during initial generation process.  To output all installed packages, including the three defaults, enter the following:

$ pip freeze > requirements.txt --all

The --all flag tells pip to also include pip, setuptools, and wheel in addition to all other installed packages.  Executing that command locally gives me a document with the following:

pip==9.0.1
setuptools==36.2.1
wheel==0.29.0

 

Importing Requirements

The setup for importing a requirements file matches that of exporting a requirements file.  Activate the target virtual environment and enter the following:

$ pip install -r requirements.txt

The -r flag in the above command tells pip to install the requirements listed in the file requirements.txt.  Anything listed in that file will be installed via pip to the active environment.  If no environment is activated, pip will install everything yto your main Python environment.

You can read more about pip freeze here.


Comprehensive Example

To wrap up virtualenv, lets make another virtual environment, and use a new requirements.txt file from my_venv as the basis.   Before creating a new environment, lets install some actual python packages to my_venv prior to the export process.

$ source my_venv/bin/activate   #Linux activation method
$ pip install Splinter          #Install Splinter package
$ pip install pyperclip         #Install pyperclip package
$ pip freeze > requirements.txt #Export package requirements
$ deactivate                    #Deactivate environment

If there were no errors, requirements.txt should contain the following

pyperclip==1.5.27
selenium==3.4.3
splinter==0.7.5

Now, lets create a new virtual environment called new_venv. Once this new environment is created, we can try importing the  requirements.txt file.

$ virtualenv new_venv               #Create new environment
$ source new_venv/bin/activate      #Linux activation method
$ pip install -r requirements.txt   #Install package requirements
$ pip list                          #List installed packages

If the import process was successful, pip list should output the following:

pip (9.0.1)
pyperclip (1.5.27)
selenium (3.4.3)
setuptools (36.2.1)
splinter (0.7.5)
wheel (0.29.0)

Performing all these steps accomplishes the following tasks, all of which make up fundamental virtual environment management practices:

  1. Creation
  2. Activation
  3. Package installation
  4. Exporting environment requirements
  5. Importing environment requirements


Wrapping Up

If you are interested in alternative method of implementing virtual enviroments, you can check out Part 2B (upcomming), which covers the conda utility.  Unlike virtualenv and pip, which are limited to Python, conda can be used by any programming language.

Virtual Environments Pt. 1 – An Overview

post4

Isn’t collaboration great?  Whether you are a part of a community driven open-source project, or an over-worked design team whose souls have long since been crushed, you can be proud of being part of something bigger!

While development work often falls between these two extremes, package and extended library management is a constant to always be mindful of.  Although personal projects grant free reign to enact every bad programming practice your fingers can muster, subjecting contributors and co-workers to your wrath is not without consequence.

Managing packages and keeping track of dependencies becomes increasingly difficult as a project grows in size.  While introductory programming resources and texts often do a great job of covering programming concepts, areas related to project organization, and working in a development environment as whole, often don’t fit within the context of what is being taught.  Some aspects of these topics really can’t be taught, but can only be learned through experience.  There’s nothing mystical about this subject; it’s just made up of concepts that are hard to quantify in a clearly structured way.

This series on virtual environments aims to introduce the more technical concepts related to project management.  The importance of this topic is defined is only defined by the needs of the project you are a part of, and my intent is to express the importance of these concepts and utilities by conveying their usefulness.  This topic, however is only a small part of the whole; even though effective project management goes well beyond package and dependency management, it’s a part of the process, and incidentally, the topic of this post.

Overview

Virtualization, at it’s core, is the process of emulating something physical in a non-physical way.  Virtual machines, for example, create an instance of an operating system that runs off of virtualized hardware.  While this virtual hardware is derived from the physical components that make up your computer, its implementation is achieved through virtualization process.  In that respect, virtualization can be thought of as the process that makes something virtual.

Unfortunately, this explanation isn’t very clear because the term “virtual” has become almost synonymous with “computer”.   Personally, I like to think of virtual environments as a container that holds a workspace of sorts.  While the virtualization process itself varies depending on the context of the virtual environment, the purpose of a virtual environment doesn’t really change.  Similar to virtual machines, which operate alongside a pre-established OS,  a programming virtual environment can operate alongside a pre-established development environment.

This series is split into three sections:

  • Part 1 provides an overview of virtual environments, along with practical applications as well
  • Part 2A covers virtualenv, a Python specific tool used to implement virtual environments.
  • Part 2B covers Conda, a language independent tool that is both a package manager, and a virtual environment manager.

While all three parts were originally in one single article, it length became problematic. The problem wasn’t the information itself; the issue was putting it all in one place.  Parts 1 and 2A are being released together, and part 2B will be released as soon as it’s finished.



Collaboration and Project Management

Python’s already extensive library can be expanded through thousands of user made libraries and packages.  Many projects incorporate at least one of these packages, which in turn creates a dependency.  While keeping track of all dependencies specific to a project is a problem, it’s not the problem.

Every package has a version number,  and most packages are continually updated.   While updates are usually beneficial, changes to a package can create conflicts with other packages and/or Python itself .   Git may address file consistency across multiple contributors, but how do you address consistency across multiple development environments as well?  For most projects with more than a single contributer, the following are important considerations:

  1. Everyone is developing and testing code using the same version of Python
  2. Everyone has all external package dependencies installed
  3. Those external packages are the correct version

Keeping track of that manually is not only error prone, but unmaintainable depending on how many external libraries are required.  An even bigger concern is accounting for completely unrelated libraries that are part of a contributor’s system that might also introduce compatibility issues.  An additional consideration, for open-source projects at least; contributors can’t be expected to limit their only development environment to the scope of one project.

Under the Hood

A Python virtual environment, at the most basic level, will include some version of Python, pip, and setuptools.  These environments function alongside a user’s main python environment, allowing for different Python versions and packages to be installed alongside each other.   A Python 2 virtual environment will work just fine in a system with only Python 3 installed.  Virtual environments must be explicitly activated, so they only “take over” when instructed to do so.

Activating a virtual environment invokes a configuration change (of sorts) that temporarily alters the path variables related to the Python interpreter.  When I activate a virtual environment, I’m not so much creating a new process, I’m just pointing my system to an isolated version of Python.  While different tools create virtual environment in different ways, the result is a folder containing copies of, or links to the required version of Python, along with any external dependencies.

When a virtual environment is activated, any installed packages are installed to only that environment.  Because each environment is effectively isolated from all other environments, they can be quickly created, deleted, or changed to suit a project’s needs. The utilities used to manage virtual environments allow users to define working environments for specific projects.  This allows contributors to easily create isolated environments that match the working environments established by project authors.

While being able to share a working development environment is extremely beneficial, use case extends to other areas as well.  For example, virtual environments could be created to test the following:

  1. Addition or removal of packages and libraries
  2. Version compatibility between packages
  3. Checks against older/newer versions of Python

The process of actually defining a development environment for all project contributors is accomplished by exporting the names of all installed packages (and their version numbers) to a  .txt or ​.yaml file.  Contributors can download this file and use a local package manager to read the contents and automatically install all dependencies to a local environment.

Creating a virtual environment is handled by an environment manager, while importing and exporting environments are handled by a package manager.  Part-2A and Part-2B discuss the utilities used to either create or import/export a virtual environment.  A python specific solution can be accomplished through it’s package manager pip, along with virtualenv package.  A language-independent approach can be achieved through conda, which manages both packages and virtual environments.  Part 2A and 2B cover using virtualenv and conda in respect to the following:

  1. Installation
  2. Creating environments
  3. Using and managing environments

 

Links to those articles can be found here as those articles become available: