Managing dependencies

For local development, Anaconda is a perfect tool: it has almost all the libraries we may ever need. This, however, also has a downside: it takes up 4 GB when unpacked, which is too large. For production, we prefer to have only the libraries we actually need.

Additionally, different services have different requirements. Often, these requirements conflict, so we cannot use the same environment for running multiple services at the same time

Pipenv

In the previous chapters, we use a few libraries, such as: Matplotlib, Seaborn, NumPy, Pandas, Scikit-Learn and Flask.

For example, the Anaconda distribution contain a large number of the libraries, but we don’t use all. We can get a fresh Python installation and install only the libraries we need with pip, such as:

pip install numpy scikit-learn flask

pip command goes to PyPI.org and gets and installs the latest version of the libraries. The installed libraries have specific versions. If we later execute a script created with these library versions in another environment, we may encounter a version conflict. We can try to solve this problem by reinstalling the correct versions, but this may not be possible due to other related issues.

We can solve these problems by creating a virtual environment for each project — a separate Python distribution with nothing else but libraries required for this particular project.

Pipenv is a tool that makes managing virtual environments easier. We can install it with pip:

pip install pipenv

After that, we use pipenv instead of pip for installing dependencies:

pipenv install numpy scikit-learn flask

After finishing the installation, it creates two files: Pipenv and Pipenv.lock .

The Pipenv file looks like this:

[[source]]
url = "<https://pypi.org/simple>"
verify_ssl = true
name = "pypi"

[packages]
numpy = "*"
scikit-learn = "*"
flask = "*"

[dev-packages]

[requires]
python_version = "3.11"

This file contains a list of libraries as well as the version of Python we use. The Pipenv.lock file is quite large, but we can look at a small section of its contents:

"flask": {
          "hashes": [
              "sha256:34e815dfaa43340d1d15a5c3a02b8476004037eb4840b34910c6e21679d288f3",
              "sha256:ceb27b0af3823ea2737928a4d99d125a06175b8512c445cbd9a9ce200ef76842"
          ],
          "index": "pypi",
          "markers": "python_version >= '3.8'",
          "version": "==3.0.3"
        }

As we can see, it records the exact version of the library that was used during installation. To make sure the library doesn’t change, it also saves the hashes. This way, we “lock” the dependencies to specific versions.