Table of Contents
1. Introduction
In the previous article Modules and Packages Installing External Packages with pip we have discussed the basic package management with pip. In this article, we will go through the more advanced topics that will help you manage your project packages professionally so let's get started.
2. Installing packages from alternative sources
Sometimes the required package is not available on PyPI. In that case, pip will allow you to install packages from local directories or version control repositories.
Installing packages from version control repositories (GitHub, GitLab, Bitbucket)
GitHub
Follow the below syntax to install packages from GitHub.
pip3 install git+https://github.com/username/repository.git
Replace the original URL of the repository here https://github.com/username/repository.git
.
You can install the pandas package from GitHub using the following syntax.
pip3 install git+https://github.com/pandas-dev/pandas.git
GitLab
Follow the below syntax to install packages from GitLab.
pip3 install git+https://gitlab.com/username/repository.git
Bitbucket
Follow the below syntax to install packages from Bitbucket.
pip3 install git+https://bitbucket.org/username/repository.git
Installing packages from local directories or archives
Installing packages from directories
To install a package from the local directory follow the below syntax.
pip3 install -e /path/to/directory
The -e
flag simply means that if there are changes in the source code then it will be reflected and you don't have to reinstall.
Consider the following syntax of installing a package from a local directory.
pip3 install -e /home/mypc/Documents/pandas
Installing packages from archives
Similarly, you can also install packages from an archive file (e.g., .tar.gz, .zip) following the syntax below.
pip3 install /path/to/directory-archive.tar.gz
Consider the following syntax of installing a package name pandas from an archive file.
pip3 install /home/mypc/Documents/pandas.tar.gz
3. Virtual environments and pip
Introduction to virtual environments
If you are working with multiple projects then there is a high chance that you need different dependencies for different projects so you can't install the dependencies at the global level because it will conflict with other projects. To avoid those conflicts you can use virtual environments. Virtual environments help you to isolate the project dependencies. By doing that you can install different dependencies for each project in their isolated environment.
Creating and activating virtual environments
You can create a virtual environment either using venv
or virtualevn
. Let's discuss them one by one.
Creating and activating virtual environments using venv
Follow the below syntax to create a virtual environment using venv
.
python3 -m venv env_name
Windows
python -m venv env
The above command will create a virtual environment named env
.
You can activate the virtual environment by using the command below.
env\Scripts\activate
Linux/macOS
python3 -m venv env
The above command will create a virtual environment named env
.
You can activate the virtual environment by using the command below.
source env/bin/activate
Creating and activating virtual environments using virtualenv
Before using virtualenv
to create virtual environments you have to install it by running the below command.
pip3 install virtualenv
After successful installation you can follow the following syntax to create a virtual environment using virtualenv
.
virtualenv env_name
Windows
virtualenv env
The above command will create a virtual environment named env
.
You can activate the virtual environment by using the command below.
env\Scripts\activate
Linux/macOS
virtualenv env
The above command will create a virtual environment named env
.
You can activate the virtual environment by using the command below.
source env/bin/activate
4. Requirements Files
Requirements files are simply text files in which the project dependencies are listed. It helps manage and reproduce project environments. Following is an example of a requirements.txt
file.
requests>=2.25.1
numpy==1.21.3
pandas~=1.3.4
Creating and using requirements files
You can create a requirements.txt
file manually by listing down all the dependencies or you can use the following command.
pip3 freeze > requirements.txt
The above command will create a requirements.txt
containing all the dependencies listed in it.
To install the dependencies using requirements.txt
you can follow the below command.
pip3 install -r requirements.txt
5. Handling package installation errors
You might encounter errors during package installation using pip. In this section we will cover the common errors and we will see how to troubleshoot them.
Common errors
Following are the common errors that you might encounter while installing packages using pip.
Package not found
If you try to install a package that doesn't exist, or you misspelled the package's name then you will get the following error.
To resolve this error verify that the package you are trying to install exists. If so then check the spelling of the package name that you are trying to install.
Dependency conflict
Dependency conflicts occur when two or more packages require different versions of the same dependency and those versions are incompatible with each other.
If you try to install the following packages you will get a dependency conflict.
pip3 install numpy==1.19.5
pip3 install pandas==1.3.4
To resolve dependency conflicts you can upgrade or downgrade the package so that there will be no dependency conflicts.
Permission error
Permission error indicates that you might not have the necessary permissions to install the package in the target directory.
Permission error in linux
If you try to install a package outside a virtual environment in linux you will get the permission error.
pip3 install pandas
The above command will generate the following error.
To resolve the above error create the virtual environment as discussed above then run the command pip3 install pandas
again.
Insufficient user privileges
Sometimes you might get this error due to user account privileges. In that case try to elevate the permissions while trying to install the packages.
Linux/macOS
Instead of using pip3 install package_name
try the following command to install the packages.
sudo pip3 install package_name
Consider the following command of installing pandas with elevated permissions.
sudo pip3 install pandas
It is not recommended to use this method it is always encouraged to create a virtual environment
Windows
In windows, you have to run the command prompt in administrator mode to elevate the permissions. Follow the below steps to do so.
- Right-click on the command prompt shortcut
- Select
Run as administrator
Once the command prompt is opened try to install the package again.
pip3 install pandas
6. Advance pip features
Pip offers a range of advanced features that can enhance your package management experience.
Environment markers
Environment markers allow you to specify conditional dependencies based on the python version, operating system, and other environment variables. By using environment markers you can avoid compatibility and conflict issues.
If you have a python project that requires different dependencies based on the python version and operating system then you can use environment markers to enhance your requirements.txt
file as follows.
#Common dependencies for all environments
flask>=2.0.0
requests>=2.0.0
#Dependencies for Python 3.6 and above
pytz; python_version >= '3.6'
#Dependencies for Linux-based systems
linux-specific-package; sys_platform == 'linux'
#Dependencies for Windows
pywin32; sys_platform == 'win32'
Above you can see that we have created a requirements.txt
file that will install dependencies considering the specific environments.
Editable installs
Editable installs also known as development mode help you to install python packages from a local source directory in such a way that changes made to the source code are directly reflected in the installed package without reinstalling it.
Suppose you have a package name my_package to install it in development mode you have to go to the root directory of my_package and run the following command.
pip3 install -e .
or
pip3 install --editable .
The above commands will install the package in editable mode the .
represents the current directory.
Q. What is pip?
A. Pip is a package manager in python that allows you to install manage and remove python packages from PyPI and other repositories.
Q. How do I install a package using pip?
A. You can install a package using pip by running the command 'pip install package_name'.
Q. How can I upgrade a package using pip?
A. You can upgrade a package using pip by running the command 'pip install --upgrade package_name'.
Q. How do I uninstall a package using pip?
A. To uninstall a package, you can use the command 'pip uninstall package_name'.
Q. What is a virtual environment, and why should I use it?
A. A virtual environment is an isolated python environment that allows you to install and manage dependencies separately from other projects. You should use it to avoid conflicts between different project dependencies.
Q. What is a requirements file, and how do I use it?
A. A requirements file is a text file that lists the dependencies of a python project. You can use it to install all dependencies at once using the command 'pip install -r requirements.txt'.
7. Conclusion
In this article we have covered some advanced package management techniques using pip. You can use pip to install packages from PyPI, local directories, or other sources like github, gitlab, etc. Pip allows you to install packages in virtual environments to avoid any conflicts. Pip allows you to create requirements files which is helpful to manage project dependencies at a larger scale. You can use environment markers to install specific dependencies for certain environments. Pip helps you to install the packages in editable mode which is helpful in the development phase. Overall pip provides basic to advanced features to manage packages in python.
8. References