Python Installing External Packages Sources

posted 9 min read

Table of Contents

1. Introduction

In the previous article Modules and Packages Installing External Packages with pip we have discussed the basic package management with pip. In this article, we will go through the more advanced topics that will help you manage your project packages professionally so let's get started.

2. Installing packages from alternative sources

Sometimes the required package is not available on PyPI. In that case, pip will allow you to install packages from local directories or version control repositories.

Installing packages from version control repositories (GitHub, GitLab, Bitbucket)

GitHub

Follow the below syntax to install packages from GitHub.

pip3 install git+https://github.com/username/repository.git

Replace the original URL of the repository here https://github.com/username/repository.git.

You can install the pandas package from GitHub using the following syntax.

pip3 install git+https://github.com/pandas-dev/pandas.git

GitLab

Follow the below syntax to install packages from GitLab.

pip3 install git+https://gitlab.com/username/repository.git

Bitbucket

Follow the below syntax to install packages from Bitbucket.

pip3 install git+https://bitbucket.org/username/repository.git

Installing packages from local directories or archives

Installing packages from directories

To install a package from the local directory follow the below syntax.

pip3 install -e /path/to/directory

The -e flag simply means that if there are changes in the source code then it will be reflected and you don't have to reinstall.

Consider the following syntax of installing a package from a local directory.

pip3 install -e /home/mypc/Documents/pandas

Installing packages from archives

Similarly, you can also install packages from an archive file (e.g., .tar.gz, .zip) following the syntax below.

pip3 install /path/to/directory-archive.tar.gz

Consider the following syntax of installing a package name pandas from an archive file.

pip3 install /home/mypc/Documents/pandas.tar.gz

3. Virtual environments and pip

Introduction to virtual environments

If you are working with multiple projects then there is a high chance that you need different dependencies for different projects so you can't install the dependencies at the global level because it will conflict with other projects. To avoid those conflicts you can use virtual environments. Virtual environments help you to isolate the project dependencies. By doing that you can install different dependencies for each project in their isolated environment.

Creating and activating virtual environments

You can create a virtual environment either using venv or virtualevn . Let's discuss them one by one.

Creating and activating virtual environments using venv

Follow the below syntax to create a virtual environment using venv.

python3 -m venv env_name

Windows

python -m venv env

The above command will create a virtual environment named env.

You can activate the virtual environment by using the command below.

env\Scripts\activate

Linux/macOS

python3 -m venv env

The above command will create a virtual environment named env.

You can activate the virtual environment by using the command below.

source env/bin/activate

Creating and activating virtual environments using virtualenv

Before using virtualenv to create virtual environments you have to install it by running the below command.

pip3 install virtualenv

After successful installation you can follow the following syntax to create a virtual environment using virtualenv.

virtualenv env_name

Windows

virtualenv env

The above command will create a virtual environment named env.

You can activate the virtual environment by using the command below.

env\Scripts\activate

Linux/macOS

virtualenv env

The above command will create a virtual environment named env.

You can activate the virtual environment by using the command below.

source env/bin/activate

4. Requirements Files

Requirements files are simply text files in which the project dependencies are listed. It helps manage and reproduce project environments. Following is an example of a requirements.txt file.

requests>=2.25.1
numpy==1.21.3
pandas~=1.3.4

Creating and using requirements files

You can create a requirements.txt file manually by listing down all the dependencies or you can use the following command.

pip3 freeze > requirements.txt

The above command will create a requirements.txt containing all the dependencies listed in it.

To install the dependencies using requirements.txt you can follow the below command.

pip3 install -r requirements.txt

5. Handling package installation errors

You might encounter errors during package installation using pip. In this section we will cover the common errors and we will see how to troubleshoot them.

Common errors

Following are the common errors that you might encounter while installing packages using pip.

Package not found

If you try to install a package that doesn't exist, or you misspelled the package's name then you will get the following error.

To resolve this error verify that the package you are trying to install exists. If so then check the spelling of the package name that you are trying to install.

Dependency conflict

Dependency conflicts occur when two or more packages require different versions of the same dependency and those versions are incompatible with each other.

If you try to install the following packages you will get a dependency conflict.

pip3 install numpy==1.19.5

pip3 install pandas==1.3.4

To resolve dependency conflicts you can upgrade or downgrade the package so that there will be no dependency conflicts.

Permission error

Permission error indicates that you might not have the necessary permissions to install the package in the target directory.

Permission error in linux

If you try to install a package outside a virtual environment in linux you will get the permission error.

pip3 install pandas

The above command will generate the following error.

To resolve the above error create the virtual environment as discussed above then run the command pip3 install pandas again.

Insufficient user privileges

Sometimes you might get this error due to user account privileges. In that case try to elevate the permissions while trying to install the packages.

Linux/macOS

Instead of using pip3 install package_name try the following command to install the packages.

sudo pip3 install package_name

Consider the following command of installing pandas with elevated permissions.

sudo pip3 install pandas

Caution: It is not recommended to use this method it is always encouraged to create a virtual environment

Windows

In windows, you have to run the command prompt in administrator mode to elevate the permissions. Follow the below steps to do so.

  • Right-click on the command prompt shortcut
  • Select Run as administrator

Once the command prompt is opened try to install the package again.

pip3 install pandas

6. Advance pip features

Pip offers a range of advanced features that can enhance your package management experience.

Environment markers

Environment markers allow you to specify conditional dependencies based on the python version, operating system, and other environment variables. By using environment markers you can avoid compatibility and conflict issues.
If you have a python project that requires different dependencies based on the python version and operating system then you can use environment markers to enhance your requirements.txt file as follows.

#Common dependencies for all environments
flask>=2.0.0
requests>=2.0.0

#Dependencies for Python 3.6 and above
pytz; python_version >= '3.6'

#Dependencies for Linux-based systems
linux-specific-package; sys_platform == 'linux'

#Dependencies for Windows
pywin32; sys_platform == 'win32'

Above you can see that we have created a requirements.txt file that will install dependencies considering the specific environments.

Editable installs

Editable installs also known as development mode help you to install python packages from a local source directory in such a way that changes made to the source code are directly reflected in the installed package without reinstalling it.

Suppose you have a package name my_package to install it in development mode you have to go to the root directory of my_package and run the following command.

pip3 install -e .
or
pip3 install --editable .

The above commands will install the package in editable mode the . represents the current directory.



FAQ
Q. What is pip?
A. Pip is a package manager in python that allows you to install manage and remove python packages from PyPI and other repositories.

Q. How do I install a package using pip?
A. You can install a package using pip by running the command 'pip install package_name'.

Q. How can I upgrade a package using pip?
A. You can upgrade a package using pip by running the command 'pip install --upgrade package_name'.

Q. How do I uninstall a package using pip?
A. To uninstall a package, you can use the command 'pip uninstall package_name'.

Q. What is a virtual environment, and why should I use it?
A. A virtual environment is an isolated python environment that allows you to install and manage dependencies separately from other projects. You should use it to avoid conflicts between different project dependencies.

Q. What is a requirements file, and how do I use it?
A. A requirements file is a text file that lists the dependencies of a python project. You can use it to install all dependencies at once using the command 'pip install -r requirements.txt'.

7. Conclusion

In this article we have covered some advanced package management techniques using pip. You can use pip to install packages from PyPI, local directories, or other sources like github, gitlab, etc. Pip allows you to install packages in virtual environments to avoid any conflicts. Pip allows you to create requirements files which is helpful to manage project dependencies at a larger scale. You can use environment markers to install specific dependencies for certain environments. Pip helps you to install the packages in editable mode which is helpful in the development phase. Overall pip provides basic to advanced features to manage packages in python.

8. References

If you read this far, tweet to the author to show them you care. Tweet a Thanks

More Posts

Python Installing External Packages with pip

muhammaduzairrazaq - Mar 25

Modules and Packages: Using Standard Library Modules

muhammaduzairrazaq - Jul 12

Mastering Importing Modules in Python

muhammaduzairrazaq - Apr 11

How to Fix 'jupyter' is Not Recognized as an Internal or External Command

mouyuan123 - May 26

In this article, I’ll break down how the csv module works in Python, from basic usage to more advanced techniques.

Nuno Bispo - Oct 23
chevron_left