Read all files in a directory in Python

Read all files in a directory in Python

posted 10 min read

Reading the contents of a file is one of the skills one has to learn to know more about the language. We read a file to search for a word, phrase, number, or some string pattern, reformat it, print what is on it, etc.

We can read all files in a directory with the use of the modules os, pathlib, glob to get the file names from a given directory, and use the built-in [1] functions open() [3] and print() [9] to open the file for reading and printing the contents on the screen.

1.  What is the problem? #

The problem is to identify the file names under the given directory and open each file for reading. No changes will be done to the file, we will just read it and send the contents to the standard output or user's screen.

Let us have an example directory structure to help us understand this better. I am using windows 10, powershell, and the tree command.

PS F:\Project\coderlegion> tree F:\Project\coderlegion /F
F:\PROJECT\CODERLEGION
│   fifa_ranking_men.txt
│   fifa_ranking_women.txt
│
└───article
        article.txt

Our directory absolute path is F:\PROJECT\CODERLEGION. It has two files namely fifa_ranking_men.txt and fifa_ranking_women.txt. It also has a subdirectory called article and within it, is a file named article.txt.

Our goal is to read the files fifa_ranking_men.txt and fifa_ranking_women.txt and show their contents on the screen.

The contents of the files (top 3) are:

filename: fifa_ranking_men.txt

Brazil 1841
Belgium 1817
Argentina 1774

filename: fifa_ranking_women.txt

USA 2087
Sweden 2066
Germany 2065

2.  Solutions #

We can break down the problem into two steps, first, we need to find all the files in a directory and second, we will open each file for reading and show its contents on the screen.

There are many ways to find the files in a directory. We can use the functions listdir() [4], walk() [6] and scandir() [5] from the os module, methods Path.iterdir() [8] and Path.glob() [7] from the pathlib module, and function glob() [2]from the glob module.

To open the file for reading, we will use the built-in functions open() and print() to show its contents.

I will show the complete python program examples that can run from the command line based on the directory structure shown in section 1.

2.1.  os.listdir() #

Here is an example code of how to use the listdir() [4] in finding files in a directory and use open() [3] to read file and use print() [9] to show its contents on the screen.

filename: read_file_listdir.py

"""Find files, read its contents and print it on the screen.

Use the listdir function of os module to search all
files in a directory. Open the file for reading with open()
and show its contents with print().

Directory tree:

  F:\PROJECT\CODERLEGION
  │   fifa_ranking_men.txt
  │   fifa_ranking_women.txt
  │
  └───article
          article.txt
"""

import os

def read_file(folder):
    """Find all the file names in a folder and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files and directories.
    for entry in os.listdir(folder):
        file_path = os.path.join(folder, entry)

        # Check if entry is a file.
        if os.path.isfile(file_path):
            print(entry)

            # Read the file if it ends with .txt.            
            if entry.endswith('.txt'):
                with open(file_path, 'r') as handle:  # open the file for reading
                    for line in handle:
                        line = line.rstrip()  # remove the newline char
                        print(line)

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_listdir.py

output

fifa_ranking_men.txt
Brazil 1841
Belgium 1817
Argentina 1774
fifa_ranking_women.txt
USA 2087
Sweden 2066
Germany 2065

2.2.  os.walk() #

filename: read_file_os_walk.py

import os

def read_file(folder):
    """Find all the file names in a folder and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files and directories.
    for dir_path, subdir_path_list, filename_list in os.walk(folder):
        print(f'dir_path: {dir_path}')
        print(f'subdir_path: {subdir_path_list}')
        print(f'filename: {filename_list}')

        # Process the files.
        for fn in filename_list:

            # Read the file with .txt extension. These files are
            # fifa_ranking_men.txt, fifa_ranking_women.txt
            if fn.endswith('.txt'):

                # Construct an absolute path using join, like
                # F:/Project/coderlegion/fifa_ranking_men.txt
                file_full_path = os.path.join(dir_path, fn)

                # Reads the file and print its contents.
                with open(file_full_path, 'r') as handle:
                    for line in handle:
                        line = line.rstrip()  # remove the newline char
                        print(line)

        # Do not read the files under the subdirectories.
        # We are only interested on the files under the F:/Project/coderlegion/
        # but not the files under its subdirectories.
        # Get out of the for loop with break statement.
        break

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_os_walk.py

output

dir_path: F:/Project/coderlegion/
subdir_path: ['article']
filename: ['fifa_ranking_men.txt', 'fifa_ranking_women.txt']
Brazil 1841
Belgium 1817
Argentina 1774
USA 2087
Sweden 2066
Germany 2065

2.3.  os.scandir() #

filename: read_file_os_scandir.py

import os
    
def read_file(folder):
    """Find all the file names in a folder and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files and directories.
    # Context manager is supported from Python version 3.6 and later.
    with os.scandir(folder) as it:
        for entry in it:

            # Check if entry is a file.
            if entry.is_file():

                # Read the file if it ends with .txt.            
                if entry.name.endswith('.txt'):
                    file_path = entry.path
                    print(file_path)

                    with open(file_path, 'r') as handle:  # open the file for reading
                        for line in handle:
                            line = line.rstrip()  # remove the newline char
                            print(line)

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_os_scandir.py

output

F:/Project/coderlegion/fifa_ranking_men.txt
Brazil 1841
Belgium 1817
Argentina 1774
F:/Project/coderlegion/fifa_ranking_women.txt
USA 2087
Sweden 2066
Germany 2065

2.4.  path.iterdir() #

filename: read_file_path_iterdir.py

from pathlib import Path

def read_file(folder):
    """Find all the file names in a folder and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files and directories.
    for path in Path(folder).iterdir():

        # Check if path is a file.
        if path.is_file():
            print(path)

            # Read the file if it ends with .txt.            
            if path.name.endswith('.txt'):
                with open(path, 'r') as handle:  # open the file for reading
                    for line in handle:
                        line = line.rstrip()  # remove the newline char
                        print(line)

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_path_iterdir.py

output

F:\Project\coderlegion\fifa_ranking_men.txt
Brazil 1841
Belgium 1817
Argentina 1774
F:\Project\coderlegion\fifa_ranking_women.txt
USA 2087
Sweden 2066
Germany 2065

2.5.  path.glob() #

filename: read_file_path_glob.py

from pathlib import Path

def read_file(folder):
    """Find all the file names in a folder and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files with .txt extension.
    pattern = '*.txt'
    for path in Path(folder).glob(pattern):
        print(path)

        # Read the contents and show on the screen.
        with open(path, 'r') as handle:  # open the file for reading
            for line in handle:
                line = line.rstrip()  # remove the newline char
                print(line)

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_path_glob.py

output

F:\Project\coderlegion\fifa_ranking_men.txt
Brazil 1841
Belgium 1817
Argentina 1774
F:\Project\coderlegion\fifa_ranking_women.txt
USA 2087
Sweden 2066
Germany 2065

2.6.  glob.glob() #

filename: read_file_glob_glob.py

import glob

def read_file(folder):
    """Find all the files in a directory and read it.

    Read each file and print the contents on the screen.

    Args:
      folder (str): The root folder to search for files.
    """
    # Find the files that ends in .txt.
    for file_path in glob.glob(f"{folder}/*.txt"):
        print(file_path)
        with open(file_path, 'r') as handle:  # open the file for reading
            for line in handle:
                line = line.rstrip()  # remove the newline char
                print(line)

if __name__ == '__main__':
    # Define the folder to search for the files.
    folder = 'F:/Project/coderlegion/'
    read_file(folder)

command line

python read_file_glob_glob.py

output

F:/Project/coderlegion\fifa_ranking_men.txt
Brazil 1841
Belgium 1817
Argentina 1774
F:/Project/coderlegion\fifa_ranking_women.txt
USA 2087
Sweden 2066
Germany 2065

3.  Conclusion #

Reading files in a directory can be achieved with the use of the os.listdir() [4], os.walk() [6], os.scandir() [5], path.iterdir() [8], path.glob() [7], and glob.glob() [2]. Once the file name is identified, it is then opened for reading using open() [3], and show the contents on the screen with print() [9].

4.  References #

If you read this far, tweet to the author to show them you care. Tweet a Thanks

More Posts

How to read a file and search specific word locations in Python

Brando - Nov 8, 2023

How to Fix the TypeError: cannot use a string pattern on a bytes-like object Error in Python

Cornel Chirchir - Oct 29, 2023

Opening & Reading Files Handling in Python

Abdul Daim - Apr 12

Writing to Files In Python

Abdul Daim - Apr 12

Mastering Reading and Writing CSV Files in Python

Abdul Daim - Mar 27
chevron_left