Writing to Files In Python

Question

Suppose you want your Python script to fetch data from the internet and then process that gathered data. If that data is limited and brief, then processing can be done every time you run the script, but in the case of humongous data, we cannot perform repetitive processing of the script. Hence, the processed data needs to be stored somewhere, and then data storage and writing to files come into the picture. It's important to remember that while writing to a file, you should maintain the consistency and integrity of the data. So, In this article, you will explore the various aspects of file handling in Python, its writing and appending operations, and some techniques for optimizing the performance of writing operations.

Writing Data to Text Files

After understanding how to read and write data in binary files, we will explore the various ways to write data to the text file.

1. Writing Text Data to Plain Text Files

A plain text file is the most basic type of file that contains the data in textual form and has no defined structure. So, using the write() operation, you can perform file handling on plain text files.

1-1. Encoding and Decoding Text Data

When dealing with text files, it is crucial to manage the non-ASCII characters and encoding properly in Python. Encoding is the process that converts plain data into numbers so that the computer can understand it. But after completing the encoding and writing operation, if you want to read the saved data from the file, you can decode the data in human-readable form. It's done by writing the encoding as a parameter in the open() function. By default, in Python the encoding is UTF-8, but you can specifically use a different one.

file_path = "Content.txt" #provide location of file in the same directory      
#used for performing writing operation    
with open(file_path, 'w', encoding= 'utf-8') as file: #here, the file open operation is performed with write mode and encode the data    
    file.write("Bonjour le monde!") #Write function will add the line in the file, "Hello World! in French"    
#This is used for reading operation    
with open(file_path, 'r', encoding= 'utf-8') as file: #here, the file open operation is performed with read mode    
    content = file.read() #print what we write in the file  
    encode = content.encode('utf-8') 
    print("Encoded Data", encode)#the encoded string is print, the 'b' string in output indicate that it is byte object 
    decode = encode.decode('utf-8') #here we decode the encoded string with utf-8 
    print("Decoded Data", decode)

Note: Remember, you don't have to specify UTF-8, when working in python-3. Here, I am just using it for explaining the example.

Output

1-2. Handling Different Text Encodings

When your program handles the data from different sources and each source follows a different encoding scheme, then handling different text encoding schemes is necessary. It is not rocketing science; you simply have to create a list of multiple encoding schemes. Then, attempt to open the file using the try-except block to handle any encoding error gracefully until the file opens successfully.

file_path = "Content.txt" #provide location of file in the same directory   
with open(file_path, 'w', encoding= 'utf-16') as file: #here, the file open operation is performed with write mode and encode the data    
    file.write("Bonjour le monde!") #write function will add the line in the file "Hello World! in french"    
encoding_Schemes = ['utf-8', 'utf-16', 'utf-32', 'latin-1']    
for ens in encoding_Schemes: 
    try: 
        with open(file_path, 'r', encoding= ens) as file:#here, the file open operation is performed with read mode    
            content = file.read() #print what we write in the file  
            print("The content is read by: ",{ens}) 
            break 
    except UnicodeDecodeError as e: 
        print("using", {ens} ,"decoding is failed, trying next")

Output

2. Writing Formatted Data to Text Files

Just like writing the plain textual data to a file, you can perform writing operations for the structured and formatted data in the text file.

2-1. Using String Formatting For Structured Data

For writing structured data, python provides us with some string formatting methods by which we can align the data nicely in columns. These are str.format() or you can also use the f-literal string introduced in Python 3.6. Here, I am using the f-literal in my example.

file_path = "Content.txt" #provide location of file in the same directory   
#structured data in the form of tuples  
data = { 
    "fruit": "Apple", 
    "size": "Large", 
    "color": "Red" 
} 
with open(file_path, 'w') as file: #here, the file open operation is performed with write mode 
#here, each filed from the data represent here with their value by adding the tab and newline to format the data nicely    
    file.write(f"{{\n\tfruit:{data['fruit']}, \n\t size:{data['size']}, \n\t color:{data['color']}\n}}")  
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode    
    print(file.read())

Output

2.2- Writing Tabular Data to Text Files

If you want to write the tabular data into the text file, you can attain it by the same above-defined method of f-literal string.

file_path = "Content.txt" #provide location of file in the same directory   
#structured data in the form of tuples  
data = [ 
   {"fruit": "Apple", "size": "Large", "color": "Red"}, 
   {"fruit": "Mango", "size": "Large", "color": "Yellow"}, 
   {"fruit": "Orange", "size": "Medium", "color": "Orange"} 
   ] 
with open(file_path, 'w') as file: #here, the file open operation is performed with write mode    
    file.write(f"{'fruit' :<10} | {'size' :<7} | {'color' :<10}\n") #this writes key from the data with between defined spaced  
    file.write(f"{'-' * 30}\n") #this creates the 30 dashed between the header and the data rows 
    for list in data: # this will iteratively write the data in the file 
       file.write(f"{list['fruit'] :<10} | {list['size'] :<7} | {list['color'] :<10}\n")  
with open(file_path, 'r') as file:#here, the file open operation is performed with read mode    
    print(file.read())

Output

Appending Data to Files: Overview of Append Mode

Appending in file handling is used when you want to insert data in the file without truncating the file. It would not overwrite the current content of the file, instead, used for writing the content at the end of the file. It is used when you want to update the file with new content without disturbing the existing file content. When you open the file in appending mode, the pointer is positioned at the end of the file if it exists otherwise, it will create a new file. To append the data, you must set the mode as ‘a’ for text files and ‘ab’ for binary files in Python.

Appending data to an Existing file: Code Example

1. Opening files in Append Mode

Before appending the content to the file, we first open() the file. To append the data into the file, use the open() function with the mode ‘a’.

with open(file_path, 'a') as file: #here, the file open operation is performed with append mode

2. Writing New Data to the End of Files

file_path = "Content.txt" #provide location of file in the same directory   
#used for performing append operation  
with open(file_path, 'a') as file: #here, the file open operation is performed with append mode 
    file.write("\nIt is first released on February 20, 1991") #add this line at the end of file  
#this is used for reading operation  
with open(file_path, 'r') as file:#here, the file open operation is performed with read mode   
    print(file.read()) #print what we write in the file

Output

Real-world Examples and Use Cases

Let's analyze real-world examples and use cases where you need to store and write the data into different file formats and illustrate them with Python code.

1. Writing log data to files

Suppose you are creating a data pipeline that ingests the journal data and performs analysis on the data. After successfully creating the pipeline, you write the events which occurred within the application. So, developers generate log files for recording the events that occur within the program. The log data consists of messages like warnings and info, which is utilized for tracking and debugging the application behavior over time.

To begin writing log data to the file, we need to import the Python built-in module first, i.e. logging that provides the flexibility to work with log messages. After that, the basicConfig() function is used to create and configure the logger to write data to the file. You have to pass the name of the file to the basicConfig() function so that it records the events. The default mode for log-like is append ‘a’ mode, but we can also write by using ‘w’ mode. Now, let's move to coding in Python.

import logging 
file_path = "example.log" #provide location of file in the same directory   
#create and config the logger with the file path, its mode and format for printing the log data 
logging.basicConfig(filename = file_path, filemode = 'w', format = '%(name)s - %(levelname)s - %(message)s') # create and config the loger 
logging.critical("A Critical Logging Message") 
logging.warning("A Warning Logging Message") 
logging.error("An Error Logging Message") 
with open(file_path, 'r') as log_file: 
    log_Data = log_file.read() 
print(log_Data)

Output

2. Saving User Configurations to Files

Moving on to the second use case, that is to save the user configuration to the file. As we know, configuration files are used to store the settings and parameters of the application. The developer uses configuration files to provide a way for users and administrators to adjust various parameters and settings of software applications so that they can run in different environments without modifying the code. There is no specific format for configuration files. Instead, we use different formats for writing config data like JSON, XML, etc. Here, I am discussing the example of writing the user configuration to the JSON file.

import json   
file_path = "sample3.json" #provide location of file in the same directory     
with open(file_path, 'w') as file: #here, the file open operation is performed with write mode   
    config_Data = {   
        "User": "admin",   
        "Password": "admin123", 
        "Host": "localhost",   
        "Port": "8080"  
        }   
    json.dump(config_Data, file) #it dumps the JSON data into the file using JSON dump function   
    #this is used for reading operation   
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode    
    print(file.read()) #print what we write in the file

Output

Performance Optimization and Best Practices

While performing writing operations, it is essential to consider several performance factors when dealing with large amounts of data. I will discuss performance considerations and optimization techniques to enhance the work quality, along with best practices for writing clean and maintainable code.

Performance Considerations for File Writing

1. Buffering: When you call the write() function, it does not write the content to the file at every call but saves the data into a buffer and writes it in large chunks. To improve the performance of the write() operation, it's better if you specify the buffer size with a larger file. Writing the large buffer into the file reduces the need for multiple writing operations. Also, python has the specialty to save the buffered output by default.
2. Writing Large Files: Writing a large amount of data at once is better than writing each line separately. It reduces the time complexity and improves performance.
3. Modes Impact: When you open the file, the mode used to perform the task can impact the performance. For example, the appended mode will take longer than the write mode as the pointer needs to be shifted to the end of the file.
4. Closing Files: When writing is complete, it's better to close the file. This will ensure the release of the allocated resources and stored written data to disk.
5. Avoid Unnecessary Write: If there is no need to change the file content, then try to avoid using the unnecessary write() function.

Techniques for Efficient File Writing

1. When performing file handling, it's better to open the file once, perform all necessary writing, and then close it, rather than opening and closing the file repeatedly.
2. Always use the right mode while dealing with files. For example, if you need to add data to an existing file, use append mode; otherwise, use write mode to overwrite the file.
3. Writing a bunch of data to the file is better than writing one piece at a time. It means adding larger chunks to perform the writing operation efficiently.
4. For storing written data, faster storage drives like SDD provide better storage than the traditional HDD.

Best Practices for Writing Clean and Maintainable Code

1. Always write code that is easy to understand. The name of the variables and functions should describe their purpose.
2. Always incorporate as many comments as possible where necessary.
3. Follow the convention for writing the clean code defined by the PEP 8 style guide to make a consistent code style.
4. Always incorporate error handling in your program. It will help you to deal with the unexpected event beforehand and prevent the program from crashing.

FAQs Q: How can I append data to an existing file?
A: You can append the data in an existing file by setting the file mode as 'a'. Q: Can i write data in XML file?
A: Yes, you can write XML data by opening the XML file with the 'w'. It is like nested structured data.

Wrapping Up

In the end, we discussed the file writing operation on text files with the encoding and decoding schemes to ensure the safety and integrity of the data. Writing the tabular and structured data format maintained the readability and consistency in the file by organizing the information clearly understood by machine and human-friendly. Throughout the article, we also illustrated the different real-world use cases, where writing operation is used. So, by mastering the Writing operation of file handling in Python, you can ensure data persistence and manipulation effectively. I hope this guide is helpful for you. Happy Coding!

---