27. Reading from and Writing to Files

Until now, all the data in our programs has been temporary—stored in variables that disappear when the program ends. To create programs that remember information between runs, we need to work with files. Files let us save data to disk and read it back later, enabling everything from configuration settings to user data storage.

In this chapter, you'll learn how to read from and write to text files in Python. We'll start with understanding file paths and how Python locates files, then move through opening, reading, writing, and properly closing files. You'll learn about different file modes, text encoding, and how to handle common errors that occur during file operations.

27.1) File Paths and the Current Working Directory

Before we can work with files, we need to understand how Python locates them on your computer's file system.

27.1.1) Understanding File Paths

A file path is the address of a file on your computer. It tells Python exactly where to find a file. There are two types of paths:

Absolute paths specify the complete location from the root of the file system:

• Windows: C:\Users\Alice\Documents\data.txt
• macOS/Linux: /home/alice/documents/data.txt

Relative paths specify a location relative to the current working directory:

• data.txt (file in the current directory)
• reports/sales.txt (file in a subdirectory)
• ../config.txt (file in the parent directory)

The current working directory (CWD) is the folder where Python looks for files when you use relative paths. When you run a Python script, the CWD is the directory from which you ran the command, not necessarily where the script file is located.

For example:

# Directory structure:
/home/alice/
  └── projects/
      └── script.py
 
# Running from the projects folder:
$ cd /home/alice/projects
$ python script.py
# CWD is /home/alice/projects
 
# Running from the parent folder:
$ cd /home/alice
$ python projects/script.py
# CWD is /home/alice (not where script.py is!)

You can check the current working directory with:

import os
 
# Get the current working directory
current_dir = os.getcwd()
print(f"Current directory: {current_dir}")

Output:

Current directory: /home/alice/projects/file_demo

The os.getcwd() function returns the absolute path of the current working directory. This is useful for understanding where Python will look for files when you use relative paths.

Understanding the current working directory is important because it determines where Python will look when you use a relative path like "data.txt". If you run your script from /home/alice/projects/ and open "data.txt", Python looks for /home/alice/projects/data.txt.

27.1.2) Using Relative Paths Effectively

When working with files, relative paths are often more convenient than absolute paths because they make your code portable—it works regardless of where the project folder is located on different computers.

Here are common relative path patterns:

# File in the current working directory
filename = "student_grades.txt"
 
# File in a subdirectory (data folder within current directory)
filename = "data/student_grades.txt"
 
# File in the parent directory
filename = "../shared_data.txt"

For this chapter's examples, we'll primarily use simple filenames like "data.txt". This means:

• The data files should be in the same directory where you run the Python command
• If your script is in /home/alice/projects/ and you run python script.py from that directory, Python will look for data.txt in /home/alice/projects/

This approach keeps examples clear and focused on file operations rather than path navigation. If you get a FileNotFoundError, use os.getcwd() to check where Python is looking for files.

27.1.3) Path Separators Across Operating Systems

Different operating systems use different characters to separate directories in paths:

• Windows uses backslashes: data\reports\sales.txt
• macOS and Linux use forward slashes: data/reports/sales.txt

Python handles this automatically when you use forward slashes in your code—they work on all operating systems:

# This works on Windows, macOS, and Linux
filename = "data/reports/sales.txt"

Python converts the forward slashes to the appropriate separator for your operating system.

27.2) Opening and Closing Files

To work with a file, we must first open it, which creates a connection between our program and the file on disk. When we're done, we must close it to release system resources and ensure all data is properly saved.

27.2.1) The open() Function

The open() function creates a file object that represents the connection to a file. At its simplest, you provide the filename:

# Open a file for reading (the default mode)
file = open("message.txt")

This opens the file message.txt in the current directory. The open() function returns a file object, which we store in the variable file. This object provides methods for reading from or writing to the file.

However, this basic approach has a critical problem: if an error occurs after opening the file, the file might never be closed. Let's see why closing files matters.

27.2.2) Why Closing Files Is Important

When you open a file, the operating system allocates resources to maintain that connection. If you don't close the file:

• Data might not be saved: When writing to files, data is often buffered in memory and only written to disk when the file is closed
• System resources are wasted: Each open file consumes memory and file handles
• Other programs might be blocked: Some systems prevent other programs from accessing a file that's already open

To close a file, call its close() method:

file = open("message.txt")
# ... work with the file ...
file.close()  # Release resources and ensure data is saved

27.2.3) The Problem with Manual Closing

Manually closing files is error-prone. If an exception occurs between opening and closing, the close() call might never execute:

file = open("data.txt")
result = process_data(file)  # If this raises an exception...
file.close()  # ...this never runs!

This is such a common problem that Python provides a better solution: the with statement, which we'll learn about in Section 27.4. For now, understand that opening and closing files manually requires careful attention to ensure close() is always called.

27.2.4) Checking If a File Is Open

A file object has a closed attribute that tells you whether the file is closed:

file = open("data.txt")
print(file.closed)  # Output: False
 
file.close()
print(file.closed)  # Output: True

Once a file is closed, attempting to read from or write to it will raise an error:

file = open("data.txt")
file.close()
 
# This raises ValueError: I/O operation on closed file
content = file.read()

The error message clearly indicates the problem: you're trying to perform an I/O (input/output) operation on a file that's already closed.

27.3) Understanding File Modes (r, w, a, text vs binary) and Encoding

When opening a file, you can specify a mode that determines what operations are allowed and how the file is treated. Understanding modes is crucial for working with files correctly.

27.3.1) Text Mode vs Binary Mode

Files can be opened in two fundamental modes:

Text mode (the default) treats the file as containing text. Python automatically:

• Converts line endings to \n regardless of the platform
• Handles text encoding (converting between bytes and strings)
• Allows reading and writing strings

Binary mode treats the file as raw bytes. Python:

• Reads and writes bytes objects, not strings
• Makes no conversions or interpretations
• Is used for images, audio, video, and other non-text files

For this chapter, we'll focus on text mode, which is what you'll use most often. Binary mode is indicated by adding 'b' to the mode string (like 'rb' or 'wb'), but we won't need it for text files.

27.3.2) The Three Primary File Modes

Python provides three primary modes for opening text files:

Read mode ('r') - Opens a file for reading only:

file = open("data.txt", "r")  # or just open("data.txt")

• The file must already exist, or Python raises FileNotFoundError
• You can read from the file but cannot write to it
• This is the default mode if you don't specify one

Write mode ('w') - Opens a file for writing:

file = open("output.txt", "w")

• Creates the file if it doesn't exist
• Erases all existing content if the file already exists
• You can write to the file but cannot read from it
• Use this when you want to create a new file or completely replace an existing one

Append mode ('a') - Opens a file for appending:

file = open("log.txt", "a")

• Creates the file if it doesn't exist
• Preserves existing content and adds new content at the end
• You can write to the file but cannot read from it
• Use this when you want to add to an existing file without losing its current content

Here's a comparison of how these modes affect an existing file:

# Suppose data.txt contains: "Hello\nWorld\n"
 
# Read mode - content unchanged
file = open("data.txt", "r")
file.close()
# File still contains: "Hello\nWorld\n"
 
# Write mode - content erased
file = open("data.txt", "w")
file.write("New content\n")
file.close()
# File now contains: "New content\n"
 
# Append mode - content preserved, new content added
file = open("data.txt", "a")
file.write("Added line\n")
file.close()
# File now contains: "New content\nAdded line\n"

27.3.3) Understanding Text Encoding

When you work with text files, Python needs to know how to convert between the bytes stored on disk and the string characters in your program. This conversion process is called encoding.

The most common encoding is UTF-8, which can represent any character from any language. It's the default encoding in Python 3 and the standard for modern text files.

# Explicitly specify UTF-8 encoding (though it's usually the default)
file = open("data.txt", "r", encoding="utf-8")

Why does encoding matter? Consider this text file containing a name with an accented character:

# Writing a file with special characters
file = open("names.txt", "w", encoding="utf-8")
file.write("José\n")
file.write("François\n")
file.write("Müller\n")
file.close()
 
# Reading it back
file = open("names.txt", "r", encoding="utf-8")
content = file.read()
file.close()
print(content)

Output:

José
François
Müller

If you try to open a file with the wrong encoding, you might see garbled characters or get an error. Always use UTF-8 for new files unless you have a specific reason to use a different encoding.

27.3.4) Additional Mode Variations

Python provides additional mode characters that can be combined with the primary modes:

Plus modes allow both reading and writing:

• 'r+' - Read and write (file must exist)
• 'w+' - Write and read (erases existing content)
• 'a+' - Append and read (preserves existing content)

For beginners, it's clearer to open a file once for reading and separately for writing rather than using plus modes. We'll stick with the simple modes ('r', 'w', 'a') in this chapter.

27.4) Using with to Manage Files Automatically

The with statement provides a cleaner, safer way to work with files. It automatically closes the file when you're done, even if an error occurs.

27.4.1) The with Statement Syntax

Here's how to use with to open a file:

with open("data.txt", "r") as file:
    content = file.read()
    print(content)
# File is automatically closed here

The syntax has several parts:

• with - Keyword that starts the context manager
• open("data.txt", "r") - Opens the file
• as file - Creates a variable to reference the file object
• : - Begins the indented block
• Indented block - Code that works with the file
• After the block - File is automatically closed

The key benefit: Python guarantees the file will be closed when the with block ends, regardless of how it ends (normally, with a return, or due to an exception).

27.4.2) Why with Is Better Than Manual Closing

Compare these two approaches:

# Manual closing - risky
file = open("data.txt", "r")
content = file.read()
result = process(content)  # If this raises an exception...
file.close()  # ...this never runs
 
# Using with - safe
with open("data.txt", "r") as file:
    content = file.read()
    result = process(content)  # Even if this raises an exception...
# ...the file is still closed automatically

The with statement uses Python's context manager protocol (which we'll explore in detail in Chapter 28). For now, think of it as a guarantee: "I'll clean up this resource when you're done, no matter what happens."

27.4.3) Working with Multiple Files

You can open multiple files in a single with statement:

with open("input.txt", "r") as infile, open("output.txt", "w") as outfile:
    content = infile.read()
    outfile.write(content.upper())
# Both files are automatically closed here

This is useful when you need to read from one file and write to another simultaneously. Both files are guaranteed to be closed properly, even if an error occurs.

27.4.4) The File Object Is Only Valid Inside the with Block

Once the with block ends, the file is closed and you can no longer use the file object:

with open("data.txt", "r") as file:
    content = file.read()
    print("Inside with block:", file.closed)  # Output: Inside with block: False
 
print("Outside with block:", file.closed)  # Output: Outside with block: True
 
# This raises ValueError: I/O operation on closed file
more_content = file.read()

This behavior is intentional—it prevents you from accidentally using a closed file. If you need the file's content outside the with block, store it in a variable (like content above) before the block ends.

From this point forward in the chapter, we'll use with for all file operations. It's the recommended approach and what you should use in your own code.

27.5) Reading Text Files

Now that we understand how to open files safely with with, let's explore the different ways to read content from text files.

27.5.1) Reading the Entire File with read()

The read() method reads the entire file content as a single string:

with open("message.txt", "r") as file:
    content = file.read()
    print(content)

If message.txt contains:

Welcome to Python!
This is a text file.
It has multiple lines.

Output:

Welcome to Python!
This is a text file.
It has multiple lines.

The read() method includes all newline characters (\n) from the file. When you print the string, Python displays each line on a separate line because of these newline characters.

You can also read a specific number of characters by passing a number to read():

with open("message.txt", "r") as file:
    first_ten = file.read(10)  # Read first 10 characters
    print(f"First 10 characters: '{first_ten}'")

Output:

First 10 characters: 'Welcome to'

Reading the entire file is simple and works well for small files. However, for large files (megabytes or gigabytes), reading everything at once can consume too much memory. For those cases, reading line by line is more efficient.

27.5.2) Reading Line by Line with readline()

The readline() method reads a single line from the file, including the newline character at the end:

with open("message.txt", "r") as file:
    line1 = file.readline()
    line2 = file.readline()
    line3 = file.readline()
    print(f"Line 1: {line1}")
    print(f"Line 2: {line2}")
    print(f"Line 3: {line3}")

Output:

Line 1: Welcome to Python!
 
Line 2: This is a text file.
 
Line 3: It has multiple lines.
 

Notice the extra blank lines in the output. Each line read from the file ends with \n, and print() adds another newline. To avoid this, use the rstrip() method to remove trailing whitespace:

with open("message.txt", "r") as file:
    line1 = file.readline().rstrip()
    line2 = file.readline().rstrip()
    print(f"Line 1: {line1}")
    print(f"Line 2: {line2}")

Output:

Line 1: Welcome to Python!
Line 2: This is a text file.

When readline() reaches the end of the file, it returns an empty string "". This lets you detect when there are no more lines to read:

with open("message.txt", "r") as file:
    while True:
        line = file.readline()
        if line == "":  # End of file
            break
        print(line.rstrip())

However, there's a more Pythonic way to read files line by line.

27.5.3) Iterating Over Lines with a for Loop

File objects are iterable, which means you can loop over them directly with a for loop. This is the most common and Pythonic way to read a file line by line:

with open("message.txt", "r") as file:
    for line in file:
        print(line.rstrip())

Output:

Welcome to Python!
This is a text file.
It has multiple lines.

This approach is:

• Cleaner: No need for readline() or checking for empty strings
• More efficient: Python reads the file in chunks, not loading the entire file into memory
• More Pythonic: Uses iteration, which is a core Python concept

Each iteration of the loop reads the next line from the file. When there are no more lines, the loop ends automatically.

27.5.4) Reading All Lines into a List with readlines()

The readlines() method reads all lines from the file and returns them as a list of strings:

with open("message.txt", "r") as file:
    lines = file.readlines()
 
print(f"Number of lines: {len(lines)}")
for i, line in enumerate(lines, start=1):
    print(f"Line {i}: {line.rstrip()}")

Output:

Number of lines: 3
Line 1: Welcome to Python!
Line 2: This is a text file.
Line 3: It has multiple lines.

Each element in the list is a string containing one line from the file, including its newline character. This is useful when you need to:

• Access lines by index: lines[0], lines[1], etc.
• Process lines multiple times
• Know the total number of lines before processing

However, like read(), readlines() loads the entire file into memory. For large files, iterating with a for loop is more memory-efficient.

27.6) Writing and Appending to Text Files

Writing to files is just as important as reading from them. Python provides simple methods to create new files or modify existing ones.

27.6.1) Writing to a File with write()

To write to a file, open it in write mode ('w') and use the write() method:

with open("output.txt", "w") as file:
    file.write("Hello, World!\n")
    file.write("This is a new file.\n")

After running this code, output.txt contains:

Hello, World!
This is a new file.

Important points about write():

• It writes a string to the file
• It does not automatically add newline characters—you must include \n yourself
• It returns the number of characters written (though we usually ignore this)
• If the file already exists, write mode erases all existing content before writing

Let's see what happens when we write to an existing file:

# First, create a file with some content
with open("demo.txt", "w") as file:
    file.write("Original content\n")
 
# Now open it again in write mode
with open("demo.txt", "w") as file:
    file.write("New content\n")
 
# Read the file to see what it contains
with open("demo.txt", "r") as file:
    print(file.read())

Output:

New content

The original content is gone. Write mode always starts with an empty file, whether it's creating a new file or overwriting an existing one.

27.6.2) Writing Multiple Lines

You can call write() multiple times to write multiple lines:

with open("shopping_list.txt", "w") as file:
    file.write("Apples\n")
    file.write("Bananas\n")
    file.write("Oranges\n")

27.6.3) Writing Data from Collections

A common task is writing data from lists or other collections to a file:

students = ["Alice", "Bob", "Carol", "David"]
 
with open("students.txt", "w") as file:
    for student in students:
        file.write(student + "\n")

This creates students.txt containing:

Alice
Bob
Carol
David

27.6.4) Appending to a File

When you want to add content to the end of an existing file without erasing what's already there, use append mode ('a'):

# Create a file with initial content
with open("log.txt", "w") as file:
    file.write("Program started\n")
 
# Later, append more content
with open("log.txt", "a") as file:
    file.write("Processing data\n")
    file.write("Processing complete\n")
 
# Read the file to see all content
with open("log.txt", "r") as file:
    print(file.read())

Output:

Program started
Processing data
Processing complete

Append mode is perfect for log files, where you want to keep a running record of events. Each time you open the file in append mode, new content is added at the end, preserving everything that was already there.

27.7) Handling Common File I/O Errors

File operations can fail for many reasons: the file doesn't exist, you don't have permission to access it, the disk is full, or the file is already open by another program. Learning to handle these errors gracefully makes your programs more robust and user-friendly.

27.7.1) FileNotFoundError: When a File Doesn't Exist

The most common file error occurs when you try to read a file that doesn't exist:

# WARNING: This will raise FileNotFoundError if data.txt doesn't exist
with open("data.txt", "r") as file:
    content = file.read()

If data.txt doesn't exist, Python raises:

FileNotFoundError: [Errno 2] No such file or directory: 'data.txt'

To handle this gracefully, use a try-except block (as we learned in Chapter 25):

try:
    with open("data.txt", "r") as file:
        content = file.read()
        print(content)
except FileNotFoundError:
    print("Error: The file 'data.txt' was not found.")
    print("Please check the filename and try again.")

Output (if file doesn't exist):

Error: The file 'data.txt' was not found.
Please check the filename and try again.

This approach prevents your program from crashing and provides a helpful message to the user.

27.7.2) PermissionError: When You Can't Access a File

Sometimes you don't have permission to read or write a file:

try:
    with open("/root/protected.txt", "r") as file:
        content = file.read()
except PermissionError:
    print("Error: You don't have permission to access this file.")

Permission errors can occur when:

• The file is owned by another user
• The file is in a protected system directory
• The file is marked as read-only and you're trying to write to it
• On Windows, the file is open in another program

27.7.3) IsADirectoryError: When You Try to Open a Directory

If you accidentally try to open a directory instead of a file:

try:
    with open("my_folder", "r") as file:
        content = file.read()
except IsADirectoryError:
    print("Error: 'my_folder' is a directory, not a file.")

This can happen when you have both a file and a directory with similar names, or when you forget to include the filename in a path.

27.7.4) UnicodeDecodeError: When Encoding Doesn't Match

If you try to read a file with the wrong encoding, you might get a UnicodeDecodeError:

try:
    with open("data.txt", "r", encoding="utf-8") as file:
        content = file.read()
except UnicodeDecodeError:
    print("Error: The file encoding doesn't match UTF-8.")
    print("The file might use a different encoding.")

This error occurs when the file contains bytes that aren't valid UTF-8. If you encounter this, the file might be:

• Using a different encoding (like Latin-1 or Windows-1252)
• A binary file that you're trying to read as text
• Corrupted

27.7.5) Handling Multiple Error Types

You can catch multiple error types in a single try-except block:

filename = input("Enter filename: ")
 
try:
    with open(filename, "r") as file:
        content = file.read()
        print(content)
except FileNotFoundError:
    print(f"Error: '{filename}' does not exist.")
except PermissionError:
    print(f"Error: You don't have permission to read '{filename}'.")
except IsADirectoryError:
    print(f"Error: '{filename}' is a directory, not a file.")
except UnicodeDecodeError:
    print(f"Error: '{filename}' contains invalid text encoding.")

This provides specific, helpful error messages for each type of problem. The user knows exactly what went wrong and can take appropriate action.

27.7.6) Using a Catch-All Exception Handler

Sometimes you want to catch any unexpected file-related error beyond the specific types we've covered. You can use a general Exception handler as a catch-all after your specific handlers:

filename = input("Enter filename: ")
 
try:
    with open(filename, "r") as file:
        content = file.read()
        print(content)
except FileNotFoundError:
    print(f"Error: '{filename}' not found.")
except PermissionError:
    print(f"Error: No permission to read '{filename}'.")
except IsADirectoryError:
    print(f"Error: '{filename}' is a directory.")
except UnicodeDecodeError:
    print(f"Error: '{filename}' has invalid encoding.")
except Exception as e:
    print(f"Unexpected error reading file: {e}")

This ensures your program handles even errors you didn't anticipate. The variable e contains the exception object, which includes a descriptive error message. Printing it gives the user technical details about what went wrong.

Working with files is a fundamental skill in programming. You've learned how to:

• Understand file paths and the current working directory
• Open and close files properly
• Use different file modes for reading, writing, and appending
• Manage files automatically with the with statement
• Read files using various methods (read(), readline(), iteration, readlines())
• Write and append content to files
• Handle common file I/O errors gracefully

These skills enable you to create programs that persist data between runs, process text files, generate reports, and much more. In the next chapter, we'll explore context managers in depth, understanding the mechanism that makes the with statement work and how to create your own context managers for managing resources beyond files.