29. Interacting with the Operating System: sys and os

Python programs don't exist in isolation—they run on an operating system that manages files, directories, environment settings, and how programs start and communicate. The sys and os modules provide tools to interact with this environment, letting your programs accept command-line arguments, read environment variables, navigate the file system, and create or remove directories.

Understanding these modules transforms your Python scripts from isolated code into programs that can integrate with the broader system, respond to user input at startup, and manage files and folders programmatically.

29.1) Command-Line Arguments and Using sys.argv

When you run a Python script from the terminal or command prompt, you can pass additional information to it—these are called command-line arguments. For example:

bash

python greet.py Alice 25

Here, greet.py is the script name, and Alice and 25 are arguments passed to the program. The sys module provides access to these arguments through sys.argv, a list containing the script name and all arguments as strings.

29.1.1) What is sys.argv?

sys.argv is a list where:

sys.argv[0] is always the name of the script being run
sys.argv[1], sys.argv[2], etc., are the arguments passed after the script name
All elements are strings, even if they look like numbers

Let's create a simple script to see how this works:

python

# show_args.py
import sys
 
print("Script name:", sys.argv[0])
print("Number of arguments:", len(sys.argv))
print("All arguments:", sys.argv)

If you run this script with:

bash

python show_args.py hello world 123

Output:

Script name: show_args.py
Number of arguments: 4
All arguments: ['show_args.py', 'hello', 'world', '123']

Notice that sys.argv contains 4 elements: the script name plus three arguments. The number 123 is stored as the string '123', not as an integer.

29.1.2) Using Command-Line Arguments in Programs

Command-line arguments let users customize program behavior without modifying code. Here's a greeting program that uses the first argument as a name:

python

# greet.py
import sys
 
if len(sys.argv) < 2:
    print("Usage: python greet.py <name>")
    sys.exit(1)  # Exit with error code
 
name = sys.argv[1]
print(f"Hello, {name}!")

Running this:

bash

python greet.py Alice

Output:

Hello, Alice!

If you forget to provide a name:

bash

python greet.py

Output:

Usage: python greet.py <name>

The program checks if enough arguments were provided. If not, it prints a usage message and exits with sys.exit(1). The number 1 is an exit code—by convention, 0 means success and non-zero values indicate errors. This helps other programs or scripts detect if your program ran successfully.

29.1.3) Converting Arguments to Other Types

Since all arguments arrive as strings, you often need to convert them. Here's a program that calculates the area of a rectangle:

python

# area.py
import sys
 
if len(sys.argv) < 3:
    print("Usage: python area.py <width> <height>")
    sys.exit(1)
 
try:
    width = float(sys.argv[1])
    height = float(sys.argv[2])
except ValueError:
    print("Error: Width and height must be numbers")
    sys.exit(1)
 
area = width * height
print(f"Area: {area}")

Running this:

bash

python area.py 5.5 3.2

Output:

Area: 17.6

If the user provides invalid input:

bash

python area.py five three

Output:

Error: Width and height must be numbers

The try-except block (from Chapter 25) handles conversion errors gracefully, providing helpful feedback instead of crashing with a traceback.

29.2) Getting Interpreter and Runtime Information with sys

The sys module provides information about the Python interpreter itself and the runtime environment. This is useful for debugging, logging, or writing code that adapts to different Python versions or platforms.

29.2.1) Python Version Information

sys.version and sys.version_info tell you which Python version is running your code:

python

import sys
 
print("Python version string:", sys.version)
print("Version info:", sys.version_info)
print(f"Major version: {sys.version_info.major}")
print(f"Minor version: {sys.version_info.minor}")

Output (example):

Python version string: 3.11.4 (main, Jul  5 2023, 13:45:01) [GCC 11.2.0]
Version info: sys.version_info(major=3, minor=11, micro=4, releaselevel='final', serial=0)
Major version: 3
Minor version: 11

sys.version is a human-readable string, while sys.version_info is a named tuple you can compare programmatically:

python

import sys
 
if sys.version_info < (3, 8):
    print("This program requires Python 3.8 or higher")
    sys.exit(1)
 
print("Python version is compatible")

This ensures your program only runs on supported Python versions.

29.2.2) Platform Information

sys.platform identifies the operating system:

python

import sys
 
print("Platform:", sys.platform)
 
if sys.platform == "win32":
    print("Running on Windows")
elif sys.platform == "darwin":
    print("Running on macOS")
elif sys.platform.startswith("linux"):
    print("Running on Linux")
else:
    print("Running on another platform")

Output (on Linux):

Platform: linux
Running on Linux

This lets you write platform-specific code when necessary, such as using different file paths or system commands.

29.2.3) Module Search Path

sys.path is a list of directories Python searches when you import modules:

python

import sys
 
print("Module search paths:")
for path in sys.path:
    print(f"  {path}")

Output (example):

Module search paths:
  /home/user/projects
  /usr/lib/python3.11
  /usr/lib/python3.11/lib-dynload
  /home/user/.local/lib/python3.11/site-packages

The first entry is usually the directory containing your script. Python searches these paths in order when you use import. Understanding sys.path helps debug import errors or add custom module directories.

29.2.4) Exiting Programs with Exit Codes

We've seen sys.exit() used to stop programs. You can pass an exit code to indicate success or failure:

python

import sys
 
def process_data(filename):
    try:
        with open(filename) as f:
            data = f.read()
        # Process data...
        return True
    except FileNotFoundError:
        print(f"Error: File '{filename}' not found", file=sys.stderr)
        return False
 
if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python script.py <filename>", file=sys.stderr)
        sys.exit(1)
    
    success = process_data(sys.argv[1])
    sys.exit(0 if success else 1)

Exit codes follow Unix conventions:

0 means success
Non-zero values indicate different types of errors
Other programs can check these codes to determine if your program succeeded

29.3) Accessing Environment Variables with os

Environment variables are key-value pairs set by the operating system or user that programs can read. They're used for configuration, storing paths, API keys, and other settings that shouldn't be hardcoded.

29.3.1) Reading Environment Variables

The os module provides os.environ, a dictionary-like object containing all environment variables:

python

import os
 
# Get a specific environment variable
home = os.environ.get('HOME')  # On Unix/Linux/macOS
print(f"Home directory: {home}")
 
# Returns None if the variable doesn't exist
api_key = os.environ.get('MY_API_KEY')
print(f"API key: {api_key}")

Output (on Linux):

Home directory: /home/alice
API key: None

You can provide a default value:

python

import os
 
# Get environment variable with default
log_level = os.environ.get('LOG_LEVEL', 'INFO')
print(f"Log level: {log_level}")

Output (if LOG_LEVEL not set):

Log level: INFO

29.3.2) Common Environment Variables

Different operating systems provide standard environment variables:

python

import os
 
# Cross-platform variables
print("Path:", os.environ.get('PATH'))
 
# Unix/Linux/macOS
print("User:", os.environ.get('USER'))
print("Home:", os.environ.get('HOME'))
print("Shell:", os.environ.get('SHELL'))
 
# Windows
print("User", os.environ.get('USERNAME'))
print("User Profile:", os.environ.get('USERPROFILE'))
print("Temp:", os.environ.get('TEMP'))

PATH is particularly important—it lists directories where the system looks for executable programs. When you type a command like python, the system searches these directories.

29.3.3) Setting Environment Variables

You can modify environment variables for your program and any subprocesses it creates:

python

import os
 
# Set an environment variable
os.environ['MY_CONFIG'] = 'production'
 
# Read it back
print(os.environ.get('MY_CONFIG'))  # Output: production
 
# Delete an environment variable
del os.environ['MY_CONFIG']

Important: Changes to os.environ only affect the current Python process and any programs it launches. They don't persist after your program exits or affect other programs.

29.4) Working with File Paths and Directories (os.path, os.getcwd)

Managing file paths correctly is crucial for programs that work with files. The os and os.path modules provide tools to build, manipulate, and query paths in a platform-independent way.

29.4.1) Getting the Current Working Directory

The current working directory (CWD) is the folder your program considers its starting point for relative paths:

python

import os
 
cwd = os.getcwd()
print(f"Current working directory: {cwd}")

Output (example):

Current working directory: /home/alice/projects/myapp

When you open a file with a relative path like 'data.txt', Python looks in the current working directory. Understanding the CWD helps debug "file not found" errors.

29.4.2) Changing the Current Directory

You can change the working directory with os.chdir():

python

import os
 
original = os.getcwd()
print("Original directory:", original)
 
# Change to a different directory
os.chdir('/tmp')
print("New directory:", os.getcwd())
 
# Change back
os.chdir(original)
print("Back to:", os.getcwd())

Output:

Original directory: /home/alice/projects
New directory: /tmp
Back to: /home/alice/projects

Note: In Chapter 28, you learned about contextlib.chdir(), which automatically restores the original directory. For simple directory changes, prefer using the context manager:

python

from contextlib import chdir
 
with chdir('/tmp'):
    print("Temporarily in:", os.getcwd())
# Automatically restored

This ensures the directory is always restored, even if an error occurs.

29.4.3) Building Paths with os.path.join()

Different operating systems use different path separators:

Unix/Linux/macOS: / (forward slash)
Windows: \ (backslash)

os.path.join() builds paths correctly for the current platform:

python

import os
 
# Build a path to a file in a subdirectory
data_dir = 'data'
filename = 'users.txt'
filepath = os.path.join(data_dir, filename)
 
print(f"File path: {filepath}")

Output (on Unix/Linux/macOS):

File path: data/users.txt

Output (on Windows):

File path: data\users.txt

You can join multiple components:

python

import os
 
base = '/home/alice'
project = 'myapp'
subdir = 'data'
file = 'config.json'
 
full_path = os.path.join(base, project, subdir, file)
print(full_path)  # Output: /home/alice/myapp/data/config.json

Using os.path.join() makes your code portable across operating systems.

29.4.4) Checking if Paths Exist

Before working with files or directories, check if they exist:

python

import os
 
path = 'data.txt'
 
if os.path.exists(path):
    print(f"'{path}' exists")
else:
    print(f"'{path}' does not exist")

You can also check specifically for files or directories:

python

import os
 
path = 'mydir'
 
if os.path.isfile(path):
    print(f"'{path}' is a file")
elif os.path.isdir(path):
    print(f"'{path}' is a directory")
elif os.path.exists(path):
    print(f"'{path}' exists but is neither a file nor directory")
else:
    print(f"'{path}' does not exist")

These checks prevent errors when trying to open non-existent files or list non-existent directories.

29.4.5) Getting Absolute Paths

os.path.abspath() converts relative paths to absolute paths:

python

import os
 
relative_path = 'data/users.txt'
absolute_path = os.path.abspath(relative_path)
 
print(f"Relative: {relative_path}")
print(f"Absolute: {absolute_path}")

Output (example):

Relative: data/users.txt
Absolute: /home/alice/projects/myapp/data/users.txt

This is useful for logging, error messages, or when you need to know the exact location of a file.

29.4.6) Splitting Paths into Components

os.path.split() separates a path into directory and filename:

python

import os
 
path = '/home/alice/projects/data.txt'
directory, filename = os.path.split(path)
 
print(f"Directory: {directory}")
print(f"Filename: {filename}")

Output:

Directory: /home/alice/projects
Filename: data.txt

os.path.basename() gets just the filename, and os.path.dirname() gets just the directory:

python

import os
 
path = '/home/alice/projects/data.txt'
 
print(f"Basename: {os.path.basename(path)}")  # Output: data.txt
print(f"Dirname: {os.path.dirname(path)}")    # Output: /home/alice/projects

29.4.7) Splitting File Extensions

os.path.splitext() separates the filename from its extension:

python

import os
 
filename = 'report.pdf'
name, extension = os.path.splitext(filename)
 
print(f"Name: {name}")        # Output: report
print(f"Extension: {extension}")  # Output: .pdf

This is useful for processing files based on their type:

python

import os
 
files = ['data.csv', 'image.png', 'document.txt', 'script.py']
 
for file in files:
    name, ext = os.path.splitext(file)
    if ext == '.csv':
        print(f"Process CSV file: {file}")
    elif ext == '.png':
        print(f"Process image file: {file}")

Output:

Process CSV file: data.csv
Process image file: image.png

29.5) Listing, Creating, and Removing Files and Directories

The os module provides functions to manipulate the file system: listing directory contents, creating new directories, and removing files and folders.

29.5.1) Listing Directory Contents

os.listdir() returns a list of all items in a directory:

python

import os
 
# List contents of current directory
contents = os.listdir('.')
print("Current directory contents:")
for item in contents:
    print(f"  {item}")

Output (example):

Current directory contents:
  script.py
  data.txt
  mydir
  README.md

The list includes both files and directories. To distinguish between them:

python

import os
 
contents = os.listdir('.')
print("Files:")
for item in contents:
    if os.path.isfile(item):
        print(f"  {item}")
 
print("\nDirectories:")
for item in contents:
    if os.path.isdir(item):
        print(f"  {item}")

Output:

Files:
  script.py
  data.txt
  README.md
 
Directories:
  mydir

29.5.2) Creating Directories

os.mkdir() creates a single directory:

python

import os
 
new_dir = 'output'
 
if not os.path.exists(new_dir):
    os.mkdir(new_dir)
    print(f"Created directory: {new_dir}")
else:
    print(f"Directory already exists: {new_dir}")

Output:

Created directory: output

Important: os.mkdir() fails if the parent directory doesn't exist. For example, trying to create 'data/output' when 'data' doesn't exist will raise an error.

os.makedirs() creates all necessary parent directories:

python

import os
 
nested_dir = 'data/processed/2024'
 
if not os.path.exists(nested_dir):
    os.makedirs(nested_dir)
    print(f"Created directory structure: {nested_dir}")

Output:

Created directory structure: data/processed/2024

This creates data, then data/processed, then data/processed/2024 if they don't exist.

29.5.3) Removing Files

os.remove() deletes a file:

python

import os
 
filename = 'temp.txt'
 
# Create a temporary file
with open(filename, 'w') as f:
    f.write('Temporary data')
 
print(f"File exists: {os.path.exists(filename)}")  # Output: True
 
# Remove the file
os.remove(filename)
print(f"File exists: {os.path.exists(filename)}")  # Output: False

Warning: os.remove() permanently deletes files—they don't go to a recycle bin.

29.5.4) Removing Directories

os.rmdir() removes an empty directory:

python

import os
 
directory = 'empty_dir'
 
# Create and then remove an empty directory
os.mkdir(directory)
print(f"Created: {directory}")
 
os.rmdir(directory)
print(f"Removed: {directory}")

os.rmdir() fails if the directory contains files. To remove a directory and all its contents, you need to delete files first:

python

import os
 
def remove_directory_contents(directory):
    """Remove all files in a directory, then remove the directory.
    
    Note: Fails if the directory contains subdirectories.
    """
    if not os.path.exists(directory):
        print(f"Directory does not exist: {directory}")
        return
    
    # Remove all files in the directory
    for item in os.listdir(directory):
        item_path = os.path.join(directory, item)
        if os.path.isfile(item_path):
            os.remove(item_path)
            print(f"Removed file: {item_path}")
    
    # Remove the now-empty directory
    os.rmdir(directory)
    print(f"Removed directory: {directory}")
 
# Example usage
test_dir = 'test_data'
os.makedirs(test_dir, exist_ok=True)
 
# Create some test files
with open(os.path.join(test_dir, 'file1.txt'), 'w') as f:
    f.write('test')
with open(os.path.join(test_dir, 'file2.txt'), 'w') as f:
    f.write('test')
 
# Remove everything
remove_directory_contents(test_dir)

Output:

Removed file: test_data/file1.txt
Removed file: test_data/file2.txt
Removed directory: test_data

Note: For more complex directory removal (including subdirectories), Python's shutil module provides shutil.rmtree(), but that's beyond our current scope.

29.5.5) Renaming Files and Directories

os.rename() renames or moves files and directories:

python

import os
 
# Rename a file
old_name = 'draft.txt'
new_name = 'final.txt'
 
# Create test file
with open(old_name, 'w') as f:
    f.write('content')
 
os.rename(old_name, new_name)
print(f"Renamed '{old_name}' to '{new_name}'")

You can also move files to different directories:

python

import os
 
# Create directories and file
os.makedirs('source', exist_ok=True)
os.makedirs('destination', exist_ok=True)
 
with open('source/file.txt', 'w') as f:
    f.write('content')
 
# Move file to different directory
os.rename('source/file.txt', 'destination/file.txt')
print("Moved file to destination directory")

The sys and os modules give your Python programs the ability to interact with the operating system, accept command-line input, read configuration, and manage files and directories. These capabilities transform simple scripts into powerful command-line tools that integrate seamlessly with the broader system environment.