Basic Usage
This notebook demonstrates the basic features of iops-profiler, including line magic and cell magic commands.
Loading the Extension
First, we need to load the iops-profiler extension. You only need to do this once per notebook session.
[1]:
%load_ext iops_profiler
Matplotlib is building the font cache; this may take a moment.
Line Magic: %iops
The %iops line magic is used to profile a single line of code. Let’s start with a simple example of writing data to a file.
[2]:
import tempfile
import os
# Create a temporary directory for our test files
test_dir = tempfile.mkdtemp()
test_file = os.path.join(test_dir, "test.txt")
print(f"Test directory: {test_dir}")
Test directory: /tmp/tmp4hvfd0vt
[3]:
# Profile a simple write operation
%iops open(test_file, 'w').write('Hello World' * 1000)
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
[3]:
11000
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0035 seconds |
| Read Operations | 6 |
| Write Operations | 5 |
| Total Operations | 11 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 12.00 KB (12,288 bytes) |
| Total Bytes | 12.00 KB (12,288 bytes) |
| IOPS | 3168.34 operations/second |
| Throughput | 3.38 MB/second |
The output shows:
Time: How long the operation took
Write Ops: Number of write operations
Bytes Written: Total bytes written to disk
Write IOPS: Write operations per second
Write Throughput: Bytes written per second
Cell Magic: %%iops
The %%iops cell magic allows you to profile an entire cell of code. This is useful for more complex operations.
[4]:
%%iops
# Write multiple lines to a file
output_file = os.path.join(test_dir, 'output.txt')
with open(output_file, 'w') as f:
for i in range(100):
f.write(f'Line {i}: ' + 'data' * 10 + '\n')
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0007 seconds |
| Read Operations | 2 |
| Write Operations | 1 |
| Total Operations | 3 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 8.00 KB (8,192 bytes) |
| Total Bytes | 8.00 KB (8,192 bytes) |
| IOPS | 4276.99 operations/second |
| Throughput | 11.14 MB/second |
Reading Files
Now let’s profile read operations. First, we’ll create a file with some data.
[5]:
# Create a file with test data
data_file = os.path.join(test_dir, "data.txt")
with open(data_file, "w") as f:
f.write("test data " * 10000)
[6]:
%%iops
# Profile reading the file
with open(data_file, 'r') as f:
content = f.read()
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0004 seconds |
| Read Operations | 4 |
| Write Operations | 0 |
| Total Operations | 4 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 0.00 B (0 bytes) |
| Total Bytes | 0.00 B (0 bytes) |
| IOPS | 9771.24 operations/second |
| Throughput | 0.00 B/second |
Notice the results now include:
Read Ops: Number of read operations
Bytes Read: Total bytes read from disk
Read IOPS: Read operations per second
Read Throughput: Bytes read per second
Comparing Different Approaches
One of the most useful applications of iops-profiler is comparing different I/O strategies. Let’s compare writing data in small chunks vs. large chunks.
[7]:
%%iops
# Strategy 1: Many small writes
small_chunks_file = os.path.join(test_dir, 'small_chunks.txt')
with open(small_chunks_file, 'w') as f:
for i in range(1000):
f.write('x' * 10)
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0007 seconds |
| Read Operations | 2 |
| Write Operations | 2 |
| Total Operations | 4 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 12.00 KB (12,288 bytes) |
| Total Bytes | 12.00 KB (12,288 bytes) |
| IOPS | 5803.26 operations/second |
| Throughput | 17.00 MB/second |
[8]:
%%iops
# Strategy 2: One large write
large_chunk_file = os.path.join(test_dir, 'large_chunk.txt')
with open(large_chunk_file, 'w') as f:
data = 'x' * 10000
f.write(data)
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0005 seconds |
| Read Operations | 2 |
| Write Operations | 1 |
| Total Operations | 3 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 12.00 KB (12,288 bytes) |
| Total Bytes | 12.00 KB (12,288 bytes) |
| IOPS | 5565.20 operations/second |
| Throughput | 21.74 MB/second |
Compare the two approaches:
Which has better throughput?
How do the number of operations differ?
Which approach is more efficient for your use case?
Binary File Operations
iops-profiler works with binary files too. Let’s test with binary data.
[9]:
%%iops
# Write binary data
binary_file = os.path.join(test_dir, 'binary.dat')
with open(binary_file, 'wb') as f:
data = bytes(range(256)) * 100
f.write(data)
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0006 seconds |
| Read Operations | 2 |
| Write Operations | 1 |
| Total Operations | 3 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 28.00 KB (28,672 bytes) |
| Total Bytes | 28.00 KB (28,672 bytes) |
| IOPS | 5061.51 operations/second |
| Throughput | 46.13 MB/second |
[10]:
%%iops
# Read binary data
with open(binary_file, 'rb') as f:
binary_content = f.read()
⚠️ Could not use strace: [Errno 2] No such file or directory: 'strace'
Falling back to psutil per-process measurement.
| IOPS Profile Results (psutil (per-process)) | |
| Execution Time | 0.0004 seconds |
| Read Operations | 4 |
| Write Operations | 0 |
| Total Operations | 4 |
| Bytes Read | 0.00 B (0 bytes) |
| Bytes Written | 0.00 B (0 bytes) |
| Total Bytes | 0.00 B (0 bytes) |
| IOPS | 11052.18 operations/second |
| Throughput | 0.00 B/second |
Cleanup
Finally, let’s clean up our temporary files.
[11]:
import shutil
shutil.rmtree(test_dir)
print("Cleanup complete!")
Cleanup complete!
Summary
In this notebook, we covered:
Loading the iops-profiler extension
Using
%iopsline magic for single-line profilingUsing
%%iopscell magic for multi-line profilingProfiling read and write operations
Comparing different I/O strategies
Working with binary files
Next steps:
Try the histogram visualization notebook to see operation distributions
Explore advanced usage with real-world data
Apply iops-profiler to your own I/O-intensive code