Optimizing Python code without profiling first is guesswork dressed up as engineering — the function you’re sure is slow is often fine, and the one you never suspected is where 80% of the runtime actually goes. Profiling tells you where time is really spent before you touch a single line; benchmarking tells you, precisely, whether your change actually helped. This post walks through both, using the tools you’d realistically reach for on a real codebase.

Benchmarking a Single Piece of Code with timeit

Before profiling a whole application, you often just need to know: is version A faster than version B? Python’s timeit module exists specifically to answer that question correctly — it disables garbage collection during the run and executes the code many times to average out noise, which a naive time.time() wrapper around a single call does not do.

import timeit

def concat_with_plus(items):
    result = ""
    for item in items:
        result += item
    return result

def concat_with_join(items):
    return "".join(items)

items = ["word"] * 10_000

plus_time = timeit.timeit(lambda: concat_with_plus(items), number=100)
join_time = timeit.timeit(lambda: concat_with_join(items), number=100)

print(f"'+=' concat:  {plus_time:.4f}s")
print(f"''.join():    {join_time:.4f}s")

'+=' concat:  0.8241s
''.join():    0.0159s

The number=100 argument runs each function 100 times and returns the total, not the per-call time — divide by number if you want an average per call. The result here is the textbook case for why str.join() exists: += concatenation rebuilds a new string object on every iteration, while join() allocates once.

From the command line, the same tool is available without writing a script:

$ python -m timeit -s "items = ['word'] * 10000" "''.join(items)"
5000 loops, best of 5: 15.9 usec per loop

Function-Level Profiling with cProfile

timeit answers “which of these two snippets is faster,” but for a real application the question is usually “where in this entire call graph is the time going.” cProfile is Python’s built-in answer, and it instruments every function call with no code changes required.

import cProfile

def fetch_data():
    return list(range(100_000))

def transform(data):
    return [x ** 2 for x in data if x % 2 == 0]

def summarize(data):
    return sum(data) / len(data)

def run_pipeline():
    data = fetch_data()
    transformed = transform(data)
    return summarize(transformed)

cProfile.run("run_pipeline()")

         100008 function calls in 0.031 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.006    0.006    0.031    0.031 <string>:1(run_pipeline)
        1    0.004    0.004    0.004    0.004 pipeline.py:3(fetch_data)
        1    0.012    0.012    0.012    0.012 pipeline.py:6(transform)
        1    0.000    0.000    0.000    0.000 pipeline.py:11(summarize)

tottime is time spent in the function itself, excluding calls to other functions; cumtime includes time spent in everything it calls. For finding the actual bottleneck, tottime is usually the more useful column — here, transform dominates at 0.012s of self-time, which tells you exactly which function to optimize first, rather than guessing.

For a real application, running via the command line and sorting by cumulative time is the more common workflow:

$ python -m cProfile -s cumulative myapp.py

Line-by-Line Profiling with line_profiler

cProfile tells you which function is slow, but not which line inside that function. For that, line_profiler is the standard tool — it requires decorating the function you want to inspect and running under a separate command.

$ pip install line_profiler

# transform.py
@profile  # provided by kernprof at runtime, not a real import
def transform(data):
    result = []
    for x in data:
        if x % 2 == 0:
            result.append(x ** 2)
    return result

if __name__ == "__main__":
    transform(list(range(1_000_000)))

$ kernprof -l -v transform.py

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           def transform(data):
     3         1          1.0      1.0      0.0      result = []
     4   1000001     98234.0      0.1     22.4      for x in data:
     5   1000000    142891.0      0.1     32.6          if x % 2 == 0:
     6    500000    196781.0      0.4     45.0              result.append(x ** 2)
     7         1          0.5      0.5      0.0      return result

This level of detail is what turns “this function is slow” into “45% of this function’s time is the append and exponentiation on line 6” — precise enough to know that rewriting this as a list comprehension (which avoids the repeated append call overhead) is the actual fix, not a vague guess at optimization.

Sampling Profilers for Production: py-spy

cProfile and line_profiler both instrument every call, which adds real overhead — fine for a local benchmark, risky to attach to a live production process. py-spy is a sampling profiler that attaches to an already-running process from the outside, with no code changes and negligible overhead, which makes it the right tool when you need to profile something that’s already running in production.

$ pip install py-spy
$ py-spy top --pid 48213

Collecting samples from 'python' (pid: 48213)
Total Samples 1400
GIL: 0.00%, Active: 89.00%, Threads: 4

  %Own   %Total  OwnTime  TotalTime  Function (filename)
 42.00%  42.00%   5.880s    5.880s   transform (pipeline.py)
 31.00%  31.00%   4.340s    4.340s   fetch_data (pipeline.py)
  9.00%   9.00%   1.260s    1.260s   summarize (pipeline.py)

py-spy dump --pid <pid> also gives a one-time stack trace of every thread, which is invaluable for diagnosing a process that appears hung — you get the exact line each thread is stuck on without touching the running code at all.

A Practical Workflow

  1. Benchmark first, with timeit, to confirm there’s actually a meaningful difference between approaches before investing effort.
  2. Profile with cProfile to find which function dominates total runtime — sort by tottime for self-cost, cumtime for total-cost-including-children.
  3. Drill into that function with line_profiler if the fix isn’t obvious from the function-level view alone.
  4. Use py-spy when the slow thing is already running in production and you can’t (or shouldn’t) restart it with instrumentation attached.
$ python -m cProfile -o profile.out myapp.py
$ python -c "import pstats; pstats.Stats('profile.out').sort_stats('tottime').print_stats(10)"

Saving profile output to a file with -o and loading it with pstats separately is the standard way to profile once and slice the results multiple ways, without re-running the (possibly slow) profiled code each time.

Conclusion

Profiling and benchmarking answer different questions: benchmarking (timeit) tells you whether a specific change made things faster, and profiling (cProfile, line_profiler, py-spy) tells you where to look for that change in the first place. Function-level profiling finds the slow function; line-level profiling finds the slow line inside it; sampling profilers let you do both against a live process you can’t afford to restart. Skipping straight to optimization without this data almost always means optimizing the wrong thing — measure first, then fix what the numbers actually point to.