Python Memory Management and Troubleshooting

Dave Bunten (@d33bs) Research Data Engineer

January 22, 2024 April 23, 2024

Python Memory Management and Troubleshooting

These blog posts are intended to provide software tips, concepts, and tools geared towards helping you achieve your goals. Views expressed in the content belong to the content creators and not the organization, its affiliates, or employees. If you have any questions or suggestions for blog posts, please don’t hesitate to reach out!

Introduction

Have you ever run Python code only to find it taking forever to complete or sometime abruptly ending with an error like: 123456 Killed or killed (program exited with code: 137)? You may have experienced memory resource or management challenges associated with these scenarios. This post will cover some computer memory definitions, how Python makes use of computer memory, and share some tools which may help with these types of challenges.

What is Software?

Computer software includes programs, documentation, and other data maintained on computer data storage.

Computer software is the collection of programs and data which are used to accomplish a specific tasks on a computer. “A computer program is a sequence or set of instructions in a programming language for a computer to execute. It is one component of software, which also includes documentation and other intangible components.” (Wikipedia: Computer program). Computer programs in their human-readable form are stored as source code. Source code is often maintained on computer data storage.

What is Memory?

Computer Memory

Computer memory is a type of computer resource available for use by processes on a computer.

Computer memory, also sometimes known as “RAM” or “random-access memory”, or “dynamic memory” is a type of resource used by computer software on a computer. “Computer memory stores information, such as data and programs for immediate use in the computer. … Main memory operates at a high speed compared to non-memory storage which is slower but less expensive and oftentimes higher in capacity. “ (Wikipedia: Computer memory). When we execute a computer program it becomes a process (or sometimes many processes). Processes are loaded into computer memory to follow the instructions and other data provided from their related computer programs.

The word “speed” in the above context is sometimes used to describe the delay before an operation on a computer completes (also known as latency). See the following on [Computer] Latency Numbers Everyone Should Know to better understand relative computer operation speeds.

Process Memory Segment	Purpose
Stack	Contains information about sequences of program instructions as functions or subroutines.
Heap	Area where memory for variables may be dynamically used.
Initialized data	Includes global and static variables which are explicitly initialized.
Uninitialized data	Includes global and static variables which are not explicitly initialized.
Text	Comprises program instructions for the process.

Process memory is divided into segments which have specific purposes (The Linux Programming Interface by Michael Kerrisk).

Memory for a process is further divided into parts which are typically called segments. Each process memory segment has a specific purpose and way of organizing things. For the purposes of this content we’ll focus on two of these segments: the stack and the heap. The stack (sometimes also known as the “call stack”) includes information about sequences of program instructions packaged as units called “functions” or “subroutines”. The stack also typically stores function local variables, arguments, and return value. The heap is an area where variables for a program may be dynamically stored. The stack can be thought of as a “roadmap” for what program will accomplish (including the location of things it will need to do that work). The heap can be imagined of as a “warehouse” store (or remove) things used as part of the stack “roadmap”. Please see The Linux Programming Interface by Michael Kerrisk, Chapter 6.3: Memory Layout of a Process for more information about processes.

Memory Blocks

A.) All memory blocks available.

Block

B.) Some memory blocks in use.

Block

Practical analogy

C.) You have limited boxes to hold things.

📦

D.) Two boxes are used, the other remains empty (ready for use).

📦

Memory blocks may be free or used at various times. They can be thought of like reusable buckets to hold things.

The heap is often further organized through the use of “blocks”. Memory blocks are chunks of memory of a certain byte or bit size (usually all the same size) (Wikipedia: Block (data storage)). Memory blocks may be in use or free at different times. If the heap is a process memory “warehouse” then blocks are like “boxes” inside the warehouse.

Process memory heaps help organize memory blocks on a computer for specific procedures. Heaps may have one or many memory pools.

Blocks may be organized in hierarchical layers to manage memory efficiently or towards a specific purpose. Blocks may sometimes be organized into pools within the process memory heap segment. Pools are areas of the heap used to efficiently manage blocks together in specific ways. Each heap may have one or many pools (each with sets of blocks). If the heap is a process memory “warehouse”, and blocks are like “boxes” inside the warehouse, pools are like “shelves” for organizing and moving those boxes within the warehouse.

Memory Allocator

Memory allocators help software reserve and free computer memory resources.

Memory management is a concept which helps enable the shared use of computer memory to avoid challenges such as memory overuse (where all memory is in use and never shared to other software). Computer memory management often occurs through the use of a memory allocator which controls how computer memory resources are used for software. Computer software is written to interact with memory allocators to use computer memory. Memory allocators may be used manually (with specific directions provided on when and how to use memory resources) or automatically (with an algorithmic approach of some kind). The memory allocator usually performs the following actions with memory (in addition to others):

“Allocation”: computer memory resource reservation (taking memory). This is sometimes also known as “alloc”, or “allocate memory”.
“Deallocation”: computer memory resource freeing (giving back memory for other uses). This is sometimes also known as “free”, or “freeing memory from allocation”.

Garbage Collection

Garbage collectors help free computer memory which is no longer referenced by software.

“Garbage collection (GC)” is used to describe a type of automated memory management. GC is typically used to help reduce human error, avoid unintentional system failures, and decrease development time (through less memory-specific code). “The garbage collector attempts to reclaim memory which was allocated by the program, but is no longer referenced; such memory is called garbage.” (Wikipedia: Garbage collection (computer science)). A garbage collector often works in tandem with a memory allocator to help control computer memory resource usage in software development.

How Does Python Interact with Computer Memory?

Python Overview

A Python interpreter executes Python code and manages memory for Python procedures.

Python is an interpreted “high-level” programming language (Python: What is Python?). Interpreted languages are those which include an “interpreter” which helps execute code written in a particular way (Wikipedia: Interpreter (computing)). High-level languages such as Python often remove the requirement for software developers to manually perform memory management (Wikipedia: High-level programming language).

Python code is executed by a commonly pre-packaged and downloaded binary call the Python interpreter. The Python interpreter reads Python code and performs memory management as the code is executed. The CPython Python interpreter is the most commonly used interpreter for Python, and what’s use as a reference for other content here. There are also other interpreters such as PyPy, Jython, and IronPython which all handle memory differently than the CPython interpreter.

Python’s Memory Manager

The Python memory manager helps manage memory in the heap for Python processes executed by the Python interpreter.

Memory is managed for Python software processes automatically (when unspecified) or manually (when specified) through the Python interpreter. The Python memory manager is an abstraction which manages memory for Python software processes through the Python interpreter (Python: Memory Management). From a high-level perspective, we assume variables and other operations written in Python will automatically allocate and deallocate memory through the Python interpreter when executed. Python’s memory manager performs work through various memory allocators and a garbage collector (or as configured with customizations) within a private Python memory heap.

Python’s Memory Allocators

The Python memory manager by default will use pymalloc internally or malloc from the system to allocate computer memory resources.

The Python memory manager allocates memory for use through memory allocators. Python may use one or many memory allocators depending on specifications in Python code and how the Python interpreter is configured (for example, see Python: Memory Management - Default Memory Allocators). One way to understand Python memory allocators is through the following distinctions.

“Python Memory Allocator” (pymalloc) The Python interpreter is packaged with a specialized memory allocator called pymalloc. “Python has a pymalloc allocator optimized for small objects (smaller or equal to 512 bytes) with a short lifetime.” (Python: Memory Management - The pymalloc allocator). Ultimately, pymalloc uses C standard library dynamic memory allocation functions to implement memory work.
C dynamic memory allocation functions (malloc, realloc, etc.) When pymalloc is disabled or a memory requirements exceed pymalloc’s constraints, the Python interpreter will directly use a function from the C standard library called C standard library dynamic memory allocation functions. When C standard library dynamic memory allocation functions are used by the Python interpreter, it uses the system’s existing implementation of the C standard library.

pymalloc makes use of arenas to further organize pools within a Python process memory heap.

It’s important to note that pymalloc adds additional abstractions to how memory is organized through the use of “arenas”. These arenas are specific to pymalloc purposes. pymalloc may be disabled through the use of a special environment variable called PYTHONMALLOC (for example, to use only C standard library dynamic memory allocation functions as seen below). This same environment variable may be used with debug settings in order to help troubleshoot in-depth questions.

Additional Python Memory Allocators

Python code may stipulate the use of additional memory allocators, such as mimalloc and jemalloc outside of the default Python memory manager’s operation.

Python provides the capability of customizing memory allocation through the use of custom code or non-default packages. See below for some notable examples of additional memory allocation possibilities.

NumPy Memory Allocation NumPy uses custom C-API’s which are backed by C dynamic memory allocation functions (alloc, free, realloc) to help address memory management. These interfaces can be controlled directly through NumPy to help manage memory effectively when using the package.
PyArrow Memory Allocators PyArrow provides the capability to use C standard library dynamic memory allocation functions, jemalloc, or mimalloc through the PyArrow Memory Pools group of functions. A default memory allocator is selected for use when PyArrow based on the operating system and the availability of the memory allocator on the system. The selection of a memory allocator for use with PyArrow can be influenced by how it performs on a particular system.

Python Reference Counting

</table> _Python reference counting at a simple level works through the use of object reference increments and decrements._ {:.center} As computer memory is allocated to Python processes the Python memory manager keeps track of these through the use of a [reference counter](https://en.wikipedia.org/wiki/Reference_counting). In Python, we could label this as an "Object reference counter" because all data in Python is represented by objects ([Python: Data model](https://docs.python.org/3/reference/datamodel.html#objects-values-and-types)). "... CPython counts how many different places there are that have a reference to an object. Such a place could be another object, or a global (or static) C variable, or a local variable in some C function." ([Python Developer's Guide: Garbage collector design](https://devguide.python.org/internals/garbage-collector/)). ### Python's Garbage Collection

_The Python garbage collector works as part of the Python memory manager to free memory which is no longer needed (based on reference count)._ {:.center} Python by default uses an optional garbage collector to automatically deallocate garbage memory through the Python interpreter in CPython. "When an object’s reference count becomes zero, the object is deallocated." ([Python Developer's Guide: Garbage collector design](https://devguide.python.org/internals/garbage-collector/)) Python's garbage collector focuses on collecting garbage created by `pymalloc`, C memory functions, as well as other memory allocators like `mimalloc` and `jemalloc`. ## Python Tools for Observing Memory Behavior ### Python Built-in Tools ```python import gc import sys # set gc in debug mode for detecting memory leaks gc.set_debug(gc.DEBUG_LEAK) # create an int object an_object = 1 # show the number of uncollectable references via COLLECTED COLLECTED = gc.collect() print(f"Uncollectable garbage references: {COLLECTED}") # show the reference count for an object print(f"Reference count of `an_object`: {sys.getrefcount(an_object)}") ``` The [`gc` module](https://docs.python.org/3/library/gc.html) provides an interface to the Python garbage collector. In addition, the [`sys` module](https://docs.python.org/3/library/sys.html) provides many functions which provide information about references and other details about Python objects as they are executed through the interpreter. These functions and other packages can help software developers observe memory behaviors within Python procedures. ### Python Package: Scalene

Scalene provides a web interface to analyze memory, CPU, and GPU resource consumption in one spot alongside suggested areas of concern.

[Scalene](https://github.com/plasma-umass/scalene) is a Python package for analyzing memory, CPU, and GPU resource consumption. It provides [a web interface](https://github.com/plasma-umass/scalene?tab=readme-ov-file#web-based-gui) to help visualize and understand how resources are consumed. Scalene provides suggestions on which portions of your code to troubleshoot through the web interface. Scalene can also be configured to work with [OpenAI](https://en.wikipedia.org/wiki/OpenAI) [LLM's](https://en.wikipedia.org/wiki/Large_language_model) by way of a an [OpenAI API provided by the user](https://github.com/plasma-umass/scalene?tab=readme-ov-file#ai-powered-optimization-suggestions). ### Python Package: Memray

Memray provides the ability to create and view flamegraphs which show how memory was consumed as a procedure executed.

[Memray](https://github.com/bloomberg/memray) is a Python package to track memory allocation within Python and compiled extension modules. Memray provides a high-level way to investigate memory performance and adds visualizations such as [flamegraphs](https://www.brendangregg.com/flamegraphs.html) (which contextualization of [stack traces](https://en.wikipedia.org/wiki/Stack_trace) and memory allocations in one spot). Memray seeks to provide a way to overcome challenges with tracking and understanding Python and other memory allocators (such as C, C++, or Rust libraries used in tandem with a Python process). ## Concluding Thoughts It's worth mentioning that this article covers only a small fraction of how and what memory is as well as how Python might make use of it. Hopefully it clarifies the process and provides a way to get started with investigating memory within the software you work with. Wishing you the very best in your software journey with memory!

Processed line of code	Reference count
`a_string = "cornucopia"`	a_string: 1
`reference_a_string = a_string`	a_string: 2 (Because `a_string` is now referenced twice.)
`del reference_a_string`	a_string: 1 (Because the additional reference has been deleted.)