Linux Memory Management (MM) and Writing Device Drivers: What’s the connection?
I wrote a blog post series on linux memory management, based on pictures and kernelshark visualizations (Parts 1, 2, 3). The purpose is to introduce concepts to kernel newbies using resources that I found along my way, and that I thought were very helpful.
How are these memory management concepts related to writing device drivers?
References
Linux Device Drivers, Third Edition
Notation
Chapter x, p. y (p. z) identifies the pdf file for Chapter x, the absolute page number y, and the page number within the chapter (z).
Terms
device: a peripheral unit (hardware or software) that adds functionality to a kernel
driver: code that implements an API defined between a specific device and the kernel
device driver: the code that implements the API defined for a specific device; the kernel invokes functions on the device and the driver implements those functions (Chapter 1, page 1 (1))
module: code linked to the kernel at runtime (Chapter 1, page 5); a device driver can be configured as a module
The Big Picture
Linux treats a device as a special file, in the /dev/<namespace>. Therefore, file objects and file operations on those objects, are the core of driver code.
Figure: A Device Drive+Kernel shows the kernel on the right, and a device driver on the left. In the figure, the device is some type of disk, and the kernel represents the disk in a data type, struct gendisk.
I overlay the Figure with the example device (scull) given in the text in Chapter 3. (Hopefully, that is not overly confusing.)
Figure: A Device Driver+Kernel
Let’s use this figure to understand how the example driver (scull character device driver), discussed in Chapter 3, fits into the big picture.
struct scull_dev , p. 56 (p. 15) is analogous to the box labeled struct gendisk in the Figure. struct scull_dev is the driver’s representation of a scull device.
struct scull_dev encapsulates another data type: struct cdev. “The kernel uses structures of type struct cdev to represent char devices internally. (p. 55 (p.14))” “struct cdev interfaces the scull device to the kernel” (p. 56 (p.15)).
Allocate cdev
struct cdev *my_cdev = cdev_alloc( );
Initialize and Add cdev
Since cdev is a member of struct scull_dev, the function cdev_init is invoked by device driver code to initialize struct scull_dev. (p. 57 (p. 16))
initialize scull device: scull_setup_cdev(struct scull_dev *dev, int index)
scull_setup_cdev function calls cdev_init:
cdev_init(&dev->cdev, &scull_fops);
In the Figure, the scull_setup_cdev function is represented by the white box [labeled init function]. The scull_setup_cdev function invokes the cdev_init function [labeled blk_init_queue], as shown by the dashed line that represents a function call.
scull_setup_cdev function sets the cdev.ops (function pointers) to point to the functions that operate on scull devices (scull_fops, p 53 (p. 12))
dev->cdev.ops = &scull_fops;
The scull_setup_cdev function does a few more things and finally invokes the cdev_add function to “tell” the kernel about the cdev device (p. 56 (p. 15)). devno is the starting device number for the device. The last argument is number of instances to be added.
cdev_add (&dev->cdev, devno, 1);
Referring to the Figure above, cdev.ops are the arrows pointing from the box [labeled struct gendisk] to the box [labeled block_device ops].
scull device driver functions: struct file_operations
Chapter 3, p. 53 (p. 12) defines the struct file_operations scull_fops. scull_fops is a struct of function pointers. In the Figure above, scull_fops is the collection of rectangles [labeled block_device ops]. Each member of scull_fops (i.e., a function pointer) is an arrow from a dark rectangle to the light gray box [Multiple functions], as shown by the function pointer arrow. The scull_fops defined on p. 53 (p. 12) are a subset of all possible functions pointers defined by struct file_operations. Some of the function pointers are read, write, open, release.
Detour: the mmap struct file_operations function
One of the functions not applicable to the scull device is mmap. mmap is the function used by the kernel to map a range of addresses in device memory to a virtual memory area (VMA) in a process’s address space. (In Part 1, we discussed memory mapping.) See Chapter 15, p. 424 (p. 13) for discussion of mmap as a driver function.
The user process invokes the system call mmap. The kernel does some work, and then calls the mmap function of struct file_operations. [Note: The name ‘mmap’ is used for the system call and the f_ops->mmap function.] Chapter 15, p. 420 (p. 9)).
[End of Detour]
scull function implementation
If the kernel wants to open a scull device (struct scull_dev *dev), for example, the kernel would invoke :
dev->cdev->ops->open->scull_open
open points to the function scull_open (Chapter 3, p. 31 (p. 18)).
The kernel dereferences open. The driver code implements scull_open. In the Figure, the call to scull_open is represented by the dashed line from the dark gray box on the left (above cleanup function) to the dark gray box on the right (above del_gendisk).
Each of the function calls in scull_fops (Chapter 3, p. 53 (p. 12)) are represented by similar dashed lines from the left gray box (in the driver code) to the right gray box (in the kernel).
Now that we have considered driver code in terms of the Figure, let’s just talk briefly about where memory mapping comes in, for those devices that use memory mapping. (Memory mapping does not apply to the scull device since it is a device that only exists in memory.)
Memory Mapping and Device Drivers
To map a device means to associate a range of user-space addresses (program's virtual pages) to device memory.
Chapter 15, page 424 (p. 13) discusses the implementation of the mmap system call. After the kernel receives the mmap system call, the kernel creates a struct vm_area_struct.
“...much of the work has been done by the kernel; to implement [filp->f_op->]mmap, the driver only has to build suitable page tables for the address range and, if necessary, replace vma->vm_ops with a new set of operations.”
Here are some data types that contain a mmap pointer (pointer to a mmap function), which can be implemented by driver code:
memory map a device, identified by a pointer to its file descriptor:
filp->f_op->mmap->driver-filedescr-mmap-here
memory map a vma, to resolve a page fault
vma->vm_ops->mmap->driver-vma-mmap-here
Example Device that Implements mmap
The scullp device, described here, p. 431 (p. 20), defines the function scullp_mmap, Chapter 15, p. 432 (p. 21). It assigns vma->vm_ops to the address of scullp_vm_ops. So, vma->vm_ops->nopage points to scullp_vma_nopage, Chapter 15, p. 433 (p. 22). scullp_vma_nopage returns a pointer to the page frame of the process’s vma that is associated with the scullp device memory.
Conclusion
This concludes the discussion of how device driver code connects itself to the kernel using device operations. We used a Figure for a different device driver to understand the example in the text for the in-memory scull device. Then we looked at a different device, discussed in Chapter 15, called scullp, to show how a device driver can implement the mmap function. The scullp example relied on the kernel’s implementation of the vma operation nopage.
