Chapter 19: Graphics, Compilers, Debugging, and Security

Introduction

This final chapter explores advanced topics that round out your system programming knowledge: graphics programming (framebuffers, GPU basics), compiler internals (how C becomes machine code), advanced debugging techniques, and security considerations. These topics are essential for professional system programmers working on real-world projects.

Why This Matters

Modern systems require understanding graphics (even servers need console output), compiler behavior (for optimization and debugging), debugging skills (for finding complex bugs), and security (to protect systems from vulnerabilities). These topics bridge system programming and practical software engineering.

How to Study This Chapter

Experiment widely - Try different tools and techniques
Read documentation - Each topic has extensive resources
Practice debugging - Debug real programs, not just examples
Think security first - Consider vulnerabilities in all code
Stay curious - These are deep topics worthy of continued study

Graphics Programming

Framebuffer Basics

A framebuffer is a region of memory representing pixel data. Writing to it changes what's displayed on screen.

Simple framebuffer (VGA text mode):

#include <stdint.h>

#define VGA_MEMORY 0xB8000 #define VGA_WIDTH 80 #define VGA_HEIGHT 25

void put_pixel(int x, int y, char c, uint8_t color) { uint16_t *vga = (uint16_t *)VGA_MEMORY; vga[y * VGA_WIDTH + x] = (color << 8) | c; }

void clear_screen(void) { for (int y = 0; y < VGA_HEIGHT; y++) { for (int x = 0; x < VGA_WIDTH; x++) { put_pixel(x, y, ' ', 0x0F); } } }

void draw_box(int x, int y, int w, int h) { for (int i = 0; i < w; i++) { put_pixel(x + i, y, '-', 0x0F); put_pixel(x + i, y + h - 1, '-', 0x0F); } for (int i = 0; i < h; i++) { put_pixel(x, y + i, '|', 0x0F); put_pixel(x + w - 1, y + i, '|', 0x0F); } }

Linear Framebuffer (Graphics Mode)

For pixel-based graphics:

struct framebuffer {
    uint32_t *buffer;
    int width;
    int height;
    int pitch;  // Bytes per scanline
};

void put_pixel_rgb(struct framebuffer *fb, int x, int y, uint32_t color) { if (x < 0 || x >= fb->width || y < 0 || y >= fb->height) return;

fb->buffer[y * fb->width + x] = color;

}

void draw_rectangle(struct framebuffer *fb, int x, int y, int w, int h, uint32_t color) { for (int dy = 0; dy < h; dy++) { for (int dx = 0; dx < w; dx++) { put_pixel_rgb(fb, x + dx, y + dy, color); } } }

void draw_line(struct framebuffer *fb, int x0, int y0, int x1, int y1, uint32_t color) { // Bresenham's line algorithm int dx = abs(x1 - x0); int dy = abs(y1 - y0); int sx = x0 < x1 ? 1 : -1; int sy = y0 < y1 ? 1 : -1; int err = dx - dy;

while (1) {
    put_pixel_rgb(fb, x0, y0, color);

    if (x0 == x1 && y0 == y1)
        break;

    int e2 = 2 * err;
    if (e2 > -dy) {
        err -= dy;
        x0 += sx;
    }
    if (e2 < dx) {
        err += dx;
        y0 += sy;
    }
}

}

Linux Framebuffer Device

#include <fcntl.h>
#include <sys/mman.h>
#include <linux/fb.h>
#include <sys/ioctl.h>

int main() { int fd = open("/dev/fb0", O_RDWR); if (fd < 0) { perror("open"); return 1; }

// Get screen info
struct fb_var_screeninfo vinfo;
ioctl(fd, FBIOGET_VSCREENINFO, &vinfo);

printf("Resolution: %dx%d, %d bpp\n",
       vinfo.xres, vinfo.yres, vinfo.bits_per_pixel);

// Map framebuffer
size_t screensize = vinfo.xres * vinfo.yres * vinfo.bits_per_pixel / 8;
uint8_t *fbp = mmap(0, screensize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

if (fbp == MAP_FAILED) {
    perror("mmap");
    close(fd);
    return 1;
}

// Draw red rectangle
for (int y = 100; y < 200; y++) {
    for (int x = 100; x < 200; x++) {
        int offset = (y * vinfo.xres + x) * 4;
        fbp[offset + 0] = 0;      // Blue
        fbp[offset + 1] = 0;      // Green
        fbp[offset + 2] = 255;    // Red
        fbp[offset + 3] = 0;      // Alpha
    }
}

munmap(fbp, screensize);
close(fd);

return 0;

}

GPU Programming Basics

OpenGL example (minimal):

#include <GL/gl.h>
#include <GL/glut.h>

void display(void) { glClear(GL_COLOR_BUFFER_BIT);

// Draw triangle
glBegin(GL_TRIANGLES);
glColor3f(1.0, 0.0, 0.0);  // Red
glVertex2f(-0.5, -0.5);
glColor3f(0.0, 1.0, 0.0);  // Green
glVertex2f(0.5, -0.5);
glColor3f(0.0, 0.0, 1.0);  // Blue
glVertex2f(0.0, 0.5);
glEnd();

glFlush();

}

int main(int argc, char **argv) { glutInit(&argc, argv); glutInitDisplayMode(GLUT_SINGLE | GLUT_RGB); glutInitWindowSize(800, 600); glutCreateWindow("OpenGL Triangle"); glutDisplayFunc(display); glutMainLoop(); return 0; }

Vulkan (modern, lower-level):

More complex but more control
Used in modern games and graphics applications
Lower overhead than OpenGL

Compiler Internals

Compilation Stages

Source Code (C)
    ↓
Preprocessing (cpp)
    ↓
Compilation (cc1)
    ↓
Assembly Code (.s)
    ↓
Assembly (as)
    ↓
Object Code (.o)
    ↓
Linking (ld)
    ↓
Executable

Viewing Intermediate Stages

# Preprocessing only
gcc -E program.c -o program.i

Compilation to assembly

gcc -S program.c -o program.s

Assembly to object file

gcc -c program.c -o program.o

Linking

gcc program.o -o program

All in one

gcc program.c -o program

Assembly Output

C code:

int add(int a, int b) {
    return a + b;
}

Assembly (x64, AT&T syntax):

gcc -S -O0 add.c

add.s

add: pushq %rbp movq %rsp, %rbp movl %edi, -4(%rbp) movl %esi, -8(%rbp) movl -4(%rbp), %edx movl -8(%rbp), %eax addl %edx, %eax popq %rbp ret

With optimization:

gcc -S -O2 add.c

add.s (much simpler!)

add: leal (%rdi,%rsi), %eax ret

Optimization Levels

-O0  # No optimization (default, good for debugging)
-O1  # Basic optimizations
-O2  # Moderate optimizations (recommended for production)
-O3  # Aggressive optimizations
-Os  # Optimize for size
-Og  # Optimize for debugging experience

Compiler Flags for System Programming

# Include debugging symbols
gcc -g program.c -o program

Enable all warnings

gcc -Wall -Wextra -Werror program.c -o program

Position-independent code (for shared libraries)

gcc -fPIC -shared library.c -o library.so

Static linking

gcc -static program.c -o program

Control optimization

gcc -O2 program.c -o program

Kernel-style compilation

gcc -nostdlib -ffreestanding -fno-stack-protector -mno-red-zone

Generate assembly listing

gcc -Wa,-adhln=program.lst -c program.c

Show what compiler is doing

gcc -v program.c -o program

Link Time Optimization (LTO)

# Enable LTO
gcc -flto -O2 file1.c file2.c -o program

Can optimize across compilation units

Advanced Debugging

GDB Power Features

Setting up GDB:

# Compile with debug symbols
gcc -g -O0 program.c -o program

Start GDB

gdb ./program

Advanced GDB commands:

# Breakpoints
break main
break file.c:42
break function if x > 10

Watchpoints (break when variable changes)

watch variable_name

Catchpoints (break on events)

catch syscall catch throw

Conditional breakpoints

break main if argc > 1

Examine memory

x/10x $rsp # 10 hex words at stack pointer x/s 0x404000 # String at address x/10i$ rip # 10 instructions at instruction pointer

Disassemble

disassemble main disassemble /m main # With source

Register manipulation

set $rax = 0x42 print$ rax

Call functions

call printf("Debug: %d\n", x)

Backtrace

bt bt full # With local variables

Thread debugging

info threads thread 2 thread apply all bt

Core dump analysis

gdb program core

GDB scripting:

# .gdbinit or script file
define printstack
    set $i = 0
    while $i < 10
        x/x ($rsp + $i * 8)
        set $i = $i + 1
    end
end

Valgrind

Memory leak detection:

# Compile with debug symbols
gcc -g program.c -o program

Run with Valgrind

valgrind --leak-check=full ./program

More detailed

valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./program

Example output:

==12345== HEAP SUMMARY:
==12345==     in use at exit: 100 bytes in 1 blocks
==12345==   total heap usage: 1 allocs, 0 frees, 100 bytes allocated
==12345==
==12345== 100 bytes in 1 blocks are definitely lost
==12345==    at malloc (in /usr/lib/valgrind/...)
==12345==    by main (program.c:10)

AddressSanitizer (ASan)

Compile-time memory error detector:

# Compile with ASan
gcc -fsanitize=address -g program.c -o program

Run

./program

Detects:

Buffer overflows
Use-after-free
Use-after-return
Double-free
Memory leaks

strace (System Call Tracer)

# Trace system calls
strace ./program

Trace specific syscalls

strace -e open,read,write ./program

Follow forks

strace -f ./program

Save to file

strace -o trace.txt ./program

Time syscalls

strace -c ./program

Example output:

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0"..., 832) = 832
close(3) = 0

ltrace (Library Call Tracer)

# Trace library calls
ltrace ./program

Show timestamps

ltrace -t ./program

Core Dumps

Enable core dumps:

# Set core dump size limit
ulimit -c unlimited

Run program (will create core file on crash)

./program

Debug with GDB

gdb ./program core

Analyzing core dump:

(gdb) bt          # Backtrace
(gdb) info registers
(gdb) x/10x $rsp  # Examine stack

Security in System Programming

Common Vulnerabilities

1. Buffer Overflow:

// Vulnerable
void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
}

// Safe void safe(char *input) { char buffer[64]; strncpy(buffer, input, sizeof(buffer) - 1); buffer[sizeof(buffer) - 1] = '\0'; }

2. Format String Vulnerability:

// Vulnerable
printf(user_input);  // User controls format string!

// Safe printf("%s", user_input);

3. Integer Overflow:

// Vulnerable
size_t size = user_provided_size;
char *buf = malloc(size);  // What if size wraps around?

// Safe if (size > MAX_ALLOWED_SIZE) return -1; char *buf = malloc(size);

4. Use-After-Free:

// Vulnerable
free(ptr);
// ... later ...
*ptr = value;  // Accessing freed memory!

// Safe free(ptr); ptr = NULL; // Prevent accidental use

Secure Coding Practices

// Always check return values
int fd = open("file.txt", O_RDONLY);
if (fd < 0) {
    perror("open");
    return -1;
}

// Use safe string functions strncpy(dest, src, sizeof(dest) - 1); dest[sizeof(dest) - 1] = '\0';

// Check array bounds if (index >= array_size) { return -1; }

// Validate input if (user_size > MAX_SIZE || user_size == 0) { return -1; }

// Clear sensitive data memset(password, 0, sizeof(password));

// Use const for read-only data void process(const char *input);

Compiler Security Features

# Stack canaries (detect buffer overflows)
gcc -fstack-protector-strong program.c -o program

Position Independent Executable (ASLR support)

gcc -fPIE -pie program.c -o program

Enable all warnings

gcc -Wall -Wextra -Werror program.c -o program

Fortify source (runtime checks)

gcc -D_FORTIFY_SOURCE=2 -O2 program.c -o program

No executable stack

gcc -z noexecstack program.c -o program

Full RELRO (read-only relocations)

gcc -Wl,-z,relro,-z,now program.c -o program

Static Analysis Tools

Cppcheck:

sudo apt-get install cppcheck
cppcheck --enable=all program.c

Clang Static Analyzer:

scan-build gcc program.c -o program

Fuzzing

AFL (American Fuzzy Lop):

# Compile with AFL
afl-gcc program.c -o program

Run fuzzer

afl-fuzz -i testcases/ -o findings/ ./program @@

System Hardening

ASLR (Address Space Layout Randomization):

# Check if ASLR enabled
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = partial, 2 = full

Enable ASLR

echo 2 | sudo tee /proc/sys/kernel/randomize_va_space

SELinux/AppArmor:

Mandatory access control
Restricts what processes can do
Essential for production systems

Performance Profiling

Using perf

# Record performance data
sudo perf record ./program

Analyze

sudo perf report

Profile CPU usage

sudo perf top

Count events

sudo perf stat ./program

Using gprof

# Compile with profiling
gcc -pg program.c -o program

Run program (generates gmon.out)

./program

Analyze

gprof program gmon.out > analysis.txt

Best Practices Summary

Code Quality

Use version control (Git)
Write tests (unit tests, integration tests)
Use static analysis tools
Enable all compiler warnings
Code review

Security

Validate all input
Check all return values
Use safe string functions
Clear sensitive data
Follow principle of least privilege

Debugging

Use version control to track changes
Add assertions liberally
Use debugger, don't just add printf
Test edge cases
Write reproducible test cases

Performance

Profile before optimizing
Optimize algorithms, not micro-optimizations
Use appropriate data structures
Consider cache effects
Benchmark changes

Key Concepts

Framebuffers provide direct pixel/text access
Compilers transform C through preprocessing, compilation, assembly, and linking
Optimization levels trade compile time and code size for speed
GDB is essential for debugging system programs
Valgrind detects memory errors and leaks
strace shows system calls
Buffer overflows are common security vulnerabilities
Compiler security features help prevent exploits
Static analysis finds bugs without running code

Common Mistakes

Trusting user input - Always validate
Ignoring return values - Check for errors
Buffer overflows - Use safe string functions
Not using debugger - GDB saves time
Premature optimization - Profile first
No error handling - Handle all error cases
Ignoring warnings - Fix them, don't ignore

Debugging Tips

Reproduce reliably - Can't fix what you can't reproduce
Minimize test case - Remove unrelated code
Binary search - Comment out half, find which half has bug
Print intermediate values - Verify assumptions
Use assertions - Catch bugs early
Check documentation - Read man pages
Ask for help - Describe problem clearly

Mini Exercises

Write framebuffer code to draw shapes
Examine assembly output of optimized vs unoptimized code
Use GDB to debug a segfault
Find memory leaks with Valgrind
Trace system calls with strace
Write secure string handling functions
Enable all compiler security features
Use static analyzer to find bugs
Profile program with perf
Write fuzzer test cases

Review Questions

What are the stages of C compilation?
How do optimization levels affect code?
What vulnerabilities does buffer overflow cause?
How does Valgrind detect memory leaks?
What compiler flags improve security?

Reference Checklist

By the end of this chapter, you should be able to:

Conclusion: Your System Programming Journey

Congratulations on completing this comprehensive system programming course! You've learned:

Foundations: What system programming is and why it matters
Architecture: CPU design, memory organization, instruction sets
Low-Level Programming: Assembly language for multiple architectures
Memory Management: Virtual memory, paging, MMU
Boot Process: BIOS, bootloaders, kernel initialization
Kernel Development: Building kernels for x86/x64 and ARM
Linux Programming: System calls, processes, IPC, shell scripting
Kernel Modules: Extending Linux kernel functionality
Advanced Topics: Graphics, compilers, debugging, security

Where to Go from Here

Continue Learning:

Contribute to open-source projects (Linux kernel, QEMU, etc.)
Build your own operating system
Study embedded systems programming
Learn about real-time systems
Explore hardware design (FPGA, HDL)

Career Paths:

Kernel developer
Device driver engineer
Embedded systems programmer
Security researcher
Systems architect
Compiler engineer

Resources:

Linux Kernel Development by Robert Love
Operating Systems: Three Easy Pieces by Remzi Arpaci-Dusseau
Computer Systems: A Programmer's Perspective by Bryant & O'Hallaron
Linux kernel source code
OSDev wiki (osdev.org)

Final Thoughts

System programming is challenging but immensely rewarding. You now have the knowledge to:

Understand how computers really work
Write low-level, efficient code
Debug complex system issues
Contribute to operating systems and drivers
Build systems from the ground up

Keep experimenting, keep learning, and keep building. The world needs skilled system programmers!

Key Takeaway: System programming encompasses graphics, compilers, debugging, and security. Master these tools and techniques to become a complete system programmer capable of tackling any low-level challenge.