Chapter 19: Graphics, Compilers, Debugging, and Security

Introduction

This final chapter explores advanced topics that round out your system programming knowledge: graphics programming (framebuffers, GPU basics), compiler internals (how C becomes machine code), advanced debugging techniques, and security considerations. These topics are essential for professional system programmers working on real-world projects.

Why This Matters

Modern systems require understanding graphics (even servers need console output), compiler behavior (for optimization and debugging), debugging skills (for finding complex bugs), and security (to protect systems from vulnerabilities). These topics bridge system programming and practical software engineering.

How to Study This Chapter

Experiment widely - Try different tools and techniques
Read documentation - Each topic has extensive resources
Practice debugging - Debug real programs, not just examples
Think security first - Consider vulnerabilities in all code
Stay curious - These are deep topics worthy of continued study

Graphics Programming

Framebuffer Basics

A framebuffer is a region of memory representing pixel data. Writing to it changes what's displayed on screen.

Simple framebuffer (VGA text mode):

#include <stdint.h>

#define VGA_MEMORY 0xB8000
#define VGA_WIDTH  80
#define VGA_HEIGHT 25

void put_pixel(int x, int y, char c, uint8_t color) {
    uint16_t *vga = (uint16_t *)VGA_MEMORY;
    vga[y * VGA_WIDTH + x] = (color << 8) | c;
}

void clear_screen(void) {
    for (int y = 0; y < VGA_HEIGHT; y++) {
        for (int x = 0; x < VGA_WIDTH; x++) {
            put_pixel(x, y, ' ', 0x0F);
        }
    }
}

void draw_box(int x, int y, int w, int h) {
    for (int i = 0; i < w; i++) {
        put_pixel(x + i, y, '-', 0x0F);
        put_pixel(x + i, y + h - 1, '-', 0x0F);
    }
    for (int i = 0; i < h; i++) {
        put_pixel(x, y + i, '|', 0x0F);
        put_pixel(x + w - 1, y + i, '|', 0x0F);
    }
}

Linear Framebuffer (Graphics Mode)

For pixel-based graphics:

struct framebuffer {
    uint32_t *buffer;
    int width;
    int height;
    int pitch;  // Bytes per scanline
};

void put_pixel_rgb(struct framebuffer *fb, int x, int y, uint32_t color) {
    if (x < 0 || x >= fb->width || y < 0 || y >= fb->height)
        return;

    fb->buffer[y * fb->width + x] = color;
}

void draw_rectangle(struct framebuffer *fb, int x, int y, int w, int h, uint32_t color) {
    for (int dy = 0; dy < h; dy++) {
        for (int dx = 0; dx < w; dx++) {
            put_pixel_rgb(fb, x + dx, y + dy, color);
        }
    }
}

void draw_line(struct framebuffer *fb, int x0, int y0, int x1, int y1, uint32_t color) {
    // Bresenham's line algorithm
    int dx = abs(x1 - x0);
    int dy = abs(y1 - y0);
    int sx = x0 < x1 ? 1 : -1;
    int sy = y0 < y1 ? 1 : -1;
    int err = dx - dy;

    while (1) {
        put_pixel_rgb(fb, x0, y0, color);

        if (x0 == x1 && y0 == y1)
            break;

        int e2 = 2 * err;
        if (e2 > -dy) {
            err -= dy;
            x0 += sx;
        }
        if (e2 < dx) {
            err += dx;
            y0 += sy;
        }
    }
}

Linux Framebuffer Device

#include <fcntl.h>
#include <sys/mman.h>
#include <linux/fb.h>
#include <sys/ioctl.h>

int main() {
    int fd = open("/dev/fb0", O_RDWR);
    if (fd < 0) {
        perror("open");
        return 1;
    }

    // Get screen info
    struct fb_var_screeninfo vinfo;
    ioctl(fd, FBIOGET_VSCREENINFO, &vinfo);

    printf("Resolution: %dx%d, %d bpp\n",
           vinfo.xres, vinfo.yres, vinfo.bits_per_pixel);

    // Map framebuffer
    size_t screensize = vinfo.xres * vinfo.yres * vinfo.bits_per_pixel / 8;
    uint8_t *fbp = mmap(0, screensize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

    if (fbp == MAP_FAILED) {
        perror("mmap");
        close(fd);
        return 1;
    }

    // Draw red rectangle
    for (int y = 100; y < 200; y++) {
        for (int x = 100; x < 200; x++) {
            int offset = (y * vinfo.xres + x) * 4;
            fbp[offset + 0] = 0;      // Blue
            fbp[offset + 1] = 0;      // Green
            fbp[offset + 2] = 255;    // Red
            fbp[offset + 3] = 0;      // Alpha
        }
    }

    munmap(fbp, screensize);
    close(fd);

    return 0;
}

GPU Programming Basics

OpenGL example (minimal):

#include <GL/gl.h>
#include <GL/glut.h>

void display(void) {
    glClear(GL_COLOR_BUFFER_BIT);

    // Draw triangle
    glBegin(GL_TRIANGLES);
    glColor3f(1.0, 0.0, 0.0);  // Red
    glVertex2f(-0.5, -0.5);
    glColor3f(0.0, 1.0, 0.0);  // Green
    glVertex2f(0.5, -0.5);
    glColor3f(0.0, 0.0, 1.0);  // Blue
    glVertex2f(0.0, 0.5);
    glEnd();

    glFlush();
}

int main(int argc, char **argv) {
    glutInit(&argc, argv);
    glutInitDisplayMode(GLUT_SINGLE | GLUT_RGB);
    glutInitWindowSize(800, 600);
    glutCreateWindow("OpenGL Triangle");
    glutDisplayFunc(display);
    glutMainLoop();
    return 0;
}

Vulkan (modern, lower-level):

More complex but more control
Used in modern games and graphics applications
Lower overhead than OpenGL

Compiler Internals

Compilation Stages

Source Code (C)
    ↓
Preprocessing (cpp)
    ↓
Compilation (cc1)
    ↓
Assembly Code (.s)
    ↓
Assembly (as)
    ↓
Object Code (.o)
    ↓
Linking (ld)
    ↓
Executable

Viewing Intermediate Stages

# Preprocessing only
gcc -E program.c -o program.i

# Compilation to assembly
gcc -S program.c -o program.s

# Assembly to object file
gcc -c program.c -o program.o

# Linking
gcc program.o -o program

# All in one
gcc program.c -o program

Assembly Output

C code:

int add(int a, int b) {
    return a + b;
}

Assembly (x64, AT&T syntax):

gcc -S -O0 add.c

# add.s
add:
    pushq   %rbp
    movq    %rsp, %rbp
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    movl    -4(%rbp), %edx
    movl    -8(%rbp), %eax
    addl    %edx, %eax
    popq    %rbp
    ret

With optimization:

gcc -S -O2 add.c

# add.s (much simpler!)
add:
    leal    (%rdi,%rsi), %eax
    ret

Optimization Levels

-O0  # No optimization (default, good for debugging)
-O1  # Basic optimizations
-O2  # Moderate optimizations (recommended for production)
-O3  # Aggressive optimizations
-Os  # Optimize for size
-Og  # Optimize for debugging experience

Compiler Flags for System Programming

# Include debugging symbols
gcc -g program.c -o program

# Enable all warnings
gcc -Wall -Wextra -Werror program.c -o program

# Position-independent code (for shared libraries)
gcc -fPIC -shared library.c -o library.so

# Static linking
gcc -static program.c -o program

# Control optimization
gcc -O2 program.c -o program

# Kernel-style compilation
gcc -nostdlib -ffreestanding -fno-stack-protector -mno-red-zone

# Generate assembly listing
gcc -Wa,-adhln=program.lst -c program.c

# Show what compiler is doing
gcc -v program.c -o program

Link Time Optimization (LTO)

# Enable LTO
gcc -flto -O2 file1.c file2.c -o program

# Can optimize across compilation units

Advanced Debugging

GDB Power Features

Setting up GDB:

# Compile with debug symbols
gcc -g -O0 program.c -o program

# Start GDB
gdb ./program

Advanced GDB commands:

# Breakpoints
break main
break file.c:42
break function if x > 10

# Watchpoints (break when variable changes)
watch variable_name

# Catchpoints (break on events)
catch syscall
catch throw

# Conditional breakpoints
break main if argc > 1

# Examine memory
x/10x $rsp         # 10 hex words at stack pointer
x/s 0x404000       # String at address
x/10i $rip         # 10 instructions at instruction pointer

# Disassemble
disassemble main
disassemble /m main  # With source

# Register manipulation
set $rax = 0x42
print $rax

# Call functions
call printf("Debug: %d\n", x)

# Backtrace
bt
bt full  # With local variables

# Thread debugging
info threads
thread 2
thread apply all bt

# Core dump analysis
gdb program core

GDB scripting:

# .gdbinit or script file
define printstack
    set $i = 0
    while $i < 10
        x/x ($rsp + $i * 8)
        set $i = $i + 1
    end
end

Valgrind

Memory leak detection:

# Compile with debug symbols
gcc -g program.c -o program

# Run with Valgrind
valgrind --leak-check=full ./program

# More detailed
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./program

Example output:

==12345== HEAP SUMMARY:
==12345==     in use at exit: 100 bytes in 1 blocks
==12345==   total heap usage: 1 allocs, 0 frees, 100 bytes allocated
==12345==
==12345== 100 bytes in 1 blocks are definitely lost
==12345==    at malloc (in /usr/lib/valgrind/...)
==12345==    by main (program.c:10)

AddressSanitizer (ASan)

Compile-time memory error detector:

# Compile with ASan
gcc -fsanitize=address -g program.c -o program

# Run
./program

Detects:

Buffer overflows
Use-after-free
Use-after-return
Double-free
Memory leaks

strace (System Call Tracer)

# Trace system calls
strace ./program

# Trace specific syscalls
strace -e open,read,write ./program

# Follow forks
strace -f ./program

# Save to file
strace -o trace.txt ./program

# Time syscalls
strace -c ./program

Example output:

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0"..., 832) = 832
close(3) = 0

ltrace (Library Call Tracer)

# Trace library calls
ltrace ./program

# Show timestamps
ltrace -t ./program

Core Dumps

Enable core dumps:

# Set core dump size limit
ulimit -c unlimited

# Run program (will create core file on crash)
./program

# Debug with GDB
gdb ./program core

Analyzing core dump:

(gdb) bt          # Backtrace
(gdb) info registers
(gdb) x/10x $rsp  # Examine stack

Security in System Programming

Common Vulnerabilities

1. Buffer Overflow:

// Vulnerable
void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
}

// Safe
void safe(char *input) {
    char buffer[64];
    strncpy(buffer, input, sizeof(buffer) - 1);
    buffer[sizeof(buffer) - 1] = '\0';
}

2. Format String Vulnerability:

// Vulnerable
printf(user_input);  // User controls format string!

// Safe
printf("%s", user_input);

3. Integer Overflow:

// Vulnerable
size_t size = user_provided_size;
char *buf = malloc(size);  // What if size wraps around?

// Safe
if (size > MAX_ALLOWED_SIZE)
    return -1;
char *buf = malloc(size);

4. Use-After-Free:

// Vulnerable
free(ptr);
// ... later ...
*ptr = value;  // Accessing freed memory!

// Safe
free(ptr);
ptr = NULL;  // Prevent accidental use

Secure Coding Practices

// Always check return values
int fd = open("file.txt", O_RDONLY);
if (fd < 0) {
    perror("open");
    return -1;
}

// Use safe string functions
strncpy(dest, src, sizeof(dest) - 1);
dest[sizeof(dest) - 1] = '\0';

// Check array bounds
if (index >= array_size) {
    return -1;
}

// Validate input
if (user_size > MAX_SIZE || user_size == 0) {
    return -1;
}

// Clear sensitive data
memset(password, 0, sizeof(password));

// Use const for read-only data
void process(const char *input);

Compiler Security Features

# Stack canaries (detect buffer overflows)
gcc -fstack-protector-strong program.c -o program

# Position Independent Executable (ASLR support)
gcc -fPIE -pie program.c -o program

# Enable all warnings
gcc -Wall -Wextra -Werror program.c -o program

# Fortify source (runtime checks)
gcc -D_FORTIFY_SOURCE=2 -O2 program.c -o program

# No executable stack
gcc -z noexecstack program.c -o program

# Full RELRO (read-only relocations)
gcc -Wl,-z,relro,-z,now program.c -o program

Static Analysis Tools

Cppcheck:

sudo apt-get install cppcheck
cppcheck --enable=all program.c

Clang Static Analyzer:

scan-build gcc program.c -o program

Fuzzing

AFL (American Fuzzy Lop):

# Compile with AFL
afl-gcc program.c -o program

# Run fuzzer
afl-fuzz -i testcases/ -o findings/ ./program @@

System Hardening

ASLR (Address Space Layout Randomization):

# Check if ASLR enabled
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = partial, 2 = full

# Enable ASLR
echo 2 | sudo tee /proc/sys/kernel/randomize_va_space

SELinux/AppArmor:

Mandatory access control
Restricts what processes can do
Essential for production systems

Performance Profiling

Using perf

# Record performance data
sudo perf record ./program

# Analyze
sudo perf report

# Profile CPU usage
sudo perf top

# Count events
sudo perf stat ./program

Using gprof

# Compile with profiling
gcc -pg program.c -o program

# Run program (generates gmon.out)
./program

# Analyze
gprof program gmon.out > analysis.txt

Best Practices Summary

Code Quality

Use version control (Git)
Write tests (unit tests, integration tests)
Use static analysis tools
Enable all compiler warnings
Code review

Security

Validate all input
Check all return values
Use safe string functions
Clear sensitive data
Follow principle of least privilege

Debugging

Use version control to track changes
Add assertions liberally
Use debugger, don't just add printf
Test edge cases
Write reproducible test cases

Performance

Profile before optimizing
Optimize algorithms, not micro-optimizations
Use appropriate data structures
Consider cache effects
Benchmark changes

Key Concepts

Framebuffers provide direct pixel/text access
Compilers transform C through preprocessing, compilation, assembly, and linking
Optimization levels trade compile time and code size for speed
GDB is essential for debugging system programs
Valgrind detects memory errors and leaks
strace shows system calls
Buffer overflows are common security vulnerabilities
Compiler security features help prevent exploits
Static analysis finds bugs without running code

Common Mistakes

Trusting user input - Always validate
Ignoring return values - Check for errors
Buffer overflows - Use safe string functions
Not using debugger - GDB saves time
Premature optimization - Profile first
No error handling - Handle all error cases
Ignoring warnings - Fix them, don't ignore

Debugging Tips

Reproduce reliably - Can't fix what you can't reproduce
Minimize test case - Remove unrelated code
Binary search - Comment out half, find which half has bug
Print intermediate values - Verify assumptions
Use assertions - Catch bugs early
Check documentation - Read man pages
Ask for help - Describe problem clearly

Mini Exercises

Write framebuffer code to draw shapes
Examine assembly output of optimized vs unoptimized code
Use GDB to debug a segfault
Find memory leaks with Valgrind
Trace system calls with strace
Write secure string handling functions
Enable all compiler security features
Use static analyzer to find bugs
Profile program with perf
Write fuzzer test cases

Review Questions

What are the stages of C compilation?
How do optimization levels affect code?
What vulnerabilities does buffer overflow cause?
How does Valgrind detect memory leaks?
What compiler flags improve security?

Reference Checklist

By the end of this chapter, you should be able to:

Conclusion: Your System Programming Journey

Congratulations on completing this comprehensive system programming course! You've learned:

Foundations: What system programming is and why it matters
Architecture: CPU design, memory organization, instruction sets
Low-Level Programming: Assembly language for multiple architectures
Memory Management: Virtual memory, paging, MMU
Boot Process: BIOS, bootloaders, kernel initialization
Kernel Development: Building kernels for x86/x64 and ARM
Linux Programming: System calls, processes, IPC, shell scripting
Kernel Modules: Extending Linux kernel functionality
Advanced Topics: Graphics, compilers, debugging, security

Where to Go from Here

Continue Learning:

Contribute to open-source projects (Linux kernel, QEMU, etc.)
Build your own operating system
Study embedded systems programming
Learn about real-time systems
Explore hardware design (FPGA, HDL)

Career Paths:

Kernel developer
Device driver engineer
Embedded systems programmer
Security researcher
Systems architect
Compiler engineer

Resources:

Linux Kernel Development by Robert Love
Operating Systems: Three Easy Pieces by Remzi Arpaci-Dusseau
Computer Systems: A Programmer's Perspective by Bryant & O'Hallaron
Linux kernel source code
OSDev wiki (osdev.org)

Final Thoughts

System programming is challenging but immensely rewarding. You now have the knowledge to:

Understand how computers really work
Write low-level, efficient code
Debug complex system issues
Contribute to operating systems and drivers
Build systems from the ground up

Keep experimenting, keep learning, and keep building. The world needs skilled system programmers!

Key Takeaway: System programming encompasses graphics, compilers, debugging, and security. Master these tools and techniques to become a complete system programmer capable of tackling any low-level challenge.