Hardware-software co-design

Note : This is a hands-on, implementation-focused class where students will transition from theoretical performance modeling to programming specilized hardware.

Serveral of the course block will be based on the textbook "Efficient Processing of Deep Neural Networks (Synthesis Lectures on Computer Architecture)" by Vivienne Sze & al.

Tentative syllabus (up to reordering; a subset of bullet points within each block might be selected from year to year)

Block 0 : Principles of Hardware-Software codesign :

• Speed of light, Roofline analysis, Arithmetic intensity
• Little's law
• Taxonomy of parallelisms
• Custom datatypes (dynamic range, precision, quantitative benefits of quantization)

Block 1 : GPU

• Architecture (multi-level parallelism and integrated accelerators)
• GPU pseudo-ISAS (PTX)
• Compilation challenges
• Divergence
• Explicit management of memory hierarchy
• Practical GPU programming
• Kernel fusion (FlashAttention)
• Static dataflow graphs on a GPU

Block 2 : Systolic Architectures

• Systolically computable functions (theory), space-time mapping
• GEMM : Input, Weight, Output stationary dataflows
• Architecture (0-D + 1-D + 2-D systolic units + data orchestrators)
• Kernel fusion on systolic architectures
• Low-level programming for a TPU-like machine

Block 3 : Spatial architectures

• BSP paradigm
• Programming around the NoC
• Hypergraph partitioning

Block 4 : Sparsity

• Sparse matrix multipliction, graph analytics, recommendation systems
• Metadata representation
• Memory amplification and sort reduce
• Historical dataflow machines and modern reincarnation

Block 5 : Compilation challenges and techniques

• Domain space exploration : space of loop nest reordering "Algorithm vs Mapping"
• MILP for DSL compilers
• Simple DSL "JIT"s
• Egraphs
• Decoupled languages (Halide)
• Tiling languages (Triton, tilelang, cuTILE
• MLIR

Note that topics are tentative and will be explored at different depths. In class participation is strongly recommended.