Graphics Processor (GPU)

Memory hierarchy

Matrix Multiplication

Shared Memory Parallelism

Roofline and Performance Modeling

Sources of Parallelism and Locality

Communication-avoiding matrix multiplication

Data Parallel Algorithms

CUDA (Compute Unified Device Architecture)

Distributed Memory Machines and Programming

Advanced MPI

Collective Communication Algorithms

UPC++: Partitioned Global Address Space Languages

Domain Specific Languages (Halide)

Distributed Data Structures (BCL)

Parallel Matrix Multiply

Dense Linear Algebra

Sparse-Matrix-Vector-Multiplication

Iterative Solvers

Structured Grids

Machine learning

Supervised learning

Deep learning

Unsupervised learning

Graph Partitioning

Fast Fourier transform

Graph Algorithms

Cloud Computing

Big Data Processing

High Performance Computing (HPC)

Dynamic Load Balancing

Hierarchical Methods for the N-Body Problem

Sorting algorithm

Search algorithm

Computational Biology

Quantum Computing

CS 267: Applications of Parallel Computers UC Berkeley Spring 2020 The course addresses programming parallel computers to solve complex scientific and engineering problems. It covers an array of parallelization strategies for numerical simulation, data analysis, and machine learning, and provides experience with popular parallel programming tools. CS267 was originally designed to teach students how to program parallel computers to efficiently solve challenging problems in science and engineering, where very fast computers are required either to perform complex simulations or to analyze enormous datasets. CS267 is intended to be useful for students from many departments and with different backgrounds, although we will assume reasonable programming skills in a conventional (non-parallel) language, as well as enough mathematical skills to understand the problems and algorithmic solutions presented. CS267 satisfies part of the course requirements for the Designated Emphasis ("graduate minor") in [Computational Science and Engineering](http://www.google.com/url?q=http%3A%2F%2Fcitris-uc.org%2Finitiatives%2Fdecse%2F&sa=D&sntz=1&usg=AOvVaw1fRFGvIaIYhkwE3kp4J1nf).

While this general outline remains, a large change in the computing world started in the mid 2000's: not only are the [fastest computers](http://www.google.com/url?q=http%3A%2F%2Fwww.top500.org%2F&sa=D&sntz=1&usg=AOvVaw1SiByfUjOpTR6ctURAV2Ef) parallel, but nearly all computers are becoming parallel, because the physics of semiconductor manufacturing will no longer let conventional sequential processors get faster year after year, as they have for so long (roughly doubling in speed every 18 months for many years). So all programs that need to run faster will have to become parallel programs. (It is considered very unlikely that compilers will be able to automatically find enough parallelism in most sequential programs to solve this problem.) For background on this trend toward parallelism, click [here](http://www.google.com/url?q=http%3A%2F%2Fwww.eecs.berkeley.edu%2FPubs%2FTechRpts%2F2006%2FEECS-2006-183.html&sa=D&sntz=1&usg=AOvVaw2XoPTheTzKVlyjp1G_YcM5).

Students in CS267 will get an overview of the parallel architecture space, gain experience using some of the most popular parallel programming tools, and be exposed to a number of open research questions. The lectures will also cover a broad set of parallelization strategies for applications covering numerical simulation and data analysis to machine learning.   

CS 267: Applications of Parallel Computers

Parallelism and Distributed Systems

Microcode

Pipelining

Superscalar

Very long instruction word (VLIW)

Branch Prediction

Memory protection

Parallel Programming

Superpipelining

Vector Processors

Multithreading

Small Multiprocessors

Multiprocessor Interconnect

Large Multiprocessors

Directory Protocols

Set Architectures

Verilog

Exception handling

Cache (computing)

ELE/COS 475 Computer Architecture Princeton University Fall 2019 This course offers an in-depth understanding of modern computer processor and system architecture. It covers topics like instruction-set architecture, processor organization, cache, memory, multiprocessors, and more. Designed for senior-level undergraduates and first-year graduate students. An in-depth study of the fundamentals of modern computer processor and system architecture. Students will develop a strong theoretical and practical understanding of modern, cutting-edge computer architectures and implementations. Studied topics include: Instruction-set architecture and high-performance processor organization including pipelining, out-of-order execution, as well as data and instruction parallelism. Cache, memory, and storage architectures. Multiprocessors and multicore processors. Coherent caches. Interconnection and network infrastructures. If the course indicates that enrollment capacity has been reached, students are encouraged to come to the class and speak with the professor about enrollment. Approximately 20 pages of reading per week. Homework - 5 problem sets during the semester.  This course is targeted at senior-level undergraduates and first-year graduate students. Students should have a good working understanding of digital logic, basic processor design and organization, pipelining, and simple cache design. ELE 375/COS 375 and ELE 206/COS 306 or equivalent knowledge are necessary to succeed in ELE 475. Use of the Verilog hardware description language will be used in this course and prior knowledge of Verilog will be very helpful, but not required. **Required text:**

1.  Computer Architecture: A Quantitative Approach 5th or 6th edition
    -  John L. Hennessy and David A. Patterson
    -  Princeton University Library Owns

**Recommended Text:**

-  Modern Processor Design: Fundamentals of Superscalar Processors (1st Edition), 2004. First and second printing are the same.
    -  John P. Shen and Mikko H. Lipasti
    -  Approximately $68 on [Amazon](http://www.amazon.com/Modern-Processor-Design-Fundamentals-Superscalar/dp/1478607831/ "http://www.amazon.com/Modern-Processor-Design-Fundamentals-Superscalar/dp/1478607831/")
    -  ISBN: 978-1478607830
    -  Princeton University Library Owns

**Other Text:**

-  Verilog HDL: A Guide to Digital Design and Synthesis, (2nd Edition), 2003.
    -  Samir Palnitkar
    -  ISBN: 0-13-044911-3
    -  This book is a good Verilog reference as we will be using Verilog in the labs for this course. If you already know Verilog for synthesis than this may not be needed. The 1st and 2nd editions of this book work. Likewise there are many other Verilog references on the web and other books are available.
    -  The 1st edition can be purchased used from Amazon for less than $10, the 2nd edition is more expensive.
    -  Princeton University Library has free online access to this book: [Online Safari Version On Princeton Network](http://proquest.safaribooksonline.com/?uiCode=princeun&xmlId=0130449113 "http://proquest.safaribooksonline.com/?uiCode=princeun&xmlId=0130449113")

ELE/COS 475 Computer Architecture

Computer Systems

Classification

Linear regression

Training/Validation/Testing

Perceptron

MNIST

Loss function

Optimization

Backpropagation

Automatic differentiation

Matrix representation of Neural Networks

TensorFlow

Multilayer perceptron

Activation Functions

Convolutional neural network (CNN)

Overfitting

Regularization (mathematics)

Language Models

Word Embeddings

Feedforward neural network

Recurrent neural network (RNN)

Long Short-Term Memory (LSTM)

Gated recurrent unit (GRU)

Machine Translation

Attention (machine learning)

Transformer (machine learning model)

Scaling Deep Learning Systems

Interpretation of Neural Networks

Autoencoders

Variational autoencoder (VAE)

Generative adversarial network (GAN)

Deepfake

Reinforcement learning (RL)

Value iteration

Deep Q-learning

Policy Gradient Methods

Actor-Critic Methods

Graph neural network (GNN)

CSCI 1470/2470 Deep Learning Brown University Spring 2022 Brown University's Deep Learning course acquaints students with the transformative capabilities of deep neural networks in computer vision, NLP, and reinforcement learning. Using the TensorFlow framework, topics like CNNs, RNNs, deepfakes, and reinforcement learning are addressed, with an emphasis on ethical applications and potential societal impacts. Over the past few years, Deep Learning has become a popular area, with deep neural network methods obtaining state-of-the-art results on applications in computer vision (Self-Driving Cars), natural language processing (Google Translate), and reinforcement learning (AlphaGo). These technologies are having transformative effects on our society, including some undesirable ones (e.g. deep fakes).

This course is there to give students a practical understanding of how Deep Learning works, how to implement neural networks, and how to apply them ethically. We introduce students to the core concepts of deep neural networks and survey the techniques used to model complex processes within the contexts of computer vision and natural language processing.

Throughout the course, we emphasize and require students to think critically about potential ethical pitfalls that can result from mis-application of these powerful models. The course is taught using the Tensorflow deep learning framework.
 By the end of this course, you will be able to:

- Learn about the fundamental algorithms that underly all modern deep learning models.
- Implement different types of deep learning models in Tensorflow.
- Think critically about using a deep learning model for a task and its potential societal impact.
- Collaborate with classmates on a team project to apply deep learning models to task of your
choice.
- Communicate your findings (both positive and negative results are encouraged) through pre-
sentations. - A basic programming course: (CSCI 0150, 0170 or 0190)
- A linear algebra course: (CSCI 0530, MATH 0520 or 0540)
- A stats / probability course: (CSCI 0220, 1450, 0450, MATH 1610, APMA 1650 or 1655)

Exceptions may be possible for those missing one of these prerequisites if (a) the student has taken
another course which covers similar material, or if (b) the student will be concurrently taking the
prerequisite. If either of these situations applies to you, use the “Request Override” feature in
Courses@Brown to request an override code (and explain why you believe your situation merits
one). ### Textbook

None required. Students are encouraged to refer to the following textbook, which is available online:

- [Deep Learning](https://www.deeplearningbook.org/), by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

CSCI 1470/2470 Deep Learning

Deep Learning

Graphics processing unit

Graphics Processor (GPU)

3 courses cover this concept

CS 267: Applications of Parallel Computers

ELE/COS 475 Computer Architecture

CSCI 1470/2470 Deep Learning