Graph neural network (GNN)

Convolutional neural network (CNN)

Computer Vision

Recurrent neural network (RNN)

Long Short-Term Memory (LSTM)

Sequence-to-sequence (Seq2Seq)

Attention (machine learning)

Self-supervision

Autoencoders

Transformer (machine learning model)

Meta-learning

Fine-tuning

Generative Models

Transfer learning

CS 182/282A: Deep Neural Networks UC Berkeley Fall 2022 An advanced course dealing with deep networks in the fields of computer vision, language technology, robotics, and control. It delves into the themes of deep learning, model families, and real-world applications. A strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning is necessary. Deep Networks have revolutionized computer vision, language technology, robotics and control. They have a growing impact in many other areas of science and engineering, and increasingly, on commerce and society. They do not however, follow any currently known compact set of theoretical principles. In Yann Lecun's words they require "an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses." This is a fancy way of saying “we don’t understand this stuff nearly well enough, but we have no choice but to muddle through anyway.” This course attempts to cover that ground and show you how to muddle through even as we aspire to do more. The goal is to teach a principled course in Deep Learning that serves the diverse needs of our students while also codifying the present understanding of the field. Topics covered may include, but are not limited to:
- Underlying themes of deep learning, including building beyond underlying machine learning concepts like supervised vs unsupervised learning, regression and classification, training/validation/testing, distribution shifts, regularization, the fundamental underlying tradeoffs;
- Defining and training neural networks: features, computation graphs, backpropagation, iterative optimization (SGD, Newton’s Method, Momentum, RMSProp, AdaGrad, Adam), strategies for training (explicit and implicit regularization, batch and layer normalization, weight initialization, gradient clipping, ensembles, dropout), hyperparameter tuning
- Families of contemporary models: fully connected networks, convolutional nets, graph neural nets, recurrent neural nets, transformers
- Problems that utilize neural networks: computer vision, natural language processing, generative models, and others.
- Conducting experiments in a systematic, repeatable way, leveraging and presenting data from experiments to reason about network behavior. This is a graduate-level/advanced undergraduate course about a particular approach to information processing using (simulated) analog circuits where the desired circuit behavior is tuned via optimization involving data since we have no idea how to do hand-tuning at scale. Probabilistic frames are useful to understand what is going on, as well as how we navigate certain design choices. Overall, we expect students to have a strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning. Berkeley undergraduate courses that can help build maturity include:
- **Calculus**: Math 53 (note: Math 1B or AP Math is not enough)
- **Linear Algebra and Optimization**: EECS 16B and EECS 127/227A is ideal, but EECS 16B alone might be enough if students have complete mastery of that material. Math 110 is also helpful. (note: Math 54 or EECS 16A is required as a minimum, but are not nearly enough.)
- **Probability**: EECS 126, Stat 134, or Stat 140 (note: CS 70 is required at a minimum, but might not be enough for everyone)
- **Statistical Learning**: CS 189/289A or Stat 154 (note: Data 102 is insufficient, even when combined with Data 100.)

*Math 53 and EECS 126 and EECS 127 and CS 189 is the recommended background.*

Prerequisites are not enforced for enrollment, but we encourage you to consider taking some of the classes listed above and save this course for a future semester if you feel shaky on the fundamentals.

The course assumes familiarity with programming in a high-level language with data structures. Homeworks and projects will typically use Python. We encourage you to check out this [tutorial](https://docs.python.org/3/tutorial/) if you haven’t used it before. Students who have taken Berkeley courses like CS 61A and CS 61B are well-prepared for the programming components of the class.

We do not have the staff bandwidth to help students with material that they should have understood before taking this course. If you choose to proceed with this course, you are accepting full responsibility to teach yourself anything in your background that you are missing. We will not be slowing down to accommodate you, and questions pertaining to background material will always have the lowest priority in all course forums. 

CS 182/282A: Deep Neural Networks

Deep Learning

Node Embeddings

Machine Learning with Heterogeneous Graphs

GNNs for Recommender Systems

Geometric Graph Learning

Theory of Graph Neural Networks

Reasoning over Knowledge Graphs

Advanced Topics in GNNs

Label Propagation on Graphs

Fast Neural Subgraph Matching and Counting

Scaling Up GNNs to Large Graphs

Feature Engineering

GNN Augmentation and Training

Knowledge Graph Embeddings

Deep Generative Models for Graphs

Trustworthy Graph AI

CS 224W: Machine Learning with Graphs Stanford University Winter 2023 The course focuses on the analysis of large graphs and uses machine learning to gain insights into social, technological, and biological systems. Topics include Graph Neural Networks, influence maximization, disease outbreak detection, and social network analysis. ## What is this course about?

Complex data can be represented as a graph of relationships between objects. Such networks are a fundamental tool for modeling social, technological, and biological systems. This course focuses on the computational, algorithmic, and modeling challenges specific to the analysis of massive graphs. By means of studying the underlying graph structure and its features, students are introduced to machine learning techniques and data mining tools apt to reveal insights on a variety of networks.
Topics include: representation learning and Graph Neural Networks; algorithms for the World Wide Web; reasoning over Knowledge Graphs; influence maximization; disease outbreak detection, social network analysis.  Students are expected to have the following background:

- Knowledge of basic computer science principles, sufficient to write a reasonably non-trivial computer program (e.g., CS107 or CS145 or equivalent are recommended)
- Familiarity with the basic probability theory (CS109 or Stat116 are sufficient but not necessary)
- Familiarity with the basic linear algebra (any one of Math 51, Math 103, Math 113, or CS 205 would be much more than necessary)

The recitation sessions in the first weeks of the class will give an overview of the expected background. ## Course Materials

Notes and reading assignments will be posted periodically on the course Web site. The following books are recommended as optional reading:

- [Graph Representation Learning](https://www.cs.mcgill.ca/~wlh/grl_book/) by William L. Hamilton
- [Networks, Crowds, and Markets: Reasoning About a Highly Connected World](http://www.cs.cornell.edu/home/kleinber/networks-book/) by David Easley and Jon Kleinberg

## Previous Offerings

You can access slides and project reports of previous versions of the course on our archived websites: [CS224W: Fall 2021](http://snap.stanford.edu/class/cs224w-2021) / [CS224W: Winter 2021](http://snap.stanford.edu/class/cs224w-2020) / [CS224W: Fall 2019](http://snap.stanford.edu/class/cs224w-2019) / [CS224W: Fall 2018](http://snap.stanford.edu/class/cs224w-2018) / [CS224W: Fall 2017](http://snap.stanford.edu/class/cs224w-2017) / [CS224W: Fall 2016](http://snap.stanford.edu/class/cs224w-2016) / [CS224W: Fall 2015](http://snap.stanford.edu/class/cs224w-2015) / [CS224W: Fall 2014](http://snap.stanford.edu/class/cs224w-2014) / [CS224W: Fall 2013](http://snap.stanford.edu/class/cs224w-2013) / [CS224W: Fall 2012](http://snap.stanford.edu/class/cs224w-2012) / [CS224W: Fall 2011](http://snap.stanford.edu/class/cs224w-2011) / [CS224W: Fall 2010](http://snap.stanford.edu/class/cs224w-2010)
- [Network Science](http://networksciencebook.com) by Albert-László Barabási

CS 224W: Machine Learning with Graphs

Machine Learning

Apache Spark

Frequent Itemsets Mining

Locality-Sensitive Hashing

Clustering

Dimensionality Reduction

Recommender Systems

PageRank

Community Detection in Graphs

Graph Representation Learning

Learning Embeddings

Decision Trees

Mining Data Streams

Matrix Sketching

Computational Advertising

Learning through Experimentation

Optimizing Submodular Functions

CS246: Mining Massive Data Sets Stanford University Spring 2023 This course focuses on data mining and machine learning algorithms for large scale data analysis. The emphasis is on parallel algorithms with tools like MapReduce and Spark. Topics include frequent itemsets, locality sensitive hashing, clustering, link analysis, and large-scale supervised machine learning. Familiarity with Java, Python, basic probability theory, linear algebra, and algorithmic analysis is required. ### What is this course about? [[Info Handout](https://web.stanford.edu/class/cs246/handouts/CS246_Info_Handout.pdf)]

The course will discuss data mining and machine learning algorithms for analyzing very large amounts of data. The emphasis will be on MapReduce and [Spark](http://spark.apache.org/) as tools for creating parallel algorithms that can process very large amounts of data.

**Topics include**: Frequent itemsets and Association rules, Near Neighbor Search in High Dimensional Data, Locality Sensitive Hashing (LSH), Dimensionality reduction, Recommendation Systems, Clustering, Link Analysis, Large-scale Supervised Machine Learning, Data streams, Mining the Web for Structured Data, Web Advertising.  Students are expected to have the following background:

- Knowledge of basic computer science principles and skills, at a level sufficient to write a reasonably non-trivial computer program (e.g., CS107 or CS145 or equivalent are recommended).
- Good knowledge of Java and Python will be extremely helpful since most assignments will require the use of Spark.
- Familiarity with basic probability theory (CS109 or Stat116 or equivalent is sufficient but not necessary).
- Familiarity with writing rigorous proofs (at a minimum, at the level of CS 103).
- Familiarity with basic linear algebra (e.g., any of Math 51, Math 103, Math 113, CS 205, or EE 263 would be much more than necessary).
- Familiarity with algorithmic analysis (e.g., CS 161 would be much more than necessary). ### Reference Text

The following text is useful, but not required. It can be downloaded for free, or purchased from Cambridge University Press.

[Leskovec-Rajaraman-Ullman: Mining of Massive Dataset](http://www.mmds.org/)

CS246: Mining Massive Data Sets

Supervised learning

Classification

Linear regression

Training/Validation/Testing

Perceptron

MNIST

Loss function

Optimization

Backpropagation

Automatic differentiation

Matrix representation of Neural Networks

Graphics Processor (GPU)

TensorFlow

Multilayer perceptron

Activation Functions

Overfitting

Regularization (mathematics)

Language Models

Word Embeddings

Feedforward neural network

Gated recurrent unit (GRU)

Machine Translation

Scaling Deep Learning Systems

Interpretation of Neural Networks

Unsupervised learning

Variational autoencoder (VAE)

Generative adversarial network (GAN)

Deepfake

Reinforcement learning (RL)

Value iteration

Deep Q-learning

Policy Gradient Methods

Actor-Critic Methods

CSCI 1470/2470 Deep Learning Brown University Spring 2022 Brown University's Deep Learning course acquaints students with the transformative capabilities of deep neural networks in computer vision, NLP, and reinforcement learning. Using the TensorFlow framework, topics like CNNs, RNNs, deepfakes, and reinforcement learning are addressed, with an emphasis on ethical applications and potential societal impacts. Over the past few years, Deep Learning has become a popular area, with deep neural network methods obtaining state-of-the-art results on applications in computer vision (Self-Driving Cars), natural language processing (Google Translate), and reinforcement learning (AlphaGo). These technologies are having transformative effects on our society, including some undesirable ones (e.g. deep fakes).

This course is there to give students a practical understanding of how Deep Learning works, how to implement neural networks, and how to apply them ethically. We introduce students to the core concepts of deep neural networks and survey the techniques used to model complex processes within the contexts of computer vision and natural language processing.

Throughout the course, we emphasize and require students to think critically about potential ethical pitfalls that can result from mis-application of these powerful models. The course is taught using the Tensorflow deep learning framework.
 By the end of this course, you will be able to:

- Learn about the fundamental algorithms that underly all modern deep learning models.
- Implement different types of deep learning models in Tensorflow.
- Think critically about using a deep learning model for a task and its potential societal impact.
- Collaborate with classmates on a team project to apply deep learning models to task of your
choice.
- Communicate your findings (both positive and negative results are encouraged) through pre-
sentations. - A basic programming course: (CSCI 0150, 0170 or 0190)
- A linear algebra course: (CSCI 0530, MATH 0520 or 0540)
- A stats / probability course: (CSCI 0220, 1450, 0450, MATH 1610, APMA 1650 or 1655)

Exceptions may be possible for those missing one of these prerequisites if (a) the student has taken
another course which covers similar material, or if (b) the student will be concurrently taking the
prerequisite. If either of these situations applies to you, use the “Request Override” feature in
Courses@Brown to request an override code (and explain why you believe your situation merits
one). ### Textbook

None required. Students are encouraged to refer to the following textbook, which is available online:

- [Deep Learning](https://www.deeplearningbook.org/), by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

CSCI 1470/2470 Deep Learning