This template has a responsive menu toggling system. The menu will appear collapsed on smaller screens, and will appear non-collapsed on larger screens. When toggled using one of the buttons below, the menu will appear/disappear. On small screens, the page content will be pushed off canvas.
MO810 / MC959 - Introduction to Deep Learning
Instructor: Joao Meidanis
Tuesdays and Thursdays, 7-9pm, room CB16
First Semester, 2018
Overview
This is an introductory course on Deep Learning, aimed at graduate students. Undergraduate students in their last years of college can also attend. Graduate students must register for MO810, while undergrads should take MC959. The prerequisites are:
- Proficiency in programming. We'll use Python for most of the course. Languages R and Octave (free replacement for Matlab) will be used for graphics, but I haven't tried all examples in these languages. For the more complex examples, it is much more convenient to use a framework such as TensorFlow. There is room in the schedule for a couple of classes on Python, R, Ocatve, and TensorFlow.
- Basic computational performance analysis, complexity concepts.
- Introductory calculus, basic linear algebra.
- Some graph theory, probability, and information theory.
The course will have both a theoretical component and a practical component. Most classes will have a theoretical part and a practical part. In the theoretical side, we'll cover deep forward networks, back-propagation, regularization, convolutional layers, recurrent networks, and learning with no training data. Guides for this part include the Deep Learning book by Goodfellow et al., and an entry in a Karpathy blog.
For the practical part, we'll start with very basic computational neural networks that can learn simple Boolean functions, and then proceed to more complex networks that can learn handwriting/images, sequences, and strategy games. Guides for this part include Nielsen's online book, and another entry in a Karpathy blog.
Grading will be based on in-class short tests , one per lecture, and on student projects.
We won't have access to powerful computers or GPUs. Therefore, our emphasis in the course will be on efficiency, i.e., trying to find the smallest net that does the work, or as much of the work as possible. Ideally, the code each one of us produces or tests will run in our own laptops, desktops, and other computers available to us.
Projects
Part of the grade will come from student projects. Students will choose a problem that they are interested in and design, train, and test a network that solves this problem. We list below a few suggestions, but your are free to choose your own problem. Students can form groups of 2 to 4 members, but in that case each member must have a definite task within the group, declared in advance, and each person will be graded separately according to their own performance.
Students will have to keep a diary, which is a written record of what they've done, to be able to communicate the results and reproduce them later on. There should be entries every day. Plan to keep yourself busy with the course for 2 to 2.5 hours a day, and use the final part of this time to summarize in the diary what you did on that day. Diaries can be requested by the instructor at any time and have to be handed out with the Project Report. Diaries are individual artifacts, so even if you are working in a group, you are responsible for keeping your own diary, because different people will probably de doing different things within the project. Even if you are doing something togehter, each person must write is own account of the events. Don't look at your partner's diary before you write your own, and don't edit your diary afterwards, except to correct typos.
There will be checkpoints for the projects and a final presentation and handout. The checkpoints are as follows:
- April 3rd (One month after classes begin): problem chosen and data obtained.
- May 3rd (Two months after classes begin): model and baseline defined.
- June 14th (Three and a half moths after classes begin): model training, fine tuning, and test. Project Report (9 pages), Diaries, Code, and Video presentation (4 min) due.
- June 14th to 28th: Fine tuning of Project Deliverables. Your reports, presentations, diaries, etc. will be checked and may receive suggestions to be applied in order to merit a better grade.
- Remember: we won't have big computers, or GPUs --- we'll need to be efficient! One of our goals will always be to find the smallest net that does the job.
In summary, students will have one month to select the problem and gather the data; another month to define the model (i.e., network architecture), and baseline (i.e., the best results in the literature for this task); and another ~40 days to train their network, fine tune the hyperparameters of the model, write a report (not longer than 9 pages), and produce a video (4 minutes maximum) summarizing the experience. The code must be made available as well. The rest of the term will be used for us all to collectively review the results of every group and suggest improvements.
Coursework must be sent to the instructor's university email no later than 11:59pm on the due date. Do not attach big files; leave them in an internet server and send just their URL instead. There will be a penalty for late submission of coursework of 0.0139% per minute (equivalent to 20% per day).
Project suggestions
- Email filter (spam killer)
- Computational biology: genes associated with a certain disease
- Handwriting (MNIST, etc.)
- Images (CIFAR, etc.)
- strategy games (checkers. etc.)
- online games (Pong, Dinousaur, etc.)
- predictions in the stock market
- writing poetry
- natural language translation
- mining governmental transparency sites
Office hours
By appointment only.
Program and Schedule
| MO810 A | Topics in Artificial Intelligence | 1st. Term | |
| MC959 A | Introduction to Deep Learning | 2018 | |
| Instructor: João Meidanis | |||
| PRELIMINARY SCHEDULE | |||
| Tu/Th | Date | Theory | Practice |
| Tue | 02/27 | Course outline | Course outline |
| Thu | 03/01 | Introduction | Introduction |
| Tue | 03/06 | Linear Algebra | Simple examples |
| Thu | 03/08 | Linear Algebra | Simple examples |
| Tue | 03/13 | Probability and Information | Examples with Octave, R |
| Thu | 03/15 | Probability and Information | Languages: Octave, R |
| Tue | 03/20 | Numerical Computation | Languages: Python |
| Thu | 03/22 | Numerical Computation | Python, TensorFlow |
| Tue | 03/27 | Machine Learning | TensorFlow, IRIS, XOR |
| Thu | 03/29 | Holiday: no class | Holiday: no class |
| Tue | 04/03 | Projects: topic, data | Projects: topic, data |
| Thu | 04/05 | Machine Learning | TensorBoard |
| Tue | 04/10 | Deep Forward Nets | MNIST |
| Thu | 04/12 | Deep Forward Nets | MNIST |
| Tue | 04/17 | Regularization | CPF Control Digits |
| Thu | 04/19 | Regularization | CPF Control Digits |
| Tue | 04/24 | Optimization | Tic-Tac-Toe |
| Thu | 04/26 | Optimization | Tic-Tac-Toe |
| Tue | 05/01 | Holiday: no class | Holiday: no class |
| Wed | 05/02 | Last day for MO810 drop requests | |
| Thu | 05/03 | Projects: model, baseline | Projects: model, baseline |
| Tue | 05/08 | Convolutional Nets | Convolutional Net |
| Thu | 05/10 | Convolutional Nets | Convolutional Net |
| Tue | 05/15 | Sequence Modeling: Recurrence | Sentiment from text , Torch Shakespeare |
| Thu | 05/17 | Sequence Modeling: Recurrence | PTB , Translation |
| Mon | 05/21 | Last day for enrollment suspension requests | |
| Tue | 05/22 | Learning with no data | DNA classification |
| Thu | 05/24 | Learning with no data | DNA classification |
| Tue | 05/29 | Classes suspended | Classes suspended |
| Thu | 05/31 | Holiday: no class | Holiday: no class |
| Tue | 06/05 | Practical Methodology | Street View House Numbers |
| Thu | 06/07 | Applications | TicTacToe/Policy Gradient |
| Tue | 06/12 | Applications | Translation |
| Thu | 06/14 | Projects Due | Projects Due |
| Sun | 06/17 | Brazil vs Switzerland, 3pm BRT (UTC -3) | |
| Tue | 06/19 | Projects: Reviews | Projects: Reviews |
| Thu | 06/21 | Projects: Reviews | Projects: Reviews |
| Fri | 06/22 | Brazil vs Costa Rica, 9am BRT (UTC -3) | |
| Tue | 06/26 | Projects: Reviews | Projects: Reviews |
| Wed | 06/27 | Serbia vs Brazil, 3pm BRT (UTC -3) | |
| Thu | 06/28 | Projects: Reviews | Projects: Reviews |
| Mon | 07/02 | Possible Brazil Match, 12pm (UTC -3) | |
| Tue | 07/03 | Possible Brazil Match, 11am (UTC -3) | |
| Tue | 07/03 | Week for study (undergrads only) | |
| Thu | 07/05 | Week for study (undergrads only) | |
| Fri | 07/06 | Possible Brazil Match, 3pm (UTC -3) | |
| Sat | 07/07 | Possible Brazil Match, 12pm (UTC -3) | |
| Tue | 07/10 | Possible Brazil Match, 3pm (UTC -3) | |
| Tue | 07/10 | Exam (undergraduates only) | |
| Thu | 07/12 |
Grading
Grading for both graduate and undergraduate students will be based on the score S obtained by the student during the term. For gradute students, this score will then be converted to a letter grade G using the table below. For undergraduates, the score, possibly averaged with the result from the final exame E, will be sent to the university administration as the final grade G.
The score S will be determined by the student's performance in the following activities:
- in-class tests (score: T)
- project (score: P)
The mean of T and P will be the student's score S, that is, S = (T+P)/2.
Test scores
In-class tests will be given on the last 15 minutes of each class, covering the previous' class topics. The tests are closed-book. Typically, there will be one of two questions per test. Each in-class test will be graded on a scale of 0 to 10. Only the k=19 best grades are kept, with should be about 75% to 80% of the number of tests administered. This way, a student can skip a few tests without significant impact on their grade. The average of the best k tests will be the test score T.
Project scores
Projects will be graded taking into account the following aspects:
- 15%: problem relevance, difficulty, compliance (adherence to stated topic, data, model, and baseline)
- 20%: student diary
- 25%: final report, after suggestions (clarity, enough information to reproduce the results, organization, conciseness, etc.) - Max. 9 pages.
- 20%: code (style, comments, efficiency, etc.)
- 20%: video presentation (clarity, objectivity, respect to time limit, etc.) - Max. 4 minutes.
Guides for coding style: PEP 8 , Google Python Style Guide , Hithchiker's Guide to Python . Projects will be graded on a 0 to 10 scale, to yield a project score P.
Score-to-letter-grade table
To be used for graduate students.
| Score | Grade |
|---|---|
| 8.5 ≤ S ≤ 10 | A |
| 7.0 ≤ S < 8.5 | B |
| 5.0 ≤ S < 7.0 | C |
| 0.0 ≤ S < 5.0 | D |
Final Exam
Undergraduate students who end up with a score S below 5.0 may take a final exam, and their grade E on this final exam will be combined with S to form the final grade:
G = (S + E)/2.
References
Deep Learning. Ian Goodfellow and Yoshua Bengio and Aaron Courville. MIT Press, 2016. www.deeplearningbook.org
The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy. Blog, 2015. karpathy.github.io/2015/05/21/rnn-effectiveness/
Deep Reinforcement Learning: Pong from Pixels. Andrej Karpathy. Blog, 2016. karpathy.github.io/2016/05/31/rl/
Neural Networks and Deep Learning. Michael A. Nielsen. Determination Press, 2015. neuralnetworksanddeeplearning.com