CNIT 126: Practical Malware Analysis

Spring 2017 Sam Bowne

37184 Mon 6:10 - 9 PM SCIE 200

Schedule · Lecture Notes · Projects · Links · Training · Home Page


Catalog Description

Learn how to analyze malware, including computer viruses, trojans, and rootkits, using disassemblers, debuggers, static and dynamic analysis, using IDA Pro, OllyDbg and other tools.

Advisory: CS 110A or equivalent familiarity with programming

Upon successful completion of this course, the student will be able to:
  1. Describe types of malware, including rootkits, Trojans, and viruses.
  2. Perform basic static analysis with antivirus scanning and strings
  3. Perform basic dynamic analysis with a sandbox
  4. Perform advanced static analysis with IDA Pro
  5. Perform advanced dynamic analysis with a debugger
  6. Operate a kernel debugger
  7. Explain malware behavior, including launching, encoding, and network signatures
  8. Understand anti-reverse-engineering techniques that impede the use of disassemblers, debuggers, and virtual machines
  9. Recognize common packers and how to unpack them


"Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software", by Michael Sikorski and Andrew Honig; ISBN-10: 1593272901 Buy from Amazon


The quizzes are multiple-choice, online, and open-book. However, you may not ask other people to help you during the quizzes. You will need to study the textbook chapter before the lecture covering it, and take the quiz before that class. Each quiz is available for one week, up till 8:30 am Saturday. Each quiz has 5 questions, you have ten minutes to take it, and you can make two attempts. If you take the quiz twice, the second score is the one that counts, not necessarily the higher score.

To take quizzes, first claim your RAM ID and then log in to Canvas here:

Schedule (may be revised)

Note: Chapter Numbers are one too high in the E-Book: Chapter 0 is mislabelled as Chapter 1, etc.

Mon 1-23  0: Malware Analysis Primer &
1: Basic Static Techniques

Mon 1-30  2: Malware Analysis in Virtual Machines &
3: Basic Dynamic Analysis

Fri 2-3 Last Day to Add Classes
Mon 2-6 Ch 0-1 Quiz due before class
Ch 4 Quiz due before class
Proj 1-2 due
4: A Crash Course in x86 Disassembly

Mon 2-13 Ch 2-3 Quiz due before class
Ch 5 Quiz due before class
Proj 3 due
5: IDA Pro

Mon 2-20 Holiday - No Class

Mon 2-27 Ch 6 Quiz due before class
Proj 4-5 due
6: Recognizing C Code Constructs in Assembly
Mon 3-6 Ch 7 Quiz due before class
Proj 6 due
7: Analyzing Malicious Windows Programs
Mon 3-13 Ch 8 Quiz due before class
Proj 7-8 due
8: Debugging
Mon 3-20 Ch 9 Quiz due before class
Proj 9 due
9: OllyDbg
Mon 3-27 Holiday - No Class
Mon 4-3 Ch 10 Quiz due before class
Proj 10-11 due
10: Kernel Debugging with WinDbg
Wed 4-6 Mid-Term Grades Due
Mon 4-10 Ch 11 Quiz due before class
Proj 12 due
11: Malware Behavior

Mon 4-17No Quiz
No Proj due
Guest Speaker: Jason Nelson
Desktop Support Technician at Child Mind Institute |
Digital Advertising/Social Media consultant to
lifestyle brands and non-profits

Mon 4-24 Ch 12 Quiz due before class
Proj 13 & 14 due
12: Covert Malware Launching
Mon 5-1 Ch 13 Quiz due before class
Proj 15 due
13: Data Encoding
Mon 5-8 Ch 14 Quiz due before class
Proj 16 due
14: Malware-Focused Network Signatures
Mon 5-15 Last Class · Ch 15 Quiz due before class
All extra credit Proj. due
15: Anti-Disassembly
Mon 5-23  Final Exam

Lecture Notes

Printable Schedule

Basic Analysis

0: Malware Analysis Primer & 1: Basic Static Techniques · KEY · PDF
2: Malware Analysis in Virtual Machines & 3: Basic Dynamic Analysis · KEY · PDF

Advanced Static Analysis

4: A Crash Course in x86 Disassembly · KEY · PDF
5: IDA Pro · KEY · PDF

Slides below this line are being updated

6: Recognizing C Code Constructs in Assembly     PPTX
7: Analyzing Malicious Windows Programs     PPT

Advanced Dynamic Analysis

8: Debugging     KEY     HTML
9: OllyDbg     KEY    HTML
10: Kernel Debugging with WinDbg     KEY    HTML

Malware Functionality

11: Malware Behavior     KEY    HTML
12: Covert Malware Launching     KEY    HTML
13: Data Encoding     KEY    HTML
14: Malware-Focused Network Signatures     KEY


15: Anti-Disassembly     KEY
16: Anti-Debugging
17: Anti-Virtual Machine Techniques
18: Packers and Unpacking

Special Topics

19: Shellcode Analysis
20: C++ Analysis
21: 64-Bit Malware

Review Questions

Student Presentations

Honeypot and Malware: Alan Wennersten and Jeffrey Tom

Click a lecture name to see it on SlideShare. If you want to use other formats, you may find this useful: Cloud Convert.

Back to Top


Download Textbook Labs Here

Downloading the Virtual Machines

Download VMware Player

Proj 1: Basic Static Techniques (Lab 1-1) (25 pts.)
Proj 2: Basic Static Techniques (Lab 1-2) (20 pts.)
Proj 3: INetSim (20 pts.) (rev. 2-1-16)
Proj 4: Basic Dynamic Techniques (Lab 3-1) (30 pts.) (rev. 2-1-16)
Proj 5: Using Jasmin to run x86 Assembly Code (15 pts.)
Proj 6: IDA Pro (20 pts.) (rev. 2-22-16)

Proj 7: Compiling C on Windows 7 (15 pts.) (rev. 3-7-16)
Proj 7: Compiling C on Windows 10 (15 pts.) (rev. 3-7-16)

Proj 8: Disassembling C on Windows (15 pts. + 10 extra credit)
Proj 9: Disassembling C on Windows Part 2 (15 pts. + 10 extra credit)
Proj 10: Analyzing Malicious Windows Programs (Lab 7-1) (15 pts.) (rev. 3-14-16)
Proj 11: Using OllyDbg to Analyze Lab09-01.exe (rev. 3-21-16) (15 pts.)

I recommend using Livekd to do proj. 12 & 13

Proj 12: Kernel Debugging with Livekd on Windows Server 2008 (20 pts.)
Proj 12: Kernel Debugging with Livekd on Windows 10 (20 pts.)
Proj 13: Using Kernel Debugging Commands with WinDbg (15 pts.)

Here are the older projects, performing full kernel debugging with two connected PC's. They are much more difficult to do. They are worth extra credit if you get them working, but I wouldn't bother.

Proj 12: Kernel Debugging with WinDbg (PC version) (20 pts.)
        (Mac version)

Project 11x: Kernel Debugging with WinDbg over Ethernet with Windows 8 (20 pts.)

Proj 13: Using Kernel Debugging Commands with WinDbg (15 pts.)

Proj 14: Malware Behavior (Lab 11-1) (35 pts.)
Proj 15: Covert Malware Launching (Lab 12-1) (rev. 4-18-16) (25 pts.)
Proj 16: Data Encoding (Lab 13-1) (25 pts.) (rev. 4-25-16)

Extra Credit Projects

Proj 1x: File and Strings (10 pts. extra credit) (rev. 8-22-13)
Proj 2x: Using IDA Pro Free to Disassemble Executable Files (10 - 40 pts. extra credit)
      crackme-121-1.exe       crackme-121-2.exe       crackme-121-3.exe       crackme-121-4.exe
Proj 3x: Harvesting Files from Packet Captures with Wireshark (10 pts.)
      pX12-121.pcap (1.2 MB)
Proj 4x: Introduction to Hopper (20 points) (rev. 2-22-16)
Proj 5x: Assembly Code Challenges (30 points) (rev. 2-9-16)
Proj 6x: Disassembling C on Windows Part 3 (15 pts. + 10 extra credit)
Proj 7x: Analyzing Malicious Windows Programs (Lab 7-2) (15 pts.)
Proj 8x: Using WinDbg on a Crash Dump (15 pts.)
Proj 9x: Using WinDbg on a Crash Dump from gogoCLIENT (10 pts.)
Project 11x: Kernel Debugging with WinDbg over Ethernet with Windows 8 (20 pts.)
Project 12x: Anti-Disassembly (Lab 15-1) (15 pts.)

Back to Top


Lab Files

Download Textbook Labs Here

Chapter Links

Ch 1a: Breach clean-up cost LinkedIn nearly $1 million, another $2-3 million in upgrades (Aug. 2012)
Ch 1b: Fake FBI warning tricks man into surrendering himself for possession of child porn

Ch 2a: VirusTotal - Free Online Virus, Malware and URL Scanner
Ch 2b: UPX NotCompressibleException
Ch 2c: Peering Inside the PE: A Tour of the Win32 Portable Executable File Format
Ch 2d: Dependency Walker (depends.exe) Home Page
Ch 2e: PEview Download
Ch 2f: Resource Hacker
Ch 2g: Download PEiD 0.95
Ch 2h: UPX: the Ultimate Packer for eXecutables - Download Ch 2i: BinText 3.03 McAfee Free Tools

Ch 3a: Process Monitor Download
Ch 3b: Process Explorer Download
Ch 3c: RegShot download
Ch 3d: Regshot user guide
Ch 3e: ApateDNS Download
Ch 3f: 3 Free Tools to Fake DNS Responses for Malware Analysis

Ch 5a: OpenRCE -- Free IDA Scripts

Ch 6a: Entry points for Windows programs

Ch 7b: Autoruns for Windows
Ch 7c: Anatomy of a Program in Memory
Ch 7d: assembly - The point of test eax eax
Ch 7e: CurrentControlSetServices Subkey Entries
Ch 7f: Globally unique identifier - Wikipedia
Ch 7g: SEH in x86 Environments
Ch 7h: assembly - What is the 'FS''GS' register intended for?
Ch 7i: winapi - FS register in Win32
Ch 7j: Ring (computer security) - Wikipedia

Ch 8a: Exploit Development for Mere Mortals Joe McCray - YouTube
Ch 8b: x86 Protected Mode Exceptions

Ch 9a: Download OllyDbg 1.10
Ch 9b: OllyDbg v. 2.01 is EVIL; just misses functions found in v. 1.10
Ch 9c: OLLYDBG TUTORIALS! The Legend Of Random
Ch 9d: OpenRCE OllyDbg Plugins (down on 10-14-13)
Ch 9e: shell-storm Shellcodes Database

Ch 10a: Download Windows Symbol Packages
Ch 10b: ntoskrnl.exe - Wikipedia, the free encyclopedia
Ch 10c: Choosing the 32-Bit or 64-Bit Debugging Tools (Windows Debuggers)
Ch 10d: How To: Debug the WRK on Mac OS X Using VMware Fusion
Ch 10e: Assembly Code Debugging in WinDbg (Windows Debuggers)
Ch 10f: Microsoft Windows library files - HAL runs in kernel mode
Ch 10g: Windbg Tutorials
Ch 10h: A word for WinDbg

Ch 11a: Portable Executable - Wikipedia
Ch 11b: Resource Hacker

Ch 13a: Tools for Examining XOR Obfuscation for Malware Analysis
Ch 13b: Base64 Decode and Encode - Online
Ch 13c:: Download FindCrypt2 (IDA Pro Plug-In)
Ch 13d: Kanal Free Download
Ch 13e: Entropy (information theory) - Wikipedia
Ch 13f: IDA Entropy Plugin
Ch 13g: IDA Entropy Plugin 0.1 -- working download link
Ch 13h: Ent -- entropy visualizer that works on Windows

Training Materials

Introductory: Chapter 0

Introduction to Malware Analysis Slides by Lenny Zeltser
Introduction to Malware Analysis - Free Recorded Webcast by Lenny Zeltser
Sam's Honeynet "Reverse Engineering Malware" Class Notes (Mar. 2012)

Assembly Language: Chapter 4

Windows Assembly Language Megaprimer -- VIDEO
Introductory Intel x86: Architecture, Assembly --Free class materials!
PE Structure--Excellent Diagram
Download jasmin x86 Assembler Interpreter
Jasmin tutorial - Java Assembler Interpreter

Windows Internals: Chapter 7

Windows 0wn3d By Default Mark Baggett -- VIDEO

Debugging: Chapter 8

Exploit Development for Mere Mortals Joe McCray -- VIDEO OllyDbg Tricks for Exploit Development

OllyDbg: Chapter 9

Exploit Dev Night School Day 2 - YouTube -- HIGHLY RECOMMENDED, MORE DEBUGGER DEMOS
Reverse Engineering 101 on Vimeo

Other Links

Catalog of key Windows kernel data structures
Malware Analysis Resources
Pwning a Spammer's Keylogger - SpiderLabs Anterior
SANS Memory Forensics Cheat Sheet (PDF)
An interesting case of Mac OSX malware
Picking Apart Malware In The Cloud - The business need for malware analysis
FakeNet -- Dynamic malware analysis tool
Static Analysis Talk
Worm 2.0, or LilyJade in action
Pwning the Herpes bothet and it's creator
A technical analysis of Adobe Flash Player CVE-2012-0779 Vulnerability - Microsoft Malware Protection Center - Site Home - TechNet Blogs
Virtual USB Analyzer - Tutorial
PolyPack: An Automated Online Packing Service for Optimal Antivirus Evasion
FileInsight McAfee Free Tools
McAfee FileInsight -- recommended malware analysis tool
CSI:Internet - PDF timebomb
Static Analysis: Following Along at Home with Hopper's Decompiler Feature, Part 1
Deconstructing an ELF File
Malware Analysis Course Lecture Slides
Defeating Flame String Obfuscation with IDAPython
System Forensics: MBR Malware Analysis
Malware Hunting with the Sysinternals Tools
Honeypot Alert PHP-CGI Vuln Targeted For Database Dumping
Th3-0uTl4wS Database -- bot source code
Fuzzy Hashing presentation by Jesse Kornblum
Malware Unpacking Level: Pintool
WireShnork and other Forensics plugins for Wireshark
Tweaking Metasploit Modules To Bypass EMET -- Part 1
corkami - reverse engineering experiments and documentations
Modifying VirtualBox settings for malware analysis
What was that Wiper thing? - EXCELLENT MALWARE ANALYSIS
Malware Must Die!: Racing with time to get the latest payload of Blackhole Exploit Kit
Extracting EXE file (in HTTP stream) from captured packets file with Wireshark
Analyzing Unknown Malware: #2 Disclosure of an interesting Botnet - The Executable (Part 1)
Malware Analysis as a Hobby slides --Cuckoo looks great!
Shamoom The Wiper: further details (Part II) - Securelist
Backdoors are Forever: Hacking Team and the Targeting of Dissent
The Case of the Unexplained FTP Connections
Analysis of malware that infects virtual machines
Deobfuscating "PluginDetect"
To Russia With Targeted Attack
Windows DLL Injection Basics
Reverse engineering challenge intended for women
India APT Attack -- Several useful tools demonstrated
MFT vs Super Timeline: Part 1
Stack Smashing On A Modern Linux System -- Good gdb examples -- EXCELLENT HONEYPOT DATA
Oh, you found a remote OpenSSH 0-day on Pastebin? Don't trust it. -- Site to get real malware samples
MalwareURL -- Site to get real malware samples
Malc0de Database -- Site to get real malware samples
PEiD 0.95 Free - Detects packers, cryptors and compilers
QUnpack -- recommended unpacker
ThreatExpert - Automated Threat Analysis
TCPView for Windows -- traffic monitoring
Total Uninstall Analyze, monitor and uninstall programs -- useful for malware analysis
Malware Analyst's Cookbook and DVD: Tools and Techniques for Fighting Malicious Code -- recommended book
Volatility Cheat Sheet
Good analysis of the malware at
Free Online Malware Analysis Class
APT #TargetedAttacks within Twitter
How to use MANDIANT Memoryze
contagio: Collection of Pcap files from malware analysis
Malware analysis lab tools
6.25 DNS DDOS Attack In Korea -- Good example of simple dynamic analysis
Mandiant Redline is Free
Windows 8 Server 2012 Memory Forensics
Structured Exception Handler EXPLOITATION
Malware and DLLs
Trojaning antivirus uninstallers with DLL injection
When Malware Meets Rootkits (from 2005)
Process Hiding
Dr. Fu's Security Blog: Malware Analysis Tutorials: a Reverse Engineering Approach -- MORE PROJECTS HERE
Malware Analyst's Cookbook and DVD: Tools and Techniques for Fighting Malicious Code -- RECOMMENDED BOOK
SANS Work Study -- Get SANS classes for cheap!
Finding Evil: Automating Autoruns Analysis
Attackers' Toolbox Makes Malware Detection More Difficult
Large botnet cause of recent Tor network overload
Pushdo Botnet detects "FakeNet" analysis tool and spams (Sept, 2013)
Reverse Engineering a D-Link Backdoor with IDA Pro
Anatomy of an exploit -- inside the CVE-2013-3893 Internet Explorer zero-day -- Part 1
binwalk - Firmware Analysis Tool
Reverse Engineering Videos
How to solve Windows system crashes in minutes --Debugging crash dumps
Kernel Pool Exploitation on Windows 7 (from 2011)
Analysis of a Malware ROP Chain
New Tool: XORStrings
Strings from CSRSS show command-line history on Windows
Reconstructing Master File Table (MFT) Entries with
The OpenIOC Framework -- for sharing threat intelligence
security-onion - recommended for Snort GUIs
Malware Research -- samples
Barracuda Launches Web-Based Malware Analysis Tool Threatglass
Malware Analysis with pedump
Practical Malware Analysis - Free Download eBook - pdf (works as of 2014-07-16)
What is a mutex? - EPIC EXPLANATION
OfficeMalScanner -- detects malware in Office files
Hopper -- Mac OS X Disassembler, highly recommended by @iamevltwin
fseventer for Mac -- observe filesystem changes
logkext - Freeware keylogger for OS X
contagio: OSX malware and exploit collection (~100 files)
Shellter -- inject Metasploit payloads into PE files to bypass AV
Exeinfo PE Download
How to setup plugins for ollydbg 2.x.x?
Download OllyScript to Automate Packing
Download OllyScript PE Compact Script
QuickUnpack Tool -- Download
Ether: Malware Analysis via Hardware Virtualization Exsensions -- Free online unpacker
MacMemoryForensics - volatility - Instructions on how access and use the Mac OS X support
PEStudio performs the static investigation of Windows executables
Valgrind Tutorial
PEStudio: static malware analysis tool ty @lennyzeltser #S4con
Process Hacker can dump strings from running processes ty @lennyzeltser #S4con
Google mutant names to help identify malware ty @lennyzeltser #S4con
Malware Analysis Database -- search for mutex values & more ty @lennyzeltser #S4con
ProcDOT - Visual Malware Analysis ty @lennyzeltser #S4con Website Reputation Checker Tool ty @lennyzeltser #S4con
Exeinfo PE -- Identifies packers ty @lennyzeltser #S4con
Hacker Disassembly Uncovered (free download)
Reversing & Malware Analysis - FREE TRAINING SLIDES
The evolution of OS X malware (Oct. 2014)
Bypassing EMET's EAF with custom shellcode using kernel pointer (from 2011)
Disarming Enhanced Mitigation Experience Toolkit (EMET) v 5.0
Cuckoo Sandbox VM Escape Vulnerability (2014)
Rootkits by Csaba Barta (from 2009)
Malwr - Malware Analysis by Cuckoo Sandbox
Malware Investigator -- from the FBI
Reversing a malvertisment: javascript, regex, and cookie
POWELIKS Levels Up With New Autostart Mechanism
Malicious Flash Files Gain the Upper Hand With New Obfuscation Techniques Security Intelligence Blog
Inside a Kippo honeypot: how the billgates botnet spreads -- PROJECT IDEA
Hook Analyser
Online JavaScript beautifier -- deobfuscates code! -- IMPORTANT FOR MALWARE ANALYSIS
Retrieve the apk signature at runtime for Android
2015-10-07: IOS Application Security Testing Cheat Sheet - OWASP
theZoo Malware Samples to Analyze ty @the_fire_dog
Malware Researcher\'s Handbook (Demystifying PE File) - InfoSec Resources
RPISEC/Malware: Course materials for Malware Analysis
Malware Analysis by Abstruse Goose
A Crash Course In DLL Hijacking -- EXCELLENT EXPLANATION
x64dbg: An open-source x64/x32 debugger for windows -- ALTERNATIVE TO IDA PRO

New Unsorted Links

Microsoft security technology EMET used to disable itself (Feb. 2016)
The Ultimate Disassembly Framework -- Capstone
Malwarebytes DLL Hijacking (works on Win 2008 Server but not Win 10) -- SHOW TO CLASS
Win32 Assembly Cheat Sheet
Ch 8c: Enabling Postmortem Debugging - Windows 10 hardware dev
Ch 8d: Using Windows Event Viewer to debug crashes
Local Kernel-Mode Debugging - Windows 10 hardware dev
WinDbg tools and tutorials
Ch 10i: Kernel Patch Protection - Wikipedia
Ch 11c: Capturing Windows 7 Credentials at Logon Using a Custom Credential Provider (Replaces MSGINA.DLL)
Ch 11d: Detecting DLL Hijacking on Windows | SANS Institute (2015)
Ch 11e: Windows 10 Hooking Nirvana explained (2016)
Linux Assembly Tutorial - Step-by-Step Guide
Ch 15a: The Bastard Linux Disassembler (Linear)
Ch 15b: JUMP and CALL - Stack Overflow
pestudio: Malware Initial Assessment Tool
Identifying malware with PEStudio
A fundamental introduction to x86 assembly programming
Practical Malware Analysis Starter Kit
Introductory Intel x86: Architecture, Assembly, Applications - YouTube
Assembly Primer for Hackers (Part 1) System Organization Tutorial.mp4 - YouTube
Automatically Extracting Obfuscated Strings from Malware using the FireEye Labs Obfuscated String Solver (FLOSS)
GitHub - RPISEC/Malware: Course materials for Malware Analysis by RPISEC
Manalyzer: free online static analysis
WARNING: Tweet to download live Locky malware (BE CAREFUL)
Kwetza: infecting android applications -- MAKE INTO PROJECT
pwning bin2json | psych0tik
Microsoft/binskim: A binary static analysis tool that provides security and correctness results for Windows portable executables.
GitHub - GoSecure/malboxes: Builds malware analysis Windows VMs so that you don't have to.
pev - the PE file analysis toolkit -- MAY BE USEFUL FOR PROJECTS
pev Video Demo
Plasma is an interactive disassembler for x86/ARM/MIPS. It can generates indented pseudo-code with colored syntax. -- TRY FOR PROJECTS

Back to Top
Last Updated: 2-13-17 7:47 pm