# Tokenizer crash when processing undecodable source code

Malformed non-UTF-8 Python source makes tokenizer's tok_nextc loop increment past heap buffer, causing OOB read and segfault.

- Payout: hidden
- Severity: low
- Program: Internet Bug Bounty
- Found by: hugbounter
- Disclosed: 2015-04-21
- Write-up quality: 6/10
- Original: https://hackerone.com/reports/73248
- Library page: https://bugbountylibrary.com/r/h1%3A73248
- Tags: area:oss domain:oss impact:dos kind:bounty platform:hackerone sev:low tech:python technique:fuzzing technique:source-review vuln:dos vuln:memory-corruption

**Target:** Python tokenizer (Parser/tokenizer.c) parsing a source file
**Root cause:** When a malformed/non-UTF-8 script is tokenized, the for loop in tok_nextc never hits an exit condition, so *tok->cur is incremented indefinitely past the allocated buffer, reading heap out of bounds.
**Steps:**
1. Craft a malformed Python script with non-UTF-8 bytes and no encoding declaration (attached heap_pattern.py / heap3.py / heap4_asan.py).
2. Run `./python ~/heap3.py` (ASAN build) to observe heap-buffer-overflow at Parser/tokenizer.c:1021 tok_nextc.
3. Run `./python ~/heap4_asan.py` to observe heap-use-after-free at Parser/tokenizer.c:902 tok_nextc.
4. Without ASAN, run `./python ~/heap1.py` under gdb; SIGSEGV at Parser/tokenizer.c:922 `return Py_CHARMASK(*tok->cur++);`.
**How found:** Fuzzing Python with malformed scripts; noticed inconsistent behavior (sometimes SyntaxError, sometimes crash) and traced the infinite increment of tok->cur in tok_nextc via ASAN and gdb.
**Exploitation:** Requires ability to supply a script for the victim to execute; results in crash/DoS of the Python process. Reporter could not confirm exploitability beyond infoleak/DoS.
**Insight:** Inconsistent outcomes (clean error vs crash) on the same malformed input signal a loop whose exit conditions depend on data the parser can't decode; instrument with ASAN to distinguish OOB read from UAF in the same tokenizer path.