Reducing a 66-node dependency cycle to 13 in Scrapy

I built PViz to help developers understand the structural shape of large codebases — surfacing module coupling, strongly connected components, and the edges driving them.

To put it through its paces, I ran it on Scrapy with a concrete goal: reduce SCCs without changing runtime behavior.

Starting point → Final result: - Largest conceptual (TYPE_CHECKING-masked) SCC: 66 → 15 nodes - Runtime SCC: 23 → 2 nodes

Going in with no prior knowledge of the codebase, the refactor took 68 iterations and surfaced some non-obvious structural behaviors:

- Runtime coupling collapsed early (23 → 4 by iteration 17) while the conceptual graph stayed largely intact — suggesting runtime and conceptual coupling respond to different kinds of changes - A ~24 iteration plateau (iterations 27–50) where the conceptual SCC held at 30 nodes, indicating a load-bearing architectural core that couldn’t be decomposed incrementally - A “kernel break” at iteration 51 where the crawler, engine, scraper, and spider middleware all exited the SCC in a single step — nonlinear progress after a long stall - A deliberate regression at the end (13 → 15): HTTP-layer coupling was identified as structurally necessary during testing and reinstated

The full progression is documented through curated dependency snapshots across key iterations, along with test logs, a detailed analysis report, and compressed analysis bundles.

Happy to discuss if you find this interesting.