AI/ML○Pass
Krira Augment – Production-ready RAG in minutes
Waitlist for RAG platform launching in 2 months with no demo.
Bold Bet
kriralabs
103mo ago
Back to browse
GitHub Repository
Open-source web crawler in Rust
1 starsHTML
I built a production-ready web crawler in Rust with TTL and anti-dupe
by qmay-rust·Jun 19, 2026·2 points·1 comment
AI Analysis
●MidShip It
Competent Rust crawler when Scrapy and crawlee already solve this.
Strengths
- •Tokio-based async architecture handles concurrent requests efficiently
- •Pause/resume preserves queue state without losing progress
Weaknesses
- •Web crawling is a solved category with mature open-source alternatives
- •Only 3 commits and zero stars suggests early-stage without traction
Category
Target Audience
Developers building search indexes or content aggregators
Similar To
Scrapy · crawlee · Heritrix
Similar Projects
Developer Tools●●Solid
Fullbleed – Rust HTML/CSS-to-PDF with Deterministic Output+Python CLI
It skips headless Chromium entirely and implements an HTML/CSS-to-PDF pipeline in Rust, exposing a Python wheel and CLI that releases the GIL and uses Rayon for parallel batch renders. The deterministic bits — fixed-point base unit, --repro-record/--repro-check, SHA256 outputs and vendored assets — are a clear, practical play for audited VDP/transactional workflows; what's still unknown is CSS spec coverage and whether subtle print-layout quirks will require hand-holding.
WizardryNiche Gem
krflol
204mo ago
Developer Tools●●Solid
DiggaByte Labs – pick your stack, download production-ready SaaS code
Drop-in SaaS scaffolding with Stripe, JWT, Prisma, and Docker pre-wired for $19/project.
Solve My ProblemShip It
GraysoftDev
103mo ago
Developer Tools●●Solid
SiteOne Crawler – Single-binary CI/CD gate for web quality regressions
Rust-powered crawler with CI/CD gates where Screaming Frog costs money.
SlickShip It
janreges
323mo ago
Developer Tools●Mid
Repolog, website audit for SEO, performance, security, and AI readiness
Yet another audit tool bundling Lighthouse, securityheaders.com, and basic robots.txt checks.
Slick
petersas
6024d ago
Developer Tools●Mid
Production-Ready Django and React Auth Starter
Wires the annoying auth plumbing — JWT with refresh/rotation, email activation, password reset, UUID user model, rate limiting and Swagger out of the box — so you can skip weeks of setup. The frontend is modern (TS + Vite, Tailwind, Radix) and includes route guards, but this is an incremental starter in a crowded space; I'd like to see deployment/CI examples or opinionated infra choices to make it truly turnkey.
Niche GemSolve My Problem
okchildhood
114mo ago