r/webhosting
Hosting for small Next.js / Node apps that need to be reliably crawlable by AI bots, what are you using?
- upvotes
- 4
- comments
- 22
Post
I'm running a couple of small Node apps (Next.js plus a lightweight JSON service) behind Cloudflare, and making them readable by AI crawlers (GPTBot, ClaudeBot, PerplexityBot and the like) is a core requirement for me, not a nice-to-have. I've found that shared and managed hosting platforms increasingly apply their own bot filtering at the server level. It's often invisible in their dashboards and not configurable per site, so a crawler you explicitly allow in robots.txt can still be turned away before it reaches your app. I'm re-evaluating where I host because of it. Currently struggling a lot with my current provider and it's also quite annoying that you allow bots in the panel and then find out their requests got blocked when you look into logs. What I'm looking for: - a long-running Node 24 process - Easy deployment, preferably from GitHub but I can handle a complex CI/CD as well - no platform-level bot filtering I can't control (Cloudflare handles that for me) - a sensible price for a startup with currently low to medium traffic. I'm weighing managed container platforms (Railway, Render, Fly.io), a plain VPS behind Cloudflare, and Vercel / Cloudflare's own compute, even thinking about AWS, Azure or Google. Curious what others are using for similar setups, and whether you've actually verified that AI crawlers get through, e.g. in your server logs or Cloudflare's...
Keep reading with a free account
The rest of this post, and every signal for DigitalOcean, is in your free account.
Extracted from these lines
[comment u/Environmental-Try-78] for a small node or nextjs setup where u need total control over bot filtering a cheap VPS on hetzner or digitalocean running docker with coolify or dokku is 100% the way to go.