Web Scrapers Maxing Out My Vercel Fast Data Transfer

The Context

I use Vercel to deploy my website. For small websites, like mine, it’s low cost, reliable, and hassle free.

The Problem

I received an email from Vercel notifying me that I had used over 80% of my Fast Data Transfer allowance on the free tier. This represents outgoing data transfer from Vercel’s CDN to readers of the website.

I checked my analytics dashboard and this represented 100k requests–y site isn’t receiving that much traffic.

After some digging, I found it was Google and OpenAI indexing my site (presumably to train their LLMs on).

The Solution

I tried adding a Crawl-Delay header of 30s to my robots.txt and it seems that the number of incoming requests has dropped significantly.

I’m not sure if this was correlated to the change I made, but I will keep an eye on it going forwards - so far it looks good!

The Code


// robots.ts
import { MetadataRoute } from "next";
import { DOMAIN } from "./constants";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      crawlDelay: 30, // tell the robots to leave me alone!
    },
    sitemap: `${DOMAIN}/sitemap.xml`,
  };
}