A friend of mine recently emailed me after he spent 11 hours creating page titles and meta descriptions for a Magento site with 139 pages. He said, and I quote, “it was brutal.”
And that’s for a site with just 139 pages, imagine working a medium-sized e-commerce site with thousands of pages! You might not have to imagine it, because if you’re like a lot of the SEOs we talk to, you’ve done it before.
Writing metadata is tedious, and time consuming. It’s not the most important part of optimizing a website, but it’s still best practice and it needs to get done.
That’s the whole premise of MetaMonster, the SEO spider tool we’re building that fixes issues for you.
But it is possible to do using a lot of the tools you’re already familiar with.
If you have a Screaming Frog license, an OpenAI (or Claude, or Gemini) API key, and you aren’t afraid to write a tiny bit of JavaScript, then you can use Screaming Frog to generate page titles, meta descriptions, and more for you using generative AI. In this post, we’ll break down exactly how to do that (including a prompt we’ve tested) and where the limitations still are with this kind of approach.
Watch the video on YouTube (or keep reading)
Enable JavaScript rendering
The way that you integrate Screaming Frog with generative AI tools (or any external API) is through Screaming Frog’s new Custom Javascript feature, which will run a little JS snippet on every page.
Doing it this way makes it super flexible, any thing you can do with a simple bit of JavaScript code is possible, although there are still some limitations that we’ll get into later.
To start, though, you’ll need to enable JavaScript rendering. Go to Configuration > Spider > Rendering to find the menu
Then select JavaScript from the rendering dropdown and click OK.
Adjust crawl settings to avoid hitting rate limits
Before we start setting up our custom JavaScript to make calls to the OpenAI API there are a few more settings I would recommend enabling to avoid hitting OpenAI’s rate limits.
OpenAI scales your rate limits based on how much you’ve spent on API costs. So if you’re already a heavy API user this may not be an issue for you, but for most folks these settings will make your experience a lot smoother.
First, navigate to Configuration > Spider > Crawl and uncheck all of the resource links. You aren’t going to write meta descriptions for the JavaScript or CSS files on your site, and they’re going to clutter up your exports and lead to unnecessary API calls.
Next, navigate to Configuration > Speed, enable the option to Limit URL/s and set Max URL/s to 0.5 or lower. This will cause your crawl to move pretty slowly, you’ll now be crawling one URL every 2 seconds instead of the default of one every tenth of a second. But if you don’t have heavy usage on your OpenAI API, this should keep you from hitting their rate limits, so you won’t have to rerun your crawl multiple times.
Add a custom JavaScript snippet
Now it’s time to get into the code! Don’t worry, it’s very minimal. From the Crawl config window select Custom > Custom JavaScript or from the top bar select Configuration > Custom > Custom JavaScript.
Click “+Add from Library” and select the “(ChatGPT) Template.”
Click the button below to edit the JavaScript snippet.
The code you’ll see in the editor is pretty straightforward. At the top you have 3 variables:
- Your OpenAI API key
- Your prompt
- The content from the page you’re crawling
You’ll only need to edit the first two in order to get basic generation working. Everything below that is the code to actually call the API and handle the response. Everything in green with two slashes (//) is a comment. That code won’t run, it’s just there to provide context to the programmer.
How to get an OpenAI API key
Before you can use the custom JavaScript you’ll need an OpenAI API key. To get your API key you’ll need to create a platform account, which is separate from your ChatGPT account. You’ll also be prompted to create a project, call it “Metadata generation.”
After you’ve created your account, go to Settings > API keys and press “+ Create new secret key.”
Give the key a name and select your MetaData generation project.
For security reasons, this key will be hidden after you close the modal, so copy it directly into Screaming Frog or into a note on your computer temporarily.
Adding your prompt
Prompts are your instructions to the AI. To add your prompt, replace the question with your desired instructions. Here is a prompt similar to what we use in MetaMonster:
const question = "Identify the primary keyword from the following web page content and generate an SEO optimized meta description using that keyword in the style of a SEO expert. IMPORTANT: Do NOT start the description with the words explore, exmbark, experience, join, dive into, or discover.\n"+
"# Steps\n"+
"1. Analyze Content: Examine the provided content to determine the primary theme or subject matter.\n"+
"2. Identify Keyword: Select the primary keyword that best represents the main subject of the web page.\n"+
"3. Craft Meta Description: Write a meta description between 120-155 characters that incorporates the keyword, is engaging, and succinctly summarizes the page content using the language in the text.\n"+
"# Output Format\n"+
"**Primary keyword:** The keyword that best represents the main subject of the web page\n"+
"**Meta Description:** Meta description between 120-155 characters\n"+
"# Examples\n"+
"**Example 1:**\n"+
"- Primary Keyword: Keyword Research Tool\n"+
"- Meta Description: The most complete keyword research tool on the market. Generate thousands of keyword ideas, group them into topical clusters, and nail search intent.\n"+
"**Example 2:**\n"+
"- Primary Keyword: Authority checker\n"+
"- Meta Description: Free tool to check the authority of any website based on the quality and quantity of its external backlinks.\n"+
"**Example 3:**\n"+
"- Primary Keyword: Screaming Frog alternatives\n"+
"- Meta Description: Looking for a Screaming Frog alternative? MetaMonster could be the tool for you.\n"+
"**Example 4:**\n"+
"- Primary Keyword: SEO spider software\n"+
"- Meta Description: Clean up your metadata in minutes instead of months with MetaMonster, the SEO spider that fixes issues for you.\n"+
"**Example 5:**\n"+
"- Primary Keyword: Page title generator\n"+
"- Meta Description: Generate page titles for your website with AI. MetaMonster is the SEO spider software that fixes issues for you.\n"+
"**Example 6:**\n"+
"- Primary Keyword: Wordpress SEO plugin\n"+
"- Meta Description: Easily publish optimizations to your site with the MetaMonster Wordpress SEO Plugin.\n"This prompt follows the 5 principles of good prompt engineering from the book Prompt Engineering for Generative AI:
- Give direction
- Specify format
- Provide examples
- Evaluate quality
- Divide labor
The prompt string is broken up into individual lines using the new line character and the plus sign to make the prompt easier to read and edit. If you add additional lines make sure to follow the same format. Once you copy and paste this prompt into the JavaScript editor it should look something like this:
If you click okay and run a new crawl, the code should work as-is. But there are two additional (optional) improvements we can make to improve the quality and make it easier to parse through the results.
Adjusting the temperature
The temperature of the prompt is a more advanced parameter that tells the AI how much randomness to use in formulating it’s response. Lower values will cause the AI to stick closer to the most common response, while higher values will cause the AI to be more “creative.”
By default, Screaming Frog uses a temperature value of 0.7 in its call to the OpenAI API. By increasing this to 1.0 we should get more variation in our responses, making them a little less repetitive. The examples included in our prompt also help with this.
Extracting the meta description from the AI’s response
Last, we want to add a little bit of code to extract just the meta description from the AI’s response. In our prompt, we instruct the AI to first identify the primary keyword and to include that in the response. Doing so is a simple version of “thinking out loud” which helps to improve the quality of the response. But we likely just want the description.
Add the following code above the function chatGptRequest:
function extractMetaDescription(content) {
const metaDescriptionRegex = /\*\*Meta Description:\*\*\s*(.+)/;
const match = content.match(metaDescriptionRegex);
return match ? match[1].trim() : null;
}Then replace the line:
return data.choices[0].message.content.trim();With:
return extractMetaDescription(data.choices[0].message.content.trim());This simple function looks for the string Meta Description: in the response from the AI and pulls out whatever text comes after. The end result should look like this:
Now when you run a crawl and navigate to the Custom JavaScript tab you’ll get a custom Meta Description generated by GPT 4o for every page on your site!
Limits of generating meta descriptions inside of Screaming Frog
You can take this same template, and easily modify it to generate page titles, open graph tags, and more. You can also use one of Screaming Frog’s included template to generate image alt text or do sentiment analysis. There’s lots of cool things that can be done leveraging this new custom JavaScript feature.
That said, there are some limitations.
First, each call to the API only has the context of the current page. There’s no way to keep track of the other metadata that has already been generated, or to feed the AI context from other pages. This can make the results more repetitive, especially for larger sites.
Second, the editor is designed for simple scripts, not complex logic. So it can be cumbersome to chain together multiple prompts, which is where you can find some of the biggest gains in prompt engineering. This includes doing things like generating a meta description, and then feeding the generated meta description back to the AI and asking it to critique and edit it.
Try MetaMonster to find, generate, and publish metadata all in one place
If you want a tool that handles prompt engineering for you, built from the ground up around AI generation, then you should try MetaMonster.
MetaMonster is the SEO spider that fixes issues for you. There’s no need to write custom JavaScript, or become a prompt engineering expert. And MetaMonster also has a Wordpress plugin to make it easy to publish the results to your CMS, with more CMS integrations coming soon.
Just crawl your site, click generate (or re-generate if you don’t like the first result you get back), then hit publish.
MetaMonster is currently in closed alpha, so join our waiting list below to be one fo the first to get access when we launch!