Screaming Frog SEO Spider is a software to crawl websites’ data and analyze sites’ SEO issues, and it offers Windows, macOS, and Linux operating versions. Screaming Frog SEO Spider offers a free trial version that allows you to crawl 500 URLs. The article shows how to use Screaming Frog SEO Spider with YiLu Proxy on Windows PC.

Download Screaming Frog SEO Spider tool at https://www.screamingfrog.co.uk/seo-spider.

1. Click Configuration - System - Proxy.

2. Set up proxy.

YiLu Proxy dynamic IP is suitable for web scraping, so I directly extract YiLu Proxy dynamic IP information here.

A.Select rotating residential IP or datacenter IP

B.Select “Change IP per request”

C.Click “Refresh(HTTP/Socks5)” button

D.Right click an IP and click “copy IP and port” or directly copy IP address and port.

Select “Use Proxy Server”.

Enter proxy IP address, and port in Screaming Frog SEO Spider proxy configuration.

Click “OK and Restart” to make the changes take effect.

3. Enter a URL to crawl.

Click “Start”.

Then a pop-up window will appear.

4. Enter proxy IP username and password to verify.

5. Start crawling.

You can see the crawling process on the top right, and click “Pause” at any time.

Here are some web pages that have been crawled.

6. Export data.

Here I export web pages’ titles.

Click Pages Title - Export.

Then enter a name, select a file type, and the save directory(desktop).

Click “Save”.

7. Open the file.

Then you can see the file appears on the computer desktop.

Double-click and open it.

Here are these crawled web pages’ titles.

The article is from