Dragonfly Crawler

Dragonfly is a web crawler operated by Savantly. It visits websites on behalf of our customers to convert existing sites into modern Next.js applications.

User Agent

All requests from the Dragonfly crawler use the following user agent string:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 SavantlyDragonfly/1.0 (+https://dragonfly.savantly.cloud/bot)

Identifying Dragonfly

You can identify Dragonfly requests by matching on the SavantlyDragonfly token in the User-Agent header.

Firewall Configuration

If your firewall or WAF blocks automated traffic, you can allowlist the Dragonfly crawler by adding a rule that permits requests where the User-Agent header contains SavantlyDragonfly.

Cloudflare WAF (skip rule)

http.user_agent contains "SavantlyDragonfly"

nginx

if ($http_user_agent ~* "SavantlyDragonfly") {
  # allow
}

Apache .htaccess

SetEnvIfNoCase User-Agent "SavantlyDragonfly" allowed_bot
Require env allowed_bot

Crawl Behavior

Dragonfly only crawls sites when initiated by a customer.
Crawls are limited in scope to the pages the customer selects.
Dragonfly respects robots.txt directives.
Requests are made from containerized environments and may originate from various cloud provider IP ranges.

Contact

If you have questions or concerns about Dragonfly crawling your site, please contact us at support@savantly.net.