A lightweight metadata bot that fetches link previews for Argon Chat. Respectful, transparent, and fully controllable.
When a user shares a link in an Argon Chat conversation, our service fetches a small amount of metadata from that page — title, description, and preview image — to render a rich link preview directly in the chat.
This is the same kind of behavior you'd see from Slack, Discord, Telegram, or Twitter when they "unfurl" a URL. ArgonCrawler exists solely to provide this functionality; it does not index content, follow links recursively, or store page contents beyond the metadata needed for a thumbnail card.
Every crawl is user-initiated — our bot never discovers or visits pages on its own. It only fetches a URL because a real person pasted it into a conversation.
ArgonCrawler/1.5 (+https://argon.gl/beaf; like TwitterBot)
You can identify our bot by this User-Agent string in your access logs.
og:title, og:descriptionog:image (stored for thumbnail)og:site_name, og:typeThat's it. No full-page content, and no JavaScript on the vast majority of fetches.
Each URL is fetched once and cached for up to 6 months. Repeat requests are served from that cache; after a week we may send one conditional revalidation request, which your server can answer with a 304. We never batch-crawl, spider, or recursively follow links on your page.
Our bot imposes a strict 3-second timeout on page fetches. If your server doesn't respond within that window, we give up. We read at most 5 MB of HTML and follow a maximum of 3 redirects. The OG image is capped at 2 MB.
ArgonCrawler fully respects your robots.txt directives. If your robots.txt disallows our User-Agent (or all bots) from accessing a path, we will not fetch that URL and the link preview will simply not be generated.
User-agent: ArgonCrawler
Disallow: /Crawl-delay — we honor per-domain rate limitsDisallow rules for * (wildcard user-agent)ArgonCrawler or TwitterBotNote: If a URL is blocked by robots.txt, Argon Chat users will see a plain text link instead of a rich preview. No data from your page is fetched or stored.
Only the extracted Open Graph metadata (title, description, image URL) and a cached copy of the OG image. We do not store the full HTML content of your page. A small allow-list of hosts that cannot be read statically is rendered in a headless browser; everything else is parsed as plain HTML.
Metadata is cached for approximately 6 months by default. Frequently accessed URLs may have their cache extended. Cached data (including stored images) is permanently deleted when the cache expires or is manually invalidated.
If the page specifies an og:image, we download and store a copy on our CDN to ensure the preview remains available even if the original image URL changes or goes offline. When the cache entry is invalidated or expires, the stored image is deleted.
ArgonCrawler does not set cookies, does not fingerprint users or devices, and does not participate in ad networks. A typical crawl is one GET for your robots.txt and one for the HTML, plus an optional oEmbed lookup and OG image download — and that's the end of the interaction.
If you'd like to request removal of cached data for your domain, have questions about our crawler's behavior, or need to report an issue, feel free to reach out.