Popular topics: How to Disable and Clear AutoFill Info in your Browser Image Size, File Size, and Image Resolution Explained

Search results for “${ term }”

A Plain-Language Guide to Bots, Scraping and AI

Who's Visiting Your Website? 
A Plain-Language Guide to Bots, Scraping and AI

When you picture visitors to your website, you probably imagine collectors, gallery owners, students and fans looking at your work. They are there, and they matter most. But a large share of the traffic to any website doesn't come from people at all. It comes from automated programs, often called "bots," that visit pages on their own.

Some of these bots help you get found. Others are there to copy, probe or misuse what you've built. And over the last few years, artificial intelligence has added a new and, for many artists, unsettling kind of visitor.

This guide explains what's going on in plain terms, what it means for your art, and how we at FASO handle it on your behalf.

What is scraping?

Scraping is when a program visits a website and automatically collects what's on it: text, images, prices, links, or anything else on the page.

Think of it as the difference between a person browsing a gallery and someone walking through with a camera, photographing every wall in a few seconds, and leaving. A person reads one page, then another. A scraper can read thousands of pages in minutes and save everything it finds.

You'll also hear the words "crawler," "spider" and "bot." They all describe roughly the same thing: software that moves from page to page on its own. "Scraping" usually puts the focus on collecting and keeping the content.

Every bot also introduces itself with a name tag, called a "user agent." Googlebot, for example, is Google's crawler. Honest bots wear their real name tag. Dishonest ones often borrow the name tag of an ordinary web browser so they look like a regular visitor.

What scraping is used for

Scraping is a tool, and like most tools it can be used well or badly. Here are the most common reasons bots visit websites:

  • Search engines. Google, Bing and others send crawlers to read your pages so they can show your site when someone searches for, say, "contemporary landscape painter in Vermont." This is how most new visitors find you.
  • Link previews. When someone shares your website on social media or in a text message, a bot quickly visits the page to fetch the title, description and a thumbnail image for the preview.
  • Security checks. Some services scan websites for malware or known weaknesses. Done by trusted companies, this helps keep the web safer.
  • Attacks. Criminals use bots to look for weak passwords, outdated software or other ways to break in, or to flood a site with traffic until it slows down.
  • Data harvesting. Some bots collect email addresses for spam, or gather personal details to sell.
  • Copying content and prices. Some bots copy text, images or prices wholesale, sometimes to republish them on other sites or to sell knockoff prints of an artist's work.
  • AI training. Some companies collect huge amounts of text and images from the web to teach their AI systems. We'll come back to this.

When you want bots to visit

Some bots are working for you, even if you never invited them.

Search engine crawlers are the clearest example. If Google can't read your website, your website won't show up in Google. The same goes for Bing, Apple's search features and others. For most artists, search is one of the main ways new collectors find them.

Link-preview bots are helpful too. When a fan shares your newest painting on Facebook or in a message to a friend, you want it to appear with a nice image and your site's name, not as a bare, unclickable-looking link.

And legitimate security scanners help spot problems before they become emergencies.

When you don't

Other bots take far more than they give. They can:

Slow down your website for real visitors by sending huge numbers of requests.
Try to break into your site or account.
Copy your images and descriptions to sell counterfeit prints or build fake shops.
Harvest contact details for spam.
Collect your artwork to train AI systems without your permission.

These bots rarely announce themselves honestly. Many go to great lengths to look like a real person using a normal web browser, which is part of what makes them hard to stop.

How AI is changing things for artists

The worry: your artwork and AI training

For many artists, this is the part that matters most, and we understand why.

Image-generating AI tools learn by studying enormous collections of pictures, many of them gathered from the web. Artists have found their own work, and even their names, being used to produce images "in the style of" their art, without having been asked, credited or paid. That can feel like a violation of something deeply personal, and it raises real questions about copyright and about how artists make a living.

These questions are being worked out in public right now. In January 2023, a group of artists including Sarah Andersen, Kelly McKernan and Karla Ortiz filed a class-action lawsuit in the United States against Stability AI, Midjourney and DeviantArt over the use of their work to train image generators. Some of their copyright claims were allowed to move forward, and the case is still making its way through the courts with no final decision yet. Until cases like this one are decided, the law here remains unsettled.

The possible upside: being found and being known

AI isn't only a threat, and it would be unfair to describe it that way.

People increasingly ask AI assistants like ChatGPT, Claude, Perplexity or Google's AI features questions such as "Who paints large abstract seascapes in New England?" or "Where can I buy an original botanical watercolor?" Those tools can only recommend you if they can read your website. Just as being invisible to Google means fewer visitors, being invisible to AI search may mean being left out of these new conversations.

There's also a longer-term argument some people make. AI models learn about the world from what they read. If your biography, artist statement and exhibition history are part of what future models learn from, those models may "know" who you are and mention you more readily. Note that this is about your written words, not your images. Some artists are comfortable sharing their text this way while keeping their artwork off-limits.

Other artists find AI useful for everyday chores, like drafting a newsletter, describing a piece for a listing or organizing their inventory. And some find AI images a source of inspiration, though plenty of others feel strongly otherwise. Both reactions are understandable.

There's no single right answer

Things are moving fast. The companies, the tools, the laws and the norms are all changing month to month. There isn't one correct choice for every artist. There's only the choice you feel comfortable with, and that choice may change as the landscape does. Our job is to make sensible decisions on your behalf in the meantime, and to give you more say over time.

How this shapes our policy at FASO

Here is how we've approached automated traffic until now:

By default, we block all identifiable automated traffic from accessing artists' websites and their content, unless it has been vetted and approved by our team. This helps prevent a significant amount of unwanted traffic, which can account for up to 70% of total server traffic. Our goal is to protect artists' websites without impacting legitimate visitors.

Identifying automated traffic is challenging. Bad actors constantly evolve their techniques to disguise automated traffic as regular user traffic. Additionally, the rise of AI models has led to more companies scraping websites for data. Depending on their sophistication and ethics, these scrapers may or may not be detectable. Stricter rules can prevent unwanted traffic but may also cause false positives, blocking legitimate users. Our engineering team continually adjusts our blocking mechanisms to maintain this balance.

That approach isn't changing. What we want to make clearer is how AI fits into it.

Useful bots are allowed by default. Search engine crawlers like Googlebot and Bingbot, along with other well-behaved bots we've vetted, can visit your site so people can find you.

AI search is treated like search. Most major AI companies now run their own search features, and many of them use separate bots for search and for training. That separation lets us say yes to one and no to the other. For example:

  • OpenAI uses OAI-SearchBot to show websites in ChatGPT's search results, and a different bot, GPTBot, to collect content that may be used to train its AI models.
  • Anthropic uses Claude-SearchBot for search in Claude and ClaudeBot for collecting content that may be used for training.
  • Perplexity says its PerplexityBot is for surfacing and linking websites in its search results and is not used to train AI models.
  • Google offers a setting called Google-Extended, separate from its regular Googlebot, that lets sites say whether their content may be used to train Google's Gemini models. Google says this setting doesn't affect a site's place in Google Search.
  • Apple offers Applebot-Extended, separate from its regular Applebot, so websites can stay in Apple's search features while opting out of training Apple's AI models.

We'll allow the search bots in that list — not the ones used for training — so you can show up when people ask AI tools for recommendations.

AI training bots won't be allowed by default, at least for now. Bots that exist to gather content for training AI models will generally stay blocked while the questions above get sorted out.

User-requested visits are allowed with limits. There's another kind of bot that doesn't crawl your site on its own. It shows up only when a person asks an AI tool a question and that tool decides to read a page so it can answer more accurately, summarize what it found, or point the person back to your site. OpenAI's ChatGPT-User, Anthropic's Claude-User, Perplexity's Perplexity-User and Meta's Meta-ExternalFetcher work this way. Because these visits depend on a user's intent, some can be beneficial, like helping someone discover your work or get a clear answer about a painting, while others can be harmful. Some of these tools may also fetch a page even when a site's usual robots.txt signals would tell a regular crawler to stay away. We'll allow them with limits, and we're exploring additional features for these kinds of use cases.

There's nuance. Not every company separates its bots this neatly, some bots do more than one job, and some scrapers simply don't identify themselves honestly. Blocking a bot also only works if we can recognize it. So no host, including us, can promise to catch every scraper. What we can promise is to keep watching closely and adjusting.

More control is coming. We're working on additional protections, and on ways for you to choose your own preferences, for example if you'd like AI tools to learn from your written content, your images, or both. Written content and images could be treated separately, and we're exploring options that give you that control. We'll share details when they're ready.

About Meta's new business terms

You may have seen worried posts about Meta, the company behind Facebook and Instagram. Here's what happened and why we don't think you need to panic.

In October 2026, Meta published updated Commercial Terms for anyone who uses Facebook, Instagram and its other apps for business purposes, which includes managing a business Page. A new paragraph gave Meta permission to use bots to crawl websites connected to those business accounts. The first version said Meta could use that content to, among other things, "train" its products. Artists, news publishers and others objected strongly.

Meta responded within days by removing the word "train" and narrowing the purpose to its advertising and business products and to keeping its products secure. It then pushed the start date back from October 30 to November 20, 2026, saying it wanted more time to explain the changes, including that website owners can still use their site's robots.txt file (a simple file that tells bots where they may and may not go) to signal their preferences. Some critics, including legal experts and publisher groups, still think the wording is too broad, and regulators in the UK and Europe have been asked to look at it. So this story isn't fully over.

Here's the updated wording from Meta's Commercial Terms:

"You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs (and any content available through them) that are associated with your products or services, or that are made available through your or users' use of Meta Company Products and services, and to periodically pull updates from such domains, web pages, or URLs in order to provide, develop, improve, research, and maintain advertising and business products (...) and maintain the integrity and security of Meta Company Products."

Read with everything above in mind, the clause mostly describes the ordinary, helpful kind of scraping:

  • Link previews. When someone shares your website on Facebook or Instagram, Meta needs to fetch the title, description and image to build the preview.
  • Ads. If you advertise, Meta checks the page your ad points to so it can display the ad properly and make sure it follows the rules.
  • Product information. If you list artwork through Meta's shopping features, it needs to read details like titles and prices from your site and keep them current.
  • Integrity and security. Meta checks shared links for malware and scams to protect its users.

Meta also uses different bots for different jobs, and lists them publicly. For example, its link-preview bot is separate from Meta-ExternalAgent, which Meta says crawls for uses such as training AI models, and from Meta-WebIndexer, which supports search in Meta AI. Because training is handled by a separate, identifiable bot, we let the helpful ones through, including Meta's AI search bot, its link-preview bot and its ads bot, and we keep the training bot, Meta-ExternalAgent, blocked by default, in line with the policy above.

So, in theory and for now, artists hosted with FASO don't need to worry that Meta is training its AI on their websites through these terms. We'll keep an eye on how the terms evolve and how Meta's bots actually behave, and we'll let you know if anything changes.

The bottom line

Bots are a normal part of the web. Some help people find you, some cause harm, and AI has made the picture more complicated. Our approach is simple: let in the visitors who help you, keep out the ones who don't, and give you more control over the gray areas as the tools to do that mature.

If you have questions about any of this, or strong feelings about how AI should be able to use your work, we'd genuinely like to hear from you. Your input helps shape what we build next.

11162026

 

Was this article helpful?

Can’t find what you’re looking for?

Our world-class customer care team is here for you

Contact Support