```
Should be written in Ebbot as:
Copy
```
.elementor-container.elementor-column
```
> Even though it's an `id`, Ebbot uses a dot (`.`) prefix here because of its custom selector rules. Just remember: replace spaces with dots, and always start with a `.` or `#` based on the rule.
## Find out why your web scraping is failing
If your web scraping efforts result in 0 documents retrieved, the issue most frequently stems from one the following reasons:
* **Scrape URL settings**: "Only scrape pages starting with URL" setting correctly configured?
* **Anti-Scraping Measures:** Your website may have active security tools or detection systems designed to identify and block automated scraping attempts. If your website use such tools make sure that ebbot's scraper's IP is whitelisted.
* **robots.txt Directives:** Your website's robots.txt file might contain rules that explicitly disallow web scrapers from accessing content on your site. This file serves as a guide for benevolent web crawlers, indicating which parts of the site they should not visit.
* **Rendering**: Does the site require a delay or a specific selector for JS?
{% tabs %}
{% tab title="URL settings" %}
### URL settings
To prevent the scraper from wandering into irrelevant parts of the site:
* Check Setting: "Only scrape pages starting with URL"
* Best Practice: If you only need the support section, use
**The Risk**: If this is too restrictive, you might miss pages; if it's too broad, you might clutter the data set with a lot of useless data.
{% endtab %}
{% tab title="Whitelist web scrape IP" %}
### Whitelist web scrape IP adress
Current web scrape IP-adress: **35.246.173.254**\
New web scrape IP-adress: **141.94.175.178**
The current IP address will be replaced with the new one during 2026. To avoid future issues, ensure both IP addresses are whitelisted now, so you don’t need to update the whitelist later.
{% endtab %}
{% tab title="Add ebbot to robots.txt" %}
### Add ebbot to your robots.txt file
To check if your website has robots.txt rules, simply go to your website's URL and append `/robots.txt` (e.g.,
).
Within the robots.txt file, you'll see directives starting with User-agent:. A User-agent identifies a specific web crawler (like a search engine bot or a web scraper).
* If you see User-agent: \* followed by Allow: /, it means all web scrapers (and other bots) are generally allowed to access the entire site.
```
User-agent: *
Allow: /
```
* Conversely, if it says User-agent: \* followed by Disallow: /, it indicates that all web scrapers (and other bots) are disallowed from crawling the entire site.
```
User-agent: *
Disallow: /
```
#### How to add ebbot to robots.txt
If you want to allow only Ebbot's scraper while still disallowing others, you can add the following specific rule:
```
User-agent: Ebbot-Scraper
Allow: /
```
{% endtab %}
{% tab title="Rending" %}
### JavaScript & SPA Rendering
If the scraper returns an empty result or a "loading" icon, the site is likely a Single Page Application (SPA). The scraper might be grabbing the HTML before the content has loaded.
* The Fix: Set a specific Query Selector to tell the scraper to wait for the content.
**Recommended**: Use main\[role="main"] or a specific container ID like #main-content.
{% endtab %}
{% endtabs %}
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.ebbot.ai/ebbot-docs/core-capabilities/ebbotgpt/ebbotgpt-knowledge/source-types/scrape-site.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
