Introducing LLM Extraction! Convert any web page into structured data using LLMs

Slack Logo

New message in: #coach-gtm

@CoachGTM: Your meeting prep for Pied Piper < > WindFlow Dynamics is ready! Meeting starts in 30 minutes
🦜🔗 Check out our LangChain integration

Turn websites into
LLM-ready data

Crawl and convert any website into clean markdown or structured data

No credit card required
A product by
Mendable LogoMendable
Mendable Website Image

Crawl, Capture, Clean

We crawl all accessible subpages and give you clean markdown for each. No sitemap required.


  [
    {
      "url": "https://www.mendable.ai/",
      "markdown": "## Welcome to Mendable
        Mendable empowers teams with AI-driven solutions - 
        streamlining sales and support."
    },
    {
      "url": "https://www.mendable.ai/features",
      "markdown": "## Features
        Discover how Mendable's cutting-edge features can 
        transform your business operations."
    },
    {
      "url": "https://www.mendable.ai/pricing",
      "markdown": "## Pricing Plans
        Choose the perfect plan that fits your business needs."
    },
    {
      "url": "https://www.mendable.ai/about",
      "markdown": "## About Us
        Learn more about Mendable's mission and the 
        team behind our innovative platform."
    }
  ]

Note: The markdown has been edited for display purposes.

We handle the hard stuff

Proxies, caching, rate limits, js-blocked content and more...

Crawling

FireCrawl crawls all accessible subpages, even without a sitemap.

Dynamic content

FireCrawl gathers data even if a website uses javascript to render content.

To Markdown

FireCrawl returns clean, well formatted markdown - ready for use in LLM applications

Crawling Orchestration

FireCrawl orchestrates the crawling process in parallel for the fastest results.

Caching

FireCrawl caches content, so you don't have to wait for a full scrape unless new content exists.

Built for AI

Built by LLM engineers, for LLM engineers. Giving you clean data the way you want it.

Our wall of love

Don't take our word for it

Greg Kamradt
Greg Kamradt
LLM structured data via API, handling requests, cleaning, and crawling. Enjoyed the early preview.
Amit Naik
Amit Naik
#llm success with RAG relies on Retrieval. Firecrawl by @mendableai structures web content for processing. 👏
Jerry Liu
Jerry Liu
Firecrawl is awesome 🔥 Turns web pages into structured markdown for LLM apps, thanks to @mendableai.
Bardia Pourvakil
Bardia Pourvakil
These guys ship. I wanted types for their node SDK, and less than an hour later, I got them. Can't recommend them enough.
latentsauce 🧘🏽
latentsauce 🧘🏽
Firecrawl simplifies data preparation significantly, exactly what I was hoping for. Thank you for creating Firecrawl ❤️❤️❤️
Michael Ning
Michael Ning
Firecrawl is impressive, saving us 2/3 the tokens and allowing gpt3.5turbo use over gpt4. Major savings in time and money.
Alex Reibman 🖇️
Alex Reibman 🖇️
Moved our internal agent's web scraping tool from Apify to FireCrawl because it benchmarked 50x faster with AgentOps.
Michael
I really like some of the design decisions FireCrawl made, so I really want to share with others.
Paul Scott
Paul Scott
Appreciating your lean approach, Firecrawl ticks off everything on our list without the cost prohibitive overkill.

Pricing Plans

Starter

50k credits ($1.00/1k)

$50/month

  • Scrape 50,000 pages
  • 10 /scrape per min
  • 2 simultaneous /crawl jobs

Standard

500k credits ($0.75/1k)

$375/month

  • Scrape 500,000 pages
  • 15 /scrape per min
  • 4 simultaneous /crawl jobs

Scale

2.5M credits ($0.30/1k)

$1,250/month

  • Scrape 2,500,000 pages
  • 20+ /scrape per min
  • 10+ simultaneous /crawl jobs

* a /scrape refers to the scrape API endpoint.

* a /crawl refers to the crawl API endpoint.

Need more credits, higher rate limits, or more concurrency?

Scrape Credits

Scrape credits are consumed for each API request, varying by endpoint and feature.

FeaturesCredits per page
Scrape(/scrape)1
Scrape + LLM extraction (/scrape)5
Crawl(/crawl)1
Search(/search)1

What sites work?

Firecrawl is best suited for business websites, docs and help centers.

Feature 01
Slack Logo

New message in: #coach-gtm

@CoachGTM: Your meeting prep for Pied Piper < > WindFlow Dynamics is ready! Meeting starts in 30 minutes
🔥

Ready to Build?

FAQ

Frequently asked questions about FireCrawl

What is FireCrawl?

FireCrawl is an advanced web crawling and data conversion tool designed to transform any website into clean, LLM-ready markdown. Ideal for AI developers and data scientists, it automates the collection, cleaning, and formatting of web data, streamlining the preparation process for Large Language Model (LLM) applications.

How does FireCrawl handle dynamic content on websites?

Unlike traditional web scrapers, FireCrawl is equipped to handle dynamic content rendered with JavaScript. It ensures comprehensive data collection from all accessible subpages, making it a reliable tool for scraping websites that rely heavily on JS for content delivery.

Why is it not crawling all the pages?

There are a few reasons why FireCrawl may not be able to crawl all the pages of a website. Some common reasons include rate limiting, and anti-scraping mechanisms, disallowing the crawler from accessing certain pages. If you're experiencing issues with the crawler, please reach out to our support team at support@mendable.ai.

Can FireCrawl crawl websites without a sitemap?

Yes, FireCrawl can access and crawl all accessible subpages of a website, even in the absence of a sitemap. This feature enables users to gather data from a wide array of web sources with minimal setup.

What formats can FireCrawl convert web data into?

FireCrawl specializes in converting web data into clean, well-formatted markdown. This format is particularly suited for LLM applications, offering a structured yet flexible way to represent web content.

How does FireCrawl ensure the cleanliness of the data?

FireCrawl employs advanced algorithms to clean and structure the scraped data, removing unnecessary elements and formatting the content into readable markdown. This process ensures that the data is ready for use in LLM applications without further preprocessing.

Is FireCrawl suitable for large-scale data scraping projects?

Absolutely. FireCrawl offers various pricing plans, including a Scale plan that supports scraping of millions of pages. With features like caching and scheduled syncs, it's designed to efficiently handle large-scale data scraping and continuous updates, making it ideal for enterprises and large projects.

Is FireCrawl open-source?

Yes, it is. You can check out the repository on GitHub. Keep in mind that this repository is currently in its early stages of development. We are in the process of merging custom modules into this mono repository.

Does it respect robots.txt?

Yes, FireCrawl crawler respects the rules set in a website's robots.txt file. If you notice any issues with the way FireCrawl interacts with your website, you can adjust the robots.txt file to control the crawler's behavior. Firecrawl user agent name is "FireCrawlAgent". If you notice any behavior that is not expected, please let us know at support@mendable.ai.

What measures does FireCrawl take to handle web scraping challenges like rate limits and caching?

FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching. It smartly manages requests and employs caching techniques to minimize bandwidth usage and avoid triggering anti-scraping mechanisms, ensuring reliable data collection.

How can I try FireCrawl?

You can start with FireCrawl by trying our free trial, which includes 100 pages. This trial allows you to experience firsthand how FireCrawl can streamline your data collection and conversion processes. Sign up and begin transforming web content into LLM-ready data today!

Who can benefit from using FireCrawl?

FireCrawl is tailored for LLM engineers, data scientists, AI researchers, and developers looking to harness web data for training machine learning models, market research, content aggregation, and more. It simplifies the data preparation process, allowing professionals to focus on insights and model development.