> ## Documentation Index
> Fetch the complete documentation index at: https://docs.neum.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Connector

> Scrape the contents of a website

Basic website scraping. Works with most websites, but might fail with javascript heavy ones. This connector takes a given URL for a website and extracts the available text from it. Once extracted, the contents are outputted as an HTML.

## Properties

Required properties:

* `url`: URL for website

Available metadata

* `url`: URL for website

Compatible loaders:

* AutoLoader
* HTMLLoader

## Usage

<CodeGroup>
  ```python Local Development
  from neumai.DataConnectors import WebsiteConnector
  from neumai.Shared import Selector

  website_connector =  WebsiteConnector(
      url = "https://www.neum.ai/post/retrieval-augmented-generation-at-scale"
      selector = Selector(
          to_metadata=['url']
      )
  )
  ```

  ```json Cloud
  {
      "sources":[
          {
              "data_connector": {
                  "connector_name":"WebsiteConnector",
                  "connector_information":{
                      "url":"https://www.neum.ai/post/retrieval-augmented-generation-at-scale",
                      "selector": {
                          "to_metadata": [
                              "url"
                          ]
                      }
                  }
              }
          }
      ]
  }
  ```
</CodeGroup>