> ## Documentation Index
> Fetch the complete documentation index at: https://docs.counso.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data sources

A data source is a managed collection of documents that Counso indexes for semantic search. Create one in the Counso interface by providing a short name and an optional description. The name can contain lowercase letters, numbers, and hyphens; a new data source starts empty. You can edit its description or delete it in Settings, but you cannot rename it.

Add documents through the interface or API. During ingestion, Counso removes repeated whitespace, splits the text into chunks using the configured `max_chunk_size`, embeds the chunks, and indexes their vectors with the document metadata and original text. The default embedding model is `text-embedding-3-large`; a different model can be used where the workspace supports it.

Use a stable `document_id` when inserting or updating a document. With `baseUrl` set to `https://app.counso.ai`, an upsert uses `POST {baseUrl}/api/v1/w/{wId}/spaces/{spaceId}/data_sources/{dsId}/documents/{documentId}` with a JSON body containing `text`. A new ID creates a document; an existing ID replaces its previous version and indexed chunks. Searching uses `GET {baseUrl}/api/v1/w/{wId}/spaces/{spaceId}/data_sources/{dsId}/search`. Deleting a document removes its associated chunks; deleting the data source removes all documents and chunks in it. Confirm the target before either delete operation.

For a directory import, a script can read PDF, TXT, and Markdown files and upsert each document. This example uses Python, `requests`, and `pdftotext`; provide workspace, Space, data source, and credential values through environment variables. It derives each document ID from the file’s relative path, so files in different folders keep separate identities.

```python theme={"system"}
import os
import pathlib
import sys
from urllib.parse import quote
import requests
import pdftotext

base_url = os.getenv("COUNSO_BASE_URL", "https://app.counso.ai")
workspace_id = os.environ["COUNSO_WORKSPACE_ID"]
space_id = os.environ["COUNSO_SPACE_ID"]
data_source_id = os.environ["COUNSO_DATA_SOURCE_ID"]
api_key = os.environ["COUNSO_API_KEY"]

def upload(text, file):
    document_id = file.relative_to(directory).as_posix()
    url = (f"{base_url}/api/v1/w/{workspace_id}/spaces/{space_id}"
           f"/data_sources/{data_source_id}/documents/{quote(document_id, safe='')}")
    return requests.post(
        url,
        headers={"Authorization": f"Bearer {api_key}"},
        json={"text": text},
    )

directory = pathlib.Path(sys.argv[1])
for file in directory.rglob("*"):
    if not file.is_file():
        continue
    if file.suffix.lower() == ".pdf":
        with file.open("rb") as source:
            text = "\n\n".join(pdftotext.PDF(source))
    elif file.suffix.lower() in {".txt", ".md"}:
        text = file.read_text()
    else:
        continue
    response = upload(text, file)
    if response.status_code != 200:
        print("Import failed:", file, response.status_code, response.text)
    else:
        print("Imported:", file)
```

Save the script as `upload.py`, install `requests` and `pdftotext`, set the environment variables above, then run `python upload.py <directory-to-import>`.

Check the response for each file and review the indexed content in Counso. Use `https://app.counso.ai` as the default `baseUrl`, or substitute your environment’s base URL.

When an Agent searches a data source, Counso embeds the query and retrieves relevant chunks. Results are grouped by original document, so an Agent can use relevant passages without loading every full document into context. This retrieval pattern is called retrieval-augmented generation (RAG). For chunking and result ordering, see [Chunks and documents](/docs/developer-platform/core-concepts/chunks-and-documents).