Skip to content

PALACE Documentation

PALACE (Platform for Automated LLMs Agentic Capabilities Evaluation) is an open benchmark format for LLM evaluation, with native support for agentic tasks. palace-eval is the reference implementation.

  • Getting Started

    Install PALACE and create your first benchmark in minutes.

    Get started

  • Task Types

    Learn about the five task types: QA, Classification, Criteria Evaluation, Instruction Following, and Agentic.

    Explore task types

  • How-To Guides

    Step-by-step guides for common tasks.

    Browse guides

  • Reference

    Complete specifications and API documentation.

    View reference

What is PALACE?

PALACE evaluates LLM agents across configurable benchmarks. It supports:

  • QA tasks — Verify factual correctness with LLM judges
  • Classification tasks — Exact-match categorical outputs
  • Criteria Evaluation tasks — Pairwise comparison of long-form content
  • Instruction Following tasks — Verify structured output constraints
  • Agentic tasks — Tool-using agents in sandboxed environments

Five Task Types

Task Type Use Case Verification
QA Factual questions, open-ended answers LLM judge with configurable criteria
Classification Categorical outputs, labels Exact match against expected labels
Criteria Evaluation Long-form documents, reports Pairwise comparison with weighted criteria
Instruction Following Structured output formats Constraint verification (length, format, keywords)
Agentic Tool-using agents, coding, web tasks External verifier in sandboxed environment

Getting Help

  • Glossary - Definitions of key terms
  • Examples - Real-world benchmark patterns
  • Concepts - How PALACE works under the hood