Human Agent

Overview

The Inspect human agent enables human baselining of agentic tasks that run in a Linux environment. Human agents are just a special type of solver that use the identical dataset, sandbox, and scorer configuration that models use when completing tasks. However, rather than entering an agent loop, the human_agent solver provides the human baseliner with:

A description of the task to be completed (input/prompt from the sample).
Means to login to the container provisioned for the sample (including creating a remote VS Code session).
CLI commands for use within the container to view instructions, submit answers, pause work, etc.

Human baselining terminal sessions are recorded by default so that you can later view which actions the user took to complete the task.

Example

Here, we run a human baseline on an Intercode CTF sample by using the --solver option to use human_agent rather than the task’s default solver:

inspect eval inspect_evals/gdm_intercode_ctf \
    --sample-id 44 --solver human_agent

The evaluation runs as normal, and a Human Agent panel appears in the task UI to orient the human baseliner to the task and provide instructions for accessing the container. The user clicks the VS Code Terminal link and a terminal interface to the container is provided within VS Code:

Note that while this example makes use of VS Code, it is in no way required. Baseliners can use their preferred editor and terminal environment using the docker exec command provided at the bottom. Human baselining can also be done in a “headless” fashion without the task display (see the Headless section below for details).

Once the user discovers the flag, they can submit it using the task submit command. For example:

task submit picoCTF{73bfc85c1ba7}

Usage

Using the human_agent solver is as straightforward as specifying the --solver option for any existing task. Repeating the example above:

inspect eval inspect_evals/gdm_intercode_ctf \
    --sample-id 44 --solver human_agent

Or alternatively from within Python:

from inspect_ai import eval
from inspect_ai.solver import human_agent
from inspect_evals import gdm_intercode_ctf

eval(gdm_intercode_ctf(), sample_id=44, solver=human_agent())

There are however some requirements that should be met by your task before using it with the human agent:

It should be solvable by using the tools available in a Linux environment (plus potentially access to the web, which the baseliner can do using an external web browser).
The dataset input must fully specify the instructions for the task. This is a requirement that many existing tasks may not meet due to doing prompt engineering within their default solver. For example, the Intercode CTF eval had to be modified in this fashion to make it compatible with human agent.

Container Access

The human agent works on the task within the default sandbox container for the task. Access to the container can be initiated using the command printed at the bottom of the Human Agent panel. For example:

docker exec -it inspect-gdm_intercod-itmzq4e-default-1 bash -l

Alternatively, if the human agent is working within VS Code then two links are provided to access the container within VS Code:

VS Code Window opens a new VS Code window logged in to the container. The human agent can than create terminals, browse the file system, etc. using the VS Code interface.
VS Code Terminal opens a new terminal in the main editor area of VS Code (so that it is afforded more space than the default terminal in the panel.

Task Commands

The Human agent solver installs agent task tools in the default sandbox and presents the user with both task instructions and documentation for the various tools (e.g. task submit, task start, task stop, task instructions, etc.). By default, the following command are available:

Command	Description
`task submit`	Submit your final answer for the task.
`task note`	Record a note in the task transcript.
`task status`	Print task status (clock, scoring , etc.)
`task start`	Start the task clock (resume working)
`task stop`	Stop the task clock (pause working).
`task instructions`	Display task command and instructions.

Note that the instructions are also copied to an instructions.txt file in the container user’s working directory.

Answer Submission

When the human agent has completed the task, they submit their answer using the task submitcommand. By default, the task submit command requires that an explicit answer be given (e.g. task submit picoCTF{73bfc85c1ba7}).

However, ff your task is scored by reading from the container filesystem then no explicit answer need be provided. Indicate this by passing answer=False to the human_agent():

solver=human_agent(answer=False)

Or from the CLI, use the -S option:

--solver human_agent -S answer=false

You can also specify a regex to match the answer against for validation, for example:

solver=human_agent(answer=r"picoCTF{\w+}")

Intermediate Scoring

You can optionally make intermediate scoring available to human baseliners so that they can check potential answers as they work. Use the intermediate_scoring option (which defaults to False) to do this:

solver=human_agent(intermediate_scoring=True)

Or from the CLI, use the -S option:

--solver human_agent -S intermediate_scoring=true

With this option enabled, the human agent can check their potential score on the task for a given answer using the task score command. For example:

task score picoCTF{73bfc85c1ba7}

Recording

By default, human agent terminal sessions are recorded using the LInux script command. Recorded sessions are saved in the sample store and available for playback within the Inspect View:

You can disable session recording with the record_session option:

--solver human_agent -S record_session=false

Headless

The examples above demonstrate human baselining being initiated from the CLI and standard task display. You might alternatively want to provision human baselining sessions in a server environment and then separately provide login instructions to users. To suppress the standard task display in favour of just printing out the container login information, pass the --display=plain CLI option:

inspect eval inspect_evals/gdm_intercode_ctf \
    --sample-id 44 --solver human_agent --display plain

Which will print the following to the terminal:

Running task gdm_intercode_ctf (1 sample): anthropic/claude-3-5-sonnet-20240620 (config: sample_id: 44)...

You are completing a task on a Linux system (task instructions will be presented when you login). Login to the system with the following command:

docker exec -it inspect-gdm_intercod-iebwzkg-default-1 bash -l