Interactivity

Overview

In some cases you may wish to introduce user interaction into the implementation of solvers, scorers, or tools. For example, you may wish to:

• Confirm consequential actions like requests made to web services
• Prompt the model dynamically based on the trajectory of the evaluation
• Score model output with human judges

The input_screen() function provides a context manager that temporarily clears the task display for user input. Note that prompting the user is a synchronous operation that pauses other activity within the evaluation (pending model requests or subprocesses will continue to execute, but their results won’t be processed until the input is complete).

Input Screen

You can prompt the user for input at any point in an evaluation using the input_screen() context manager, which clears the normal task display and provides access to a Console object for presenting content and asking for user input. For example:

from inspect_ai.util import input_screen

with input_screen() as console:
    console.print("Some preamble text")
    input = console.input("Please enter your name: ")

The console object provided by the context manager is from the Rich Python library used by Inspect, and has many other capabilities beyond simple text input. Read on to learn more.

Prompts

Rich includes Prompt and Confirm classes with additional capabilities including default values, choice lists, and re-prompting. For example:

from inspect_ai.util import input_screen
from rich.prompt import Prompt

with input_screen() as console:
    name = Prompt.ask(
        "Enter your name", 
        choices=["Paul", "Jessica", "Duncan"], 
        default="Paul"
    )

The Prompt class is designed to be subclassed for more specialized inputs. The IntPrompt and FloatPrompt classes are built-in, but you can also create your own more customised prompts (the Confirm class is another example of this). See the prompt.py source code for additional details.

Progress

Evaluations with user input alternate between asking for input and displaying task progress. By default, the normal task status display is shown when a user input screen is not active.

However, if your evaluation is dominated by user input with very short model interactions in between, the task display flashing on and off might prove distracting. For these cases, you can specify the transient=False option, to indicate that the input screen should be shown at all times. For example:

with input_screen(transient=False) as console:
    console.print("Some preamble text")
    input = console.input("Please enter your name: ")

This will result in the input screen staying active throughout the evaluation. A small progress indicator will be shown whenever user input isn’t being requested so that the user knows that the evaluation is still running.

Header

You can add a header to your console input via the header parameter. For example:

with input_screen(header="Input Request") as console:
    input = console.input("Please enter your name: ")

The header option is a useful way to delineate user input requests (especially when switching between input display and the normal task display). You might also prefer to create your own heading treatments–under the hood, the header option calls console.rule() with a blue bold treatment:

console.rule(f"[blue bold]{header}[/blue bold]", style="blue bold")

You can also use the Layout primitives (columns, panels, and tables) to present your input user interface.

Formatting

The console.print() method supports formatting using simple markup. For example:

with input_screen() as console:
    console.print("[bold red]alert![/bold red] Something happened")

See the documentation on console markup for additional details.

You can also render markdown directly, for example:

from inspect_ai.util import input_screen
from rich.markdown import Markdown

with input_screen() as console:
    console.print(Markdown('The _quick_ brown **fox**'))

Layout

Rich includes Columns, Table and Panel classes for more advanced layout. For example, here is a simple table:

from inspect_ai.util import input_screen
from rich.table import Table

with input_screen() as console:
    table = Table(title="Tool Calls")
    table.add_column("Function", justify="left", style="cyan")
    table.add_column("Parameters", style="magenta")
    table.add_row("bash", "ls /usr/bin")
    table.add_row("python", "print('foo')")
    console.print(table)