VS Code Extension
The Inspect VS Code Extension provides a variety of tools, including:
- An integrated viewer for evaluation log files
- Commands and key-bindings for running tasks
- Commands and key-bindings for debugging tasks
- A configuration panel that edits config in workspace
.env
files - A task panel for setting task CLI options and task arguments
- A panel for browsing all tasks contained in the workspace
Installation
To install, search for “Inspect AI” in the extensions marketplace panel within VS Code.
The Inspect extension will automatically bind to the Python interpreter associated with the current workspace, so you should be sure that the inspect-ai
package is installed within that environment. Use the Python: Select Interpreter command to associate a version of Python with your workspace.
Running Tasks
There are many ways to run evaluation tasks within VS Code, including:
- Calling
inspect eval
in the terminal - Calling
eval()
directly in a notebook or script - Using the Run Task button in the editor
- Using the Cmd+Shift+U keyboard shortcut (which will run the task currently selected in the active editor).
No matter how you run a task, when the task is complete the eval log will automatically display to the right of the active editor:
If you’d rather have the log viewer display in its own window (e.g. to move it to a different monitor) you can drag the Inspect View tab out of VS Code and move it to wherever is most convenient.
If you prefer not to have logs shown automatically, there is a setting which can disable this behaviour (search settings for “inspect log”). In that case, you can still view eval logs by clicking on the link at the bottom of the eval results:
See the article on the Log Viewer for additional details on using it to explore eval results.
Debugging Tasks
Note that when debugging a task, the Inspect extension will automatically limit the eval to a single sample (--limit 1
on the command line). If you prefer to debug with many samples, there is a setting that can disable the default behavior (search settings for “inspect debug”).
Activity Bar
The Inspect Activity Bar provides an interface for tuning both global configuration and task CLI arguments. Access the Activity Bar by clicking the Inspect icon on the left side of the VS Code workspace:
The activity bar has three panels:
Configuration edits global configuration by reading and writing values from the workspace
.env
config file (see the documentation on Configuration for more details on.env
files).Task provides a way to tweak the CLI arguments passed to
inspect eval
when it is run from the user interface.Tasks (not shown above) lists all tasks in the current workspace, and can be used to both navigate among tasks as well as run and debug tasks directly.
Troubleshooting
If the Inspect extension is not loading into the workspace, you should investigate what version of Python it is discovering as well as whether the inspect-ai
package is detected within that Python environment. Use the Output panel (at the bottom of VS Code in the same panel as the Terminal) and select the Inspect output channel using the picker on the right side of the panel:
Note that the Inspect extension will automatically bind to the Python interpreter associated with the current workspace, so you should be sure that the inspect-ai
package is installed within that environment. Use the Python: Select Interpreter command to associate a version of Python with your workspace.