* fix(sandbox): reject apply_patch create_file on an existing file
Add File is documented to the model as creating a new file, and the
delete and update operations both enforce their existing-file
precondition. create_file enforced nothing, so an Add File operation
aimed at a path that already existed overwrote it and reported
"Created <path>", losing the previous contents with no error.
Check that the destination is absent before writing, mirroring the
existing _ensure_exists precondition used by delete_file.
* fix(sandbox): keep the create precondition probe out of failed read spans
The absence check reached SandboxSession.read(), which records a failed
sandbox.read child span when the file is missing. Missing is the success
case for a create, so every successful Add File looked like it contained
a failed sandbox operation.
Use the existing _read_with_expected_span_errors helper, the same path
the skills capability already uses for an existence probe.
* test(sandbox): skip the create span test on Windows
The span assertion needs FilesystemTestSandboxSession, which is typed to
UnixLocalSandboxSessionState, and importing agents.sandbox.sandboxes.unix_local
raises ImportError on Windows by design. tests/conftest.py already
collect-ignores the other files that depend on it, but test_apply_patch.py
must stay collectible because its remaining tests are platform independent.
The unix-only imports stay inside the test body so collection does not
touch unix_local on Windows.
* fix(sandbox): claim the create_file name at the backend write boundary
The previous read-then-write check left a window: a concurrent sandbox
command could create the file after the probe, and the unconditional
write then destroyed that content. Reading also did not establish that a
dangling symlink entry was absent, because the unix_local path policy
resolves symlinks and the write landed on the link target.
Add BaseSandboxSession.write_new_file(), which claims the target name
before the payload is written and raises FileExistsError when the name is
already taken. The shared implementation uses a shell noclobber
redirection, so the redirect itself is the O_EXCL attempt; UnixLocal
overrides it with os.open(O_CREAT|O_EXCL) for its direct path. Both
validate the parent through the normal policy, preserving grants and the
bound user, and leave the final component unresolved so a symlink at that
name is rejected rather than followed.
apply_patch create_file now uses it and drops the probe, so the tracing
workaround for the probe read is no longer needed. Existing files still
have to go through update_file.
* fix(sandbox): link a completed payload into place for create_file
Three problems with the previous commit.
SandboxSession did not forward write_new_file, so a session built by
SandboxClient fell back to the shared implementation and never reached
the UnixLocal os.open override. Forward it.
The shared implementation created an empty file and then wrote the
payload in a separate step, so a concurrent writer could be overwritten
and a failed upload left an empty file holding the name. Write the
payload under a staging name first, then claim the target with ln, which
fails when the name is taken. The content is complete before the name
exists, and a failed create leaves only the staging entry, which is
removed.
The script used ':' for the noclobber redirection. ':' is a POSIX special
builtin, so on dash a redirection failure ended the shell before the exit
mapping ran and a collision surfaced as a generic write error instead of
FileExistsError. ln is a regular command, and a test now runs the script
through sh, dash and bash so this cannot regress silently.
Also create local files with 0o666 so the process umask decides the final
mode, matching Path.open("wb") on the ordinary write path.
* fix(sandbox): claim the requested name, not its resolved target
WorkspaceEditor normalizes the destination before dispatching, and
UnixLocal resolves leaf symlinks, so create_file handed the primitive the
link target. Add File on a dangling link.txt created missing.txt and
reported success. Pass the unresolved path for create.
Stage the payload locally too, then os.link it into place. os.link fails
with EEXIST for a file, a directory or a dangling symlink, and a write
that fails partway now leaves only the staging entry instead of a file
holding the name.
Reject a name held by a directory in the shared script. Bare ln treats an
existing directory as a target directory and would have linked the
staging file inside it while reporting the directory as created.
Move the staging write inside the cleanup scope so a failed upload cannot
leak the staging entry, and create the parent as the bound user so a
fresh nested path is owned the way the previous write path owned it.
The new tests drive session.apply_patch() rather than the primitive, which
is the path that was actually broken.
* fix(sandbox): drop the login shell and narrow the create collision
Use sh -c instead of sh -lc for the exclusive create. This path runs for a
filesystem-only capability set, so it must not source shell startup files
that live in the workspace it is editing.
A parent that is a regular file makes mkdir raise FileExistsError, and the
broad handler reported that as a collision on the requested name, telling
the model to use update_file for a target that does not exist. Only the
os.link call can report a collision now; parent and staging failures are
wrapped as write errors.
* fix(sandbox): use a fixed-length staging basename
The staging name was derived from the destination, so it was always
longer than the destination itself. A filename that fits the filesystem's
component limit, and that the ordinary write path accepts, could then
fail to stage: a 254 character name raised WorkspaceArchiveWriteError
where a plain write succeeded before this branch.
The staging basename is now constant at 52 characters regardless of the
destination.
Reported by fscfede-beep in #4930.
* fix(sandbox): classify a visible collision before staging the payload
The staging write ran before the link script could classify the target, so
a target inside an executable but non-writable parent failed on the
staging write and the caller saw WorkspaceArchiveWriteError instead of the
ApplyPatchDiffError that points it at update_file. It also meant an
Add File onto an occupied name uploaded a payload that was then discarded.
Probe the target first in both paths, then keep the atomic claim to decide
real races. A creator that wins between the probe and the link still loses
the name, and its staging entry is still cleaned up.
Reported by Codex and by fscfede-beep in #4930.
* fix(sandbox): build exclusive create on the descriptor-relative file ops
#4931 gave UnixLocal a dir_fd plus O_NOFOLLOW file-ops layer and routed the
bound-user path through a Python worker running that same module. That is
the primitive this change was approximating, so use it instead.
write_new is the existing write with O_TRUNC replaced by O_EXCL, which
claims the name in the same syscall that creates the file and fails with
EEXIST when the name is taken by a file, a directory, or a dangling
symlink. The worker exits with a distinct status for an existing target so
the caller can report FileExistsError rather than a generic write error.
This deletes the machinery the previous approach needed: the shell script,
the staging file, the hard link, and the staging cleanup. With no hard link
there is nothing to fail on writable object-storage mounts or across owners
under fs.protected_hardlinks, and with no staging entry there is nothing for
concurrent work to replace. The shared base now documents what it actually
guarantees: it rejects an occupied target but is not atomic, and a backend
that can claim a name should override it.
test_parent_swap_after_validation_cannot_access_outside[patch] injected at
session.normalize_path, which the create path no longer calls, so its swap
stopped firing. Re-pointed the patch case at the file-ops authorize
boundary the create path does traverse, keeping the coverage rather than
letting it pass vacuously. Verified separately that a parent symlinked
outside the workspace is still refused, with the outside file untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): keep supported parent symlinks and stop reading the target
Two problems with the previous commit.
Passing the whole path into the file ops unresolved made a supported
internal symlink parent fail. The ordinary write path resolves those safe
aliases, and test_safe_symlinks_grants_and_listing_paths_remain_supported
establishes the support, but a create through "internal -> real" opened
"internal" with O_NOFOLLOW | O_DIRECTORY and failed. Resolve the parent the
way write does and keep only the leaf name unresolved, so a dangling
symlink at the target name is still rejected without its target being
created.
The shared default probed the target with read(), which eagerly fetches the
whole payload on the remote backends that inherit it. An Add File aimed at
an existing large file would download it before reporting the collision,
and an existing file the bound user cannot read reported a read failure
instead of the collision. List the parent instead, which needs no payload
and no read permission on the target.
Both regressions have a test that fails on the unfixed source, including
one asserting the rejected create issues no read at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): probe the target directly instead of listing its parent
Listing the parent to decide a collision failed when the parent did not
exist. BaseSandboxSession.ls() raises ExecNonZeroError in that case, which
the probe did not catch, so a nested create such as newdir/file.txt aborted
before write() on every backend inheriting the default, while succeeding on
UnixLocal. Those backends' write paths create missing parents.
Use a bare existence test instead. It needs only execute permission on the
parent, never reads the target, and reports absent when the parent is
missing so the create still reaches write(). Only a positive result rejects;
any other probe outcome falls through rather than blocking a valid create.
The test is also more accurate than the listing: it sees a file inside an
executable but unreadable parent, which listing could not, so an existing
file there is no longer silently overwritten.
Validated the exact shipped script under sh, bash and dash for an existing
file, a dangling symlink, an absent name, a missing parent, and a file in a
0311 parent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): fail closed on a failed probe and keep scripted creates working
Two problems with the previous commit.
The probe treated any unexpected status as "absent" so a valid create would
not be blocked. That was wrong: exit 0 already means absent, including when
the parent does not exist, so the permissive fallback only covered the case
where the probe itself did not run. On provider sessions whose write() uses
a separate upload API, the write then succeeds and overwrites an existing
target precisely when the precondition could not be checked. Only 0 may
proceed now; 13 stays the collision and every other status is a write error.
scripted_sandbox_session() broke. The inherited default reaches for exec,
and a script that configures only file steps hides exec, so a previously
valid mkdir-and-write script failed with AttributeError. Give the scripted
session an implementation that consumes the same mkdir and write pair, and
restore the explicit parent creation in the default so the observable call
sequence matches what the create path did before this branch.
Both have a test that fails on the unfixed source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(sandbox): keep scripted create calls on their normalized paths
The create path hands write_new_file an unresolved path on purpose, so a
symlink at the target name is rejected rather than followed. The scripted
session forwarded that path straight into its compatibility mkdir and write
pair, so a script using the documented match callback saw "." and
"notes.txt" where it previously saw the workspace paths, and failed with
SandboxCallMatcherError.
Normalize before emitting the pair. Verified against the previous commit:
with a custom manifest root the calls were "sub" and "sub/notes.txt" and
are now "/custom/root/sub" and "/custom/root/sub/notes.txt", matching what
this surface recorded before this branch.
The earlier test only asserted method names, which is why it could not catch
this. It now asserts the recorded paths too and fails on the unfixed source.
* fix(sandbox): narrow exclusive creates and clarify partial-write recovery
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Justin Beckwith <jbeckwith@openai.com>
599 lines
20 KiB
Python
599 lines
20 KiB
Python
from __future__ import annotations
|
|
|
|
import inspect
|
|
import io
|
|
from pathlib import Path
|
|
from types import MappingProxyType
|
|
from typing import Any, cast, get_type_hints
|
|
|
|
import pytest
|
|
from typing_extensions import assert_type
|
|
|
|
from agents import RunConfig, Runner
|
|
from agents.editor import ApplyPatchOperation
|
|
from agents.sandbox import ExecResult, Manifest, SandboxAgent
|
|
from agents.sandbox.apply_patch import WorkspaceEditor
|
|
from agents.sandbox.capabilities import Shell
|
|
from agents.sandbox.files import FileEntry
|
|
from agents.sandbox.session.base_sandbox_session import BaseSandboxSession
|
|
from agents.sandbox.session.pty_types import PtyExecUpdate
|
|
from agents.testing import (
|
|
InvalidSandboxStep,
|
|
SandboxCall,
|
|
SandboxCallMatcherError,
|
|
ScriptedModel,
|
|
ScriptedSandboxSession,
|
|
UnconsumedSandboxSteps,
|
|
UnexpectedSandboxCall,
|
|
assistant_message,
|
|
function_call,
|
|
scripted_sandbox_session,
|
|
)
|
|
from agents.testing.sandbox import ScriptedSandboxSession as CanonicalScriptedSandboxSession
|
|
|
|
|
|
class _CallableBytesIO(io.BytesIO):
|
|
def __call__(self) -> None:
|
|
pass
|
|
|
|
|
|
class _CallableBufferedReader(io.BufferedReader):
|
|
def __call__(self) -> None:
|
|
pass
|
|
|
|
|
|
def test_scripted_sandbox_has_public_result_type() -> None:
|
|
session = scripted_sandbox_session()
|
|
assert_type(session, ScriptedSandboxSession)
|
|
base_session: BaseSandboxSession = session
|
|
|
|
assert ScriptedSandboxSession is CanonicalScriptedSandboxSession
|
|
assert get_type_hints(scripted_sandbox_session)["return"] is ScriptedSandboxSession
|
|
assert inspect.isabstract(ScriptedSandboxSession)
|
|
assert isinstance(session, ScriptedSandboxSession)
|
|
assert base_session is session
|
|
|
|
|
|
def test_scripted_sandbox_exposes_only_configured_scriptable_methods() -> None:
|
|
session = scripted_sandbox_session(
|
|
[{"method": "exec", "result": ExecResult(stdout=b"", stderr=b"", exit_code=0)}]
|
|
)
|
|
|
|
assert isinstance(session, BaseSandboxSession)
|
|
assert hasattr(session, "exec")
|
|
assert not hasattr(session, "read")
|
|
assert not hasattr(session, "apply_patch")
|
|
assert not hasattr(session, "pty_exec_start")
|
|
assert "exec" in dir(session)
|
|
assert "read" not in dir(session)
|
|
assert "apply_patch" not in dir(session)
|
|
assert "pty_exec_start" not in dir(session)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_cleanup_does_not_advertise_pty_termination() -> None:
|
|
session = scripted_sandbox_session()
|
|
|
|
assert not hasattr(session, "pty_terminate_all")
|
|
assert "pty_terminate_all" not in dir(session)
|
|
|
|
await session.aclose()
|
|
|
|
async with scripted_sandbox_session() as context_session:
|
|
assert await context_session.running() is True
|
|
|
|
assert await context_session.running() is False
|
|
|
|
|
|
def test_scripted_sandbox_snapshots_manifest_and_derives_pty_support() -> None:
|
|
manifest = Manifest(root="/configured")
|
|
session = scripted_sandbox_session(
|
|
[{"method": "pty_exec_start", "result": None}],
|
|
manifest=manifest,
|
|
)
|
|
manifest.root = "/mutated"
|
|
|
|
assert session.state.manifest.root == "/configured"
|
|
assert session.supports_pty() is True
|
|
|
|
pty_session = scripted_sandbox_session(
|
|
[
|
|
{"method": "pty_exec_start", "result": None},
|
|
{"method": "pty_write_stdin", "result": None},
|
|
]
|
|
)
|
|
assert pty_session.supports_pty() is True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("configured_method", "missing_method"),
|
|
[
|
|
("pty_exec_start", "pty_write_stdin"),
|
|
("pty_write_stdin", "pty_exec_start"),
|
|
],
|
|
)
|
|
async def test_scripted_sandbox_exposes_pty_methods_as_one_capability(
|
|
configured_method: str,
|
|
missing_method: str,
|
|
) -> None:
|
|
session = scripted_sandbox_session([{"method": configured_method, "result": None}])
|
|
|
|
assert session.supports_pty() is True
|
|
assert hasattr(session, "pty_exec_start")
|
|
assert hasattr(session, "pty_write_stdin")
|
|
assert "pty_exec_start" in dir(session)
|
|
assert "pty_write_stdin" in dir(session)
|
|
|
|
with pytest.raises(UnexpectedSandboxCall) as exc_info:
|
|
if missing_method == "pty_exec_start":
|
|
await session.pty_exec_start("pwd")
|
|
else:
|
|
await session.pty_write_stdin(session_id=1, chars="")
|
|
|
|
assert exc_info.value.actual_method == missing_method
|
|
assert exc_info.value.expected_method == configured_method
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_supports_capability_method_inventory() -> None:
|
|
read_result = io.BytesIO(b"contents")
|
|
list_result: list[FileEntry] = []
|
|
pty_start_result = PtyExecUpdate(
|
|
process_id=123,
|
|
output=b"started",
|
|
exit_code=None,
|
|
original_token_count=None,
|
|
)
|
|
pty_write_result = PtyExecUpdate(
|
|
process_id=None,
|
|
output=b"done",
|
|
exit_code=0,
|
|
original_token_count=None,
|
|
)
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{"method": "read", "result": read_result},
|
|
{"method": "write", "result": None},
|
|
{"method": "ls", "result": list_result},
|
|
{"method": "mkdir", "result": None},
|
|
{"method": "rm", "result": None},
|
|
{"method": "apply_patch", "result": "Done!"},
|
|
{"method": "pty_exec_start", "result": pty_start_result},
|
|
{"method": "pty_write_stdin", "result": pty_write_result},
|
|
]
|
|
)
|
|
stream = io.BytesIO(b"payload")
|
|
|
|
returned_read_result = cast(io.BytesIO, await session.read(Path("in.txt")))
|
|
assert returned_read_result is not read_result
|
|
assert returned_read_result.getvalue() == b"contents"
|
|
await session.write(Path("out.txt"), stream)
|
|
assert await session.ls(".") == []
|
|
await session.mkdir("new", parents=True)
|
|
await session.rm("old", recursive=True)
|
|
assert await session.apply_patch({"type": "delete_file", "path": "old.txt"}) == "Done!"
|
|
assert (await session.pty_exec_start("sh", tty=True)).process_id == 123
|
|
assert (await session.pty_write_stdin(session_id=123, chars="exit")).exit_code == 0
|
|
|
|
assert [call.method for call in session.calls] == [
|
|
"read",
|
|
"write",
|
|
"ls",
|
|
"mkdir",
|
|
"rm",
|
|
"apply_patch",
|
|
"pty_exec_start",
|
|
"pty_write_stdin",
|
|
]
|
|
recorded_stream = cast(io.BytesIO, session.calls[1].args[1])
|
|
assert recorded_stream is not stream
|
|
assert recorded_stream.getvalue() == b"payload"
|
|
assert recorded_stream.tell() == 0
|
|
session.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
("configured", "original_content"),
|
|
[(io.BytesIO(b"before"), b"before"), (io.StringIO("before"), "before")],
|
|
)
|
|
async def test_scripted_sandbox_snapshots_supported_stream_results(
|
|
configured: io.BytesIO | io.StringIO,
|
|
original_content: bytes | str,
|
|
) -> None:
|
|
configured.seek(2)
|
|
session = scripted_sandbox_session([{"method": "read", "result": configured}])
|
|
configured.seek(0)
|
|
configured.write(b"after" if isinstance(configured, io.BytesIO) else "after")
|
|
configured.close()
|
|
|
|
result = cast(io.BytesIO | io.StringIO, await session.read(Path("input.txt")))
|
|
|
|
assert result is not configured
|
|
assert result.tell() == 2
|
|
assert result.getvalue() == original_content
|
|
session.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_snapshots_supported_stream_call_arguments() -> None:
|
|
session = scripted_sandbox_session([{"method": "write", "result": None}])
|
|
source = io.BytesIO(b"before")
|
|
source.seek(3)
|
|
|
|
await session.write(Path("output.bin"), source)
|
|
source.seek(0)
|
|
source.write(b"after")
|
|
source.close()
|
|
|
|
recorded = cast(io.BytesIO, session.calls[0].args[1])
|
|
assert recorded is not source
|
|
assert recorded.tell() == 3
|
|
assert recorded.getvalue() == b"before"
|
|
recorded.write(b"changed")
|
|
retained = cast(io.BytesIO, session.calls[0].args[1])
|
|
assert retained.tell() == 3
|
|
assert retained.getvalue() == b"before"
|
|
session.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_snapshots_callable_supported_streams() -> None:
|
|
result_source = _CallableBytesIO(b"result")
|
|
call_source = _CallableBytesIO(b"call")
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{"method": "read", "result": result_source},
|
|
{"method": "write", "result": None},
|
|
]
|
|
)
|
|
|
|
result = cast(io.BytesIO, await session.read(Path("input.bin")))
|
|
await session.write(Path("output.bin"), call_source)
|
|
|
|
assert result is not result_source
|
|
assert result.getvalue() == b"result"
|
|
recorded = cast(io.BytesIO, session.calls[1].args[1])
|
|
assert recorded is not call_source
|
|
assert recorded.getvalue() == b"call"
|
|
session.assert_complete()
|
|
|
|
|
|
def test_scripted_sandbox_rejects_unsupported_or_closed_stream_results() -> None:
|
|
with io.BufferedReader(io.BytesIO(b"payload")) as unsupported:
|
|
with pytest.raises(InvalidSandboxStep) as unsupported_info:
|
|
scripted_sandbox_session([{"method": "read", "result": unsupported}])
|
|
assert unsupported_info.value.reason == "invalid_outcome"
|
|
|
|
closed = io.BytesIO(b"payload")
|
|
closed.close()
|
|
with pytest.raises(InvalidSandboxStep) as closed_info:
|
|
scripted_sandbox_session([{"method": "read", "result": closed}])
|
|
assert closed_info.value.reason == "invalid_outcome"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_rejects_unsupported_stream_call_before_commit() -> None:
|
|
session = scripted_sandbox_session([{"method": "write", "result": None}])
|
|
|
|
with io.BufferedReader(io.BytesIO(b"payload")) as unsupported:
|
|
with pytest.raises(TypeError, match="support only io.BytesIO and io.StringIO"):
|
|
await session.write(Path("output.bin"), unsupported)
|
|
|
|
assert session.calls == ()
|
|
assert session.remaining_steps == 1
|
|
|
|
|
|
def test_scripted_sandbox_rejects_callable_unsupported_stream_result() -> None:
|
|
with _CallableBufferedReader(io.BytesIO(b"payload")) as unsupported:
|
|
with pytest.raises(InvalidSandboxStep) as exc_info:
|
|
scripted_sandbox_session([{"method": "read", "result": unsupported}])
|
|
|
|
assert exc_info.value.reason == "invalid_outcome"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"stream_factory",
|
|
[
|
|
lambda: _CallableBytesIO(b"payload"),
|
|
lambda: _CallableBufferedReader(io.BytesIO(b"payload")),
|
|
],
|
|
)
|
|
@pytest.mark.parametrize(
|
|
("field", "reason"),
|
|
[("match", "invalid_matcher"), ("responder", "invalid_outcome")],
|
|
)
|
|
def test_scripted_sandbox_rejects_callable_stream_matchers_and_responders(
|
|
stream_factory: Any,
|
|
field: str,
|
|
reason: str,
|
|
) -> None:
|
|
stream = cast(io.IOBase, stream_factory())
|
|
step: dict[str, Any] = {"method": "exec", field: stream}
|
|
if field == "match":
|
|
step["result"] = ExecResult(stdout=b"", stderr=b"", exit_code=0)
|
|
|
|
try:
|
|
with pytest.raises(InvalidSandboxStep) as exc_info:
|
|
scripted_sandbox_session([step])
|
|
finally:
|
|
stream.close()
|
|
|
|
assert exc_info.value.reason == reason
|
|
assert exc_info.value.input_index == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_rejects_callable_unsupported_stream_call_before_commit() -> None:
|
|
session = scripted_sandbox_session([{"method": "write", "result": None}])
|
|
|
|
with _CallableBufferedReader(io.BytesIO(b"payload")) as unsupported:
|
|
with pytest.raises(TypeError, match="support only io.BytesIO and io.StringIO"):
|
|
await session.write(Path("output.bin"), unsupported)
|
|
|
|
assert session.calls == ()
|
|
assert session.remaining_steps == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_snapshots_static_results_when_queued() -> None:
|
|
configured = ExecResult(stdout=b"before", stderr=b"", exit_code=0)
|
|
session = scripted_sandbox_session([{"method": "exec", "result": configured}])
|
|
configured.stdout = b"after"
|
|
|
|
result = await session.exec("pwd")
|
|
|
|
assert result.stdout == b"before"
|
|
session.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_records_detached_fifo_calls() -> None:
|
|
source_operations: list[dict[str, object]] = [{"type": "delete_file", "path": "before.txt"}]
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{"method": "apply_patch", "result": "Done!"},
|
|
{"method": "exec", "result": ExecResult(stdout=b"ok", stderr=b"", exit_code=0)},
|
|
]
|
|
)
|
|
|
|
assert await session.apply_patch(cast(Any, source_operations)) == "Done!"
|
|
source_operations[0]["path"] = "after.txt"
|
|
result = await session.exec("pwd", shell=False)
|
|
|
|
assert result.stdout == b"ok"
|
|
assert session.remaining_steps == 0
|
|
session.assert_complete()
|
|
assert session.calls[0].method == "apply_patch"
|
|
assert session.calls[0].args[0] == [{"type": "delete_file", "path": "before.txt"}]
|
|
assert session.calls[1] == SandboxCall(
|
|
call_index=1,
|
|
method="exec",
|
|
args=("pwd",),
|
|
kwargs=MappingProxyType({"timeout": None, "shell": False, "user": None}),
|
|
)
|
|
|
|
returned_operations = cast(list[dict[str, object]], session.calls[0].args[0])
|
|
returned_operations[0]["path"] = "mutated.txt"
|
|
assert session.calls[0].args[0] == [{"type": "delete_file", "path": "before.txt"}]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_snapshot_failure_does_not_commit_call() -> None:
|
|
class Uncopyable:
|
|
def __deepcopy__(self, memo: dict[int, object]) -> object:
|
|
_ = memo
|
|
raise RuntimeError("cannot snapshot")
|
|
|
|
session = scripted_sandbox_session(
|
|
[{"method": "exec", "result": ExecResult(stdout=b"", stderr=b"", exit_code=0)}]
|
|
)
|
|
|
|
with pytest.raises(RuntimeError, match="cannot snapshot"):
|
|
await session.exec(cast(Any, Uncopyable()))
|
|
|
|
assert session.calls == ()
|
|
assert session.remaining_steps == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_supports_matchers_responders_and_errors() -> None:
|
|
injected_error = RuntimeError("sandbox unavailable")
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{
|
|
"method": "exec",
|
|
"match": lambda call: call.args == ("pwd",),
|
|
"responder": lambda call: ExecResult(
|
|
stdout=f"call {len(call.args)}".encode(), stderr=b"", exit_code=0
|
|
),
|
|
},
|
|
{"method": "exec", "error": injected_error},
|
|
]
|
|
)
|
|
|
|
assert (await session.exec("pwd")).stdout == b"call 1"
|
|
with pytest.raises(RuntimeError) as exc_info:
|
|
await session.exec("next")
|
|
assert exc_info.value is injected_error
|
|
session.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_reports_structured_payload_free_failures() -> None:
|
|
mismatch = scripted_sandbox_session(
|
|
[
|
|
{"method": "exec", "result": ExecResult(stdout=b"", stderr=b"", exit_code=0)},
|
|
{"method": "read", "result": None},
|
|
]
|
|
)
|
|
|
|
with pytest.raises(UnexpectedSandboxCall) as mismatch_info:
|
|
await mismatch.read(Path("/secret/payload.txt"))
|
|
mismatch_error = mismatch_info.value
|
|
assert mismatch_error.call_index == 0
|
|
assert "call #1" in str(mismatch_error)
|
|
assert mismatch_error.actual_method == "read"
|
|
assert mismatch_error.expected_method == "exec"
|
|
assert mismatch_error.remaining_steps == 2
|
|
assert mismatch.remaining_steps == 2
|
|
assert "/secret/payload.txt" not in str(mismatch_error)
|
|
await mismatch.exec("retry")
|
|
assert mismatch.remaining_steps == 1
|
|
|
|
rejected = scripted_sandbox_session(
|
|
[
|
|
{
|
|
"method": "exec",
|
|
"match": lambda call: call.args == ("expected",),
|
|
"result": ExecResult(stdout=b"", stderr=b"", exit_code=0),
|
|
}
|
|
]
|
|
)
|
|
with pytest.raises(SandboxCallMatcherError) as rejected_info:
|
|
await rejected.exec("secret command")
|
|
assert rejected_info.value.call_index == 0
|
|
assert "call #1" in str(rejected_info.value)
|
|
assert rejected_info.value.method == "exec"
|
|
assert "secret command" not in str(rejected_info.value)
|
|
assert rejected.remaining_steps == 1
|
|
await rejected.exec("expected")
|
|
rejected.assert_complete()
|
|
|
|
extra = scripted_sandbox_session(
|
|
[{"method": "exec", "result": ExecResult(stdout=b"", stderr=b"", exit_code=0)}]
|
|
)
|
|
await extra.exec("first")
|
|
assert hasattr(extra, "exec")
|
|
with pytest.raises(UnexpectedSandboxCall) as extra_info:
|
|
await extra.exec("second secret")
|
|
assert extra_info.value.call_index == 1
|
|
assert "call #2" in str(extra_info.value)
|
|
assert extra_info.value.expected_method is None
|
|
assert extra_info.value.remaining_steps == 0
|
|
assert "second secret" not in str(extra_info.value)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_retains_step_after_matcher_exception() -> None:
|
|
matcher_error = RuntimeError("matcher failed")
|
|
matcher_calls = 0
|
|
|
|
def match(_call: SandboxCall) -> bool:
|
|
nonlocal matcher_calls
|
|
matcher_calls += 1
|
|
if matcher_calls == 1:
|
|
raise matcher_error
|
|
return True
|
|
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{
|
|
"method": "exec",
|
|
"match": match,
|
|
"result": ExecResult(stdout=b"ok", stderr=b"", exit_code=0),
|
|
}
|
|
]
|
|
)
|
|
|
|
with pytest.raises(RuntimeError) as exc_info:
|
|
await session.exec("first")
|
|
|
|
assert exc_info.value is matcher_error
|
|
assert session.remaining_steps == 1
|
|
assert (await session.exec("retry")).stdout == b"ok"
|
|
session.assert_complete()
|
|
|
|
|
|
def test_scripted_sandbox_reports_unconsumed_steps() -> None:
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{"method": "exec", "result": ExecResult(stdout=b"", stderr=b"", exit_code=0)},
|
|
{"method": "read", "result": None},
|
|
]
|
|
)
|
|
|
|
with pytest.raises(UnconsumedSandboxSteps) as exc_info:
|
|
session.assert_complete()
|
|
assert exc_info.value.remaining_steps == 2
|
|
assert exc_info.value.pending_methods == ("exec", "read")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("step", "reason"),
|
|
[
|
|
("not a mapping", "invalid_input"),
|
|
({"method": "exec", "matc": lambda _call: True, "result": None}, "invalid_input"),
|
|
({"method": "unknown", "result": None}, "unknown_method"),
|
|
({"method": "exec", "match": "no", "result": None}, "invalid_matcher"),
|
|
({"method": "exec"}, "invalid_outcome"),
|
|
({"method": "exec", "result": None, "error": RuntimeError()}, "invalid_outcome"),
|
|
({"method": "exec", "responder": "no"}, "invalid_outcome"),
|
|
({"method": "exec", "error": "no"}, "invalid_outcome"),
|
|
],
|
|
)
|
|
def test_scripted_sandbox_validates_steps_before_use(step: object, reason: str) -> None:
|
|
with pytest.raises(InvalidSandboxStep) as exc_info:
|
|
scripted_sandbox_session(cast(Any, [step]))
|
|
assert exc_info.value.reason == reason
|
|
assert exc_info.value.input_index == 0
|
|
assert "step #1" in str(exc_info.value)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_drives_black_box_sandbox_agent_workflow() -> None:
|
|
session = scripted_sandbox_session(
|
|
[
|
|
{
|
|
"method": "exec",
|
|
"match": lambda call: call.args == ("pwd",),
|
|
"result": ExecResult(stdout=b"/workspace\n", stderr=b"", exit_code=0),
|
|
}
|
|
]
|
|
)
|
|
model = ScriptedModel(
|
|
[
|
|
[function_call("exec_command", {"cmd": "pwd"}, call_id="call_1")],
|
|
[assistant_message("The workspace is /workspace.")],
|
|
]
|
|
)
|
|
agent = SandboxAgent(
|
|
name="Test agent",
|
|
model=model,
|
|
capabilities=[Shell()],
|
|
)
|
|
|
|
result = await Runner.run(
|
|
agent,
|
|
"Where am I?",
|
|
run_config=RunConfig(sandbox={"session": session}),
|
|
)
|
|
|
|
assert result.final_output == "The workspace is /workspace."
|
|
assert len(session.calls) == 1
|
|
assert len(model.calls) == 2
|
|
session.assert_complete()
|
|
model.assert_complete()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_scripted_sandbox_supports_apply_patch_create() -> None:
|
|
"""Creation keeps the existing file steps without requiring a scripted exec call."""
|
|
session = scripted_sandbox_session(
|
|
[{"method": "mkdir", "result": None}, {"method": "write", "result": None}]
|
|
)
|
|
|
|
result = await WorkspaceEditor(session).apply_operation(
|
|
ApplyPatchOperation(type="create_file", path="notes.txt", diff="+hello\n")
|
|
)
|
|
|
|
assert result.output == "Created notes.txt"
|
|
assert session.remaining_steps == 0
|
|
# The recorded paths matter as much as the methods. The create path hands over an
|
|
# unresolved path so a symlinked leaf is not followed, and a script matching on
|
|
# arguments still expects the workspace paths these calls have always carried.
|
|
assert [(call.method, call.args[0]) for call in session.calls] == [
|
|
("mkdir", Path("/workspace")),
|
|
("write", Path("/workspace/notes.txt")),
|
|
]
|