* fix(sandbox): make Docker persist_workspace archives restorable
Docker's persist_workspace() stages a copy of the workspace, has the
daemon archive it, and rewrites the member prefix in Python with
strip_tar_member_prefix(). That rewrite raised UnsafeTarMemberError
("hardlink member not allowed", "unsupported member type") as soon as
the archive contained a hardlink member or a FIFO, so snapshotting a
workspace where uv or pnpm had hardlinked installed packages, or a dev
server had left a FIFO behind, failed outright. An absolute symlink
target under the workspace root survived persist but was refused by
the strict hydrate extractor.
Rewrite those members while stripping the prefix: hardlink members are
stored as regular files carrying the target's payload (the source is
spooled to a temporary file so the earlier member can be re-read),
FIFOs and device nodes are dropped, and, when the caller passes the
workspace root, absolute symlink targets under it become relative to
the link's directory. Absolute targets outside the workspace are left
unchanged for hydrate's policy.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BN4v25msJgrjNgac97g1Az
* fix(sandbox): stream the Docker archive while rewriting hardlink members
Keep reading the source archive as a stream instead of spooling it to a
temporary file first. A hardlink member's payload is read back from the
rewritten archive being written (recorded by original member name), so
peak temporary usage stays at one archive rather than the source plus
the output.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCNKceEs9sPdb6aHK3FqPf
* fix(sandbox): drop the stale PAX linkpath when relativizing a symlink target
A symlink target longer than the ustar field is carried in a PAX
"linkpath" record. Rewriting only TarInfo.linkname left that record
pointing at the original absolute target, and addfile() emitted it, so
the rewritten archive still held the absolute link and strict hydrate
refused it. Remove the record; tobuf() re-derives it from the new
linkname when needed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFSMrLZZ3oq1dKmJFAofz6
* fix(sandbox): relativize in-workspace symlink targets that start with a double slash
posixpath.normpath keeps two leading slashes, so //workspace/a.txt was
not recognized as under the workspace root and stayed absolute; Linux
resolves // as /, so collapse it before the containment check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFSMrLZZ3oq1dKmJFAofz6
* fix(sandbox): narrow Docker snapshot normalization to FIFOs and in-workspace symlinks
Address review: Docker stages the workspace with `cp -R`, which already
copies hardlinked files independently, so drop the hardlink expansion
(payload read-back, spooling) and keep only what the staged copy really
carries and the strict hydrate extractor refuses: FIFO/device members
are dropped, and absolute symlink targets under the workspace root are
rebased onto the link's directory with their components kept verbatim.
Only the root prefix is replaced, never normalized: with
`alias -> sub/deep`, `/workspace/alias/../data.txt` names `sub/data.txt`
and must stay `alias/../data.txt`. Add a Docker-session persist/hydrate
test that goes through staging and reads the restored links.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFSMrLZZ3oq1dKmJFAofz6
* fix(sandbox): consume the whole separator run after the workspace root when rebasing symlinks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFSMrLZZ3oq1dKmJFAofz6
* fix(sandbox): prove rebased symlink targets stay under the root, test link metadata only
The textual rebase keeps every component after the workspace root verbatim, so
`a/link/../tmp` is only inside the workspace if `a/link` resolves there. With
`a/link -> ..` it names `/tmp` after extraction while hydrate's lexical check
accepts it, turning an absolute target the strict extractor would have refused
into a relative one it lets through. strip_tar_member_prefix now resolves each
rebased target through the archive's own symlink members, applying `..` to a
link's target the way the kernel does, and rejects the archive when the walk
leaves the root, passes through a link whose target stayed absolute, or exceeds
the ELOOP budget. This is the same outcome as the hardlink case: the snapshot is
refused at persist time with a named member instead of failing on restore.
The regression tests no longer read through restored POSIX symlinks on the
host, which raised EINVAL on the Windows jobs, and the Docker test extracts the
archive member by member as GNU tar in the container does instead of relying on
`extractall`, whose default filter rewrites `alias/../data.txt` to `data.txt`
on Python 3.14. Assertions are on the archive and restored link metadata.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRNVNFM6bqUNRQBnw5GBVC
* fix(sandbox): only rebase symlinks whose every component the archive establishes
hydrate_workspace() extracts into an existing root, so a component the archive
does not create may already be a symlink in the destination: a persisted
`victim -> /workspace/alias/../secret` rebased to `alias/../secret` resolves to
`/tmp/secret` when the destination holds `alias -> /tmp/sub`. The containment
walk now requires each intermediate component to be a directory member and the
leaf to be any member of the archive itself; those are the paths the extractor
protects with its own destination checks. Symlink members are still followed
and applied to `..` as before; anything the archive does not establish, a
dangling leaf included, fails the proof and the archive is rejected as with the
other unprovable cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRNVNFM6bqUNRQBnw5GBVC
* fix(sandbox): rebase only simple archive-established symlink targets, drop the resolver
Addresses the Codex P1 on 37a5c403 ("replace the symlink graph emulator with
narrow rejection"). `_resolve_through_archive_members` and
`_validate_rebased_symlinks_contained` are removed.
`strip_tar_member_prefix` now works in two passes. The first strips the prefix
and drops FIFOs and device nodes as before, keeping every symlink target
verbatim. The second, which sees every member, rewrites an absolute target
under the root only when it is *simple* and *established*: every component is
a plain name (no `.` or `..`), every directory the relative form walks through
(the link's own parents and the target's parents) is an explicit directory
member of the archive, and the leaf is a regular file or directory member.
Anything else (`/workspace/alias/../data.txt`, a leaf that is itself a
symlink, a directory only implied by a file path, a missing leaf) is left
absolute so the strict hydrate validation refuses it with its existing
"absolute symlink target not allowed" error, instead of the rewrite modelling
how another link resolves. `rebase_symlink_target` keeps its signature and
returns the input unchanged for non-simple targets.
Tests: the resolver cases become one parametrized
`test_strip_tar_member_prefix_leaves_non_simple_targets_absolute` (nine
shapes, each asserting the target stays absolute and strict validation
refuses it); positive coverage in
`..._rebases_simple_targets_established_by_the_archive`; the fixture's
alias-relative link is asserted to stay absolute and to be refused by the
strict check; the Docker round trip keeps only archive-established targets and
a sibling test shows `hydrate_workspace` refusing the alias-relative one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DFSMrLZZ3oq1dKmJFAofz6
* fix(sandbox): rebase links to the workspace root, leave trailing-separator targets absolute
Two restoration edge cases in the narrowed symlink rebase:
- `link -> /workspace` (or `/workspace/`) has no components after the root, and
`_archive_establishes` refused the empty tuple, so the target stayed absolute
and hydrate rejected the archive. The root is established by hydration
itself; only the climb out of the link's own parents has to be proven, so
such links now become `.` / `../..`.
- `link -> /workspace/a.txt/` lost its trailing separator when empty segments
were filtered, turning a link that fails with ENOTDIR on the source into a
working file link. A trailing separator after a component is no longer a
simple target; it is left absolute for strict hydration to refuse.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(sandbox): stream Docker snapshots once and narrow link rebasing
---------
Co-authored-by: coderdailyone <codecookie@proton.me>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Kazuhiro Sera <seratch@openai.com>
Co-authored-by: root <root@ser539252086371.local>
Co-authored-by: Justin Beckwith <jbeckwith@openai.com>
856 lines
31 KiB
Python
856 lines
31 KiB
Python
from __future__ import annotations
|
|
|
|
import errno
|
|
import io
|
|
import os
|
|
import stat
|
|
import sys
|
|
import tarfile
|
|
from dataclasses import dataclass
|
|
from pathlib import Path, PurePosixPath
|
|
|
|
import pytest
|
|
from typing_extensions import Buffer
|
|
|
|
from agents.sandbox.util import tar_utils
|
|
from agents.sandbox.util.tar_utils import (
|
|
UnsafeTarMemberError,
|
|
safe_extract_tarfile,
|
|
safe_tar_member_rel_path,
|
|
strip_tar_member_prefix,
|
|
validate_tar_bytes,
|
|
validate_tarfile,
|
|
)
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class _Member:
|
|
info: tarfile.TarInfo
|
|
payload: bytes | None = None
|
|
|
|
|
|
def _tar_bytes(*members: _Member) -> bytes:
|
|
buf = io.BytesIO()
|
|
with tarfile.open(fileobj=buf, mode="w") as tar:
|
|
for member in members:
|
|
if member.payload is None:
|
|
tar.addfile(member.info)
|
|
else:
|
|
tar.addfile(member.info, io.BytesIO(member.payload))
|
|
return buf.getvalue()
|
|
|
|
|
|
def _dir(name: str) -> _Member:
|
|
member = tarfile.TarInfo(name)
|
|
member.type = tarfile.DIRTYPE
|
|
return _Member(member)
|
|
|
|
|
|
def _file(name: str, payload: bytes = b"payload", mode: int | None = None) -> _Member:
|
|
member = tarfile.TarInfo(name)
|
|
member.size = len(payload)
|
|
if mode is not None:
|
|
member.mode = mode
|
|
return _Member(member, payload)
|
|
|
|
|
|
def _symlink(name: str, target: str) -> _Member:
|
|
member = tarfile.TarInfo(name)
|
|
member.type = tarfile.SYMTYPE
|
|
member.linkname = target
|
|
return _Member(member)
|
|
|
|
|
|
def _hardlink(name: str, target: str) -> _Member:
|
|
member = tarfile.TarInfo(name)
|
|
member.type = tarfile.LNKTYPE
|
|
member.linkname = target
|
|
return _Member(member)
|
|
|
|
|
|
def _fifo(name: str) -> _Member:
|
|
member = tarfile.TarInfo(name)
|
|
member.type = tarfile.FIFOTYPE
|
|
return _Member(member)
|
|
|
|
|
|
def _safe_extract(raw: bytes, root: Path) -> None:
|
|
with tarfile.open(fileobj=io.BytesIO(raw), mode="r:*") as tar:
|
|
safe_extract_tarfile(tar, root=root)
|
|
|
|
|
|
def test_safe_extract_tarfile_preserves_venv_style_symlinks(tmp_path: Path) -> None:
|
|
raw = _tar_bytes(
|
|
_dir("."),
|
|
_dir("./uv-project"),
|
|
_dir("./uv-project/.venv"),
|
|
_dir("./uv-project/.venv/bin"),
|
|
_dir("./uv-project/.venv/lib"),
|
|
_file("./uv-project/main.py", b'print("snapshot smoke")\n'),
|
|
_symlink("./uv-project/.venv/lib64", "lib"),
|
|
_symlink("./uv-project/.venv/bin/python3", "/usr/local/bin/python3"),
|
|
_symlink("./uv-project/.venv/bin/python", "python3"),
|
|
)
|
|
|
|
validate_tar_bytes(raw)
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
assert (tmp_path / "uv-project" / "main.py").read_text() == 'print("snapshot smoke")\n'
|
|
assert os.readlink(tmp_path / "uv-project" / ".venv" / "lib64") == "lib"
|
|
assert (
|
|
os.readlink(tmp_path / "uv-project" / ".venv" / "bin" / "python3")
|
|
== "/usr/local/bin/python3"
|
|
)
|
|
assert os.readlink(tmp_path / "uv-project" / ".venv" / "bin" / "python") == "python3"
|
|
|
|
|
|
def test_safe_tar_member_rel_path_requires_symlink_opt_in() -> None:
|
|
symlink = _symlink("link.txt", "target.txt").info
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="symlink member not allowed"):
|
|
safe_tar_member_rel_path(symlink)
|
|
|
|
assert safe_tar_member_rel_path(symlink, allow_symlinks=True) == Path("link.txt")
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_root_symlink() -> None:
|
|
raw = _tar_bytes(_symlink(".", "/tmp/outside"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="archive root symlink"):
|
|
validate_tar_bytes(raw)
|
|
|
|
|
|
@pytest.mark.parametrize("member_name", ["C:/tmp/evil.txt", r"C:\tmp\evil.txt"])
|
|
def test_validate_tar_bytes_rejects_windows_drive_member_paths(member_name: str) -> None:
|
|
raw = _tar_bytes(_file(member_name, b"evil"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="windows drive path"):
|
|
validate_tar_bytes(raw)
|
|
|
|
|
|
@pytest.mark.parametrize("member_name", [r"..\evil.txt", r"\evil.txt", r"nested\evil.txt"])
|
|
def test_validate_tar_bytes_rejects_windows_separator_member_paths(member_name: str) -> None:
|
|
raw = _tar_bytes(_file(member_name, b"evil"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="windows path separator"):
|
|
validate_tar_bytes(raw)
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_member_under_non_directory_member() -> None:
|
|
raw = _tar_bytes(
|
|
_file("nested/hello.txt", b"hello"),
|
|
_file("nested", b"not a directory"),
|
|
)
|
|
|
|
with pytest.raises(
|
|
UnsafeTarMemberError,
|
|
match="archive path descends through non-directory: nested",
|
|
):
|
|
validate_tar_bytes(raw)
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_absolute_symlink_target_in_strict_mode() -> None:
|
|
raw = _tar_bytes(_symlink("leak", "/etc/passwd"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="absolute symlink target not allowed"):
|
|
validate_tar_bytes(raw, allow_external_symlink_targets=False)
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_parent_escape_symlink_target_in_strict_mode() -> None:
|
|
raw = _tar_bytes(_dir("nested"), _symlink("nested/leak", "../../etc/passwd"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="symlink target escapes archive root"):
|
|
validate_tar_bytes(raw, allow_external_symlink_targets=False)
|
|
|
|
|
|
def test_validate_tar_bytes_allows_internal_symlink_target_in_strict_mode() -> None:
|
|
raw = _tar_bytes(_dir("nested"), _symlink("nested/python", "../bin/python3"))
|
|
|
|
validate_tar_bytes(raw, allow_external_symlink_targets=False)
|
|
|
|
|
|
def test_strip_tar_member_prefix_returns_workspace_relative_archive() -> None:
|
|
raw = _tar_bytes(
|
|
_dir("workspace"),
|
|
_dir("workspace/pkg"),
|
|
_file("workspace/pkg/main.py", b"print('hello')\n"),
|
|
_symlink("workspace/pkg/python", "python3"),
|
|
)
|
|
|
|
normalized = strip_tar_member_prefix(io.BytesIO(raw), prefix="workspace")
|
|
|
|
with tarfile.open(fileobj=normalized, mode="r:*") as tar:
|
|
assert tar.getnames() == [".", "pkg", "pkg/main.py", "pkg/python"]
|
|
|
|
|
|
@pytest.mark.parametrize("absolute_link", [False, True])
|
|
def test_strip_tar_member_prefix_retains_only_one_archive_payload(
|
|
monkeypatch: pytest.MonkeyPatch, absolute_link: bool
|
|
) -> None:
|
|
payload = b"workspace content\n" * 65536
|
|
entries = [_dir("workspace")]
|
|
if absolute_link:
|
|
# The target follows the link in the input stream.
|
|
entries.append(_symlink("workspace/link", "/workspace/data.txt"))
|
|
entries.append(_file("workspace/data.txt", payload))
|
|
raw = _tar_bytes(*entries)
|
|
storage: list[io.BytesIO] = []
|
|
|
|
class BudgetedArchive(io.BytesIO):
|
|
def write(self, data: Buffer) -> int:
|
|
written = super().write(data)
|
|
retained = sum(len(stream.getbuffer()) for stream in storage if not stream.closed)
|
|
if retained > len(raw) + tarfile.RECORDSIZE:
|
|
raise OSError(errno.ENOSPC, "archive storage budget exceeded")
|
|
return written
|
|
|
|
def temporary_file() -> io.BytesIO:
|
|
stream = BudgetedArchive()
|
|
storage.append(stream)
|
|
return stream
|
|
|
|
monkeypatch.setattr(tar_utils.tempfile, "TemporaryFile", temporary_file)
|
|
source = io.BytesIO(raw)
|
|
with strip_tar_member_prefix(
|
|
source, prefix="workspace", relativize_symlinks_under="/workspace"
|
|
) as normalized:
|
|
assert source.closed
|
|
with tarfile.open(fileobj=normalized, mode="r:*") as archive:
|
|
validate_tarfile(archive, allow_external_symlink_targets=False)
|
|
restored = archive.extractfile("data.txt")
|
|
assert restored is not None
|
|
with restored:
|
|
assert restored.read() == payload
|
|
if absolute_link:
|
|
assert archive.getmember("link").linkname == "data.txt"
|
|
assert all(stream.closed for stream in storage)
|
|
|
|
|
|
@pytest.mark.parametrize("failure", ["read", "write", "validation"])
|
|
def test_strip_tar_member_prefix_closes_streams_on_failure(
|
|
monkeypatch: pytest.MonkeyPatch, failure: str
|
|
) -> None:
|
|
entries = [_dir("workspace"), _file("workspace/data.txt")]
|
|
if failure == "validation":
|
|
entries.append(_file("workspace/data.txt", b"duplicate"))
|
|
|
|
class Source(io.BytesIO):
|
|
def read(self, size: int | None = -1) -> bytes:
|
|
if failure == "read":
|
|
raise OSError("source read failed")
|
|
return super().read(size)
|
|
|
|
class Output(io.BytesIO):
|
|
def write(self, data: Buffer) -> int:
|
|
if failure == "write":
|
|
raise OSError("archive write failed")
|
|
return super().write(data)
|
|
|
|
source = Source(_tar_bytes(*entries))
|
|
outputs: list[Output] = []
|
|
|
|
def temporary_file() -> Output:
|
|
output = Output()
|
|
outputs.append(output)
|
|
return output
|
|
|
|
monkeypatch.setattr(tar_utils.tempfile, "TemporaryFile", temporary_file)
|
|
error = UnsafeTarMemberError if failure == "validation" else OSError
|
|
message = "duplicate archive path" if failure == "validation" else f"{failure} failed"
|
|
with pytest.raises(error, match=message):
|
|
strip_tar_member_prefix(source, prefix="workspace", relativize_symlinks_under="/workspace")
|
|
assert source.closed
|
|
assert outputs and all(output.closed for output in outputs)
|
|
|
|
|
|
def _prefixed_workspace_archive(
|
|
*, external_symlink: bool, link_through_alias: bool = True
|
|
) -> io.BytesIO:
|
|
"""A `workspace/...` archive shaped like Docker's staged copy, with members that the
|
|
strict hydrate extractor refuses as-is."""
|
|
|
|
def add_dir(tar: tarfile.TarFile, name: str) -> None:
|
|
info = tarfile.TarInfo(name)
|
|
info.type = tarfile.DIRTYPE
|
|
tar.addfile(info)
|
|
|
|
def add_file(tar: tarfile.TarFile, name: str, payload: bytes) -> None:
|
|
info = tarfile.TarInfo(name)
|
|
info.size = len(payload)
|
|
tar.addfile(info, io.BytesIO(payload))
|
|
|
|
def add_symlink(tar: tarfile.TarFile, name: str, target: str) -> None:
|
|
info = tarfile.TarInfo(name)
|
|
info.type = tarfile.SYMTYPE
|
|
info.linkname = target
|
|
tar.addfile(info)
|
|
|
|
buf = io.BytesIO()
|
|
with tarfile.open(fileobj=buf, mode="w") as tar:
|
|
add_dir(tar, "workspace")
|
|
add_dir(tar, "workspace/sub")
|
|
add_dir(tar, "workspace/sub/deep")
|
|
add_file(tar, "workspace/a.txt", b"shared")
|
|
add_file(tar, "workspace/data.txt", b"wrong")
|
|
add_file(tar, "workspace/sub/data.txt", b"right")
|
|
fifo = tarfile.TarInfo("workspace/dev.fifo")
|
|
fifo.type = tarfile.FIFOTYPE
|
|
tar.addfile(fifo)
|
|
add_symlink(tar, "workspace/sub/abs_up", "/workspace/a.txt")
|
|
add_symlink(tar, "workspace/rel", "a.txt")
|
|
add_symlink(tar, "workspace/double_slash", "//workspace/a.txt")
|
|
add_symlink(tar, "workspace/double_sep", "/workspace//a.txt")
|
|
add_symlink(tar, "workspace/alias", "sub/deep")
|
|
if link_through_alias:
|
|
# `alias/..` resolves against the alias target (sub/deep): where this lands
|
|
# depends on another symlink, so the rewrite must leave it absolute.
|
|
add_symlink(tar, "workspace/abs_alias", "/workspace/alias/../data.txt")
|
|
# Longer than the 100-byte ustar field, so tarfile records it in a PAX linkpath.
|
|
nested = "workspace"
|
|
for _ in range(5):
|
|
nested += "/deeply-nested-directory"
|
|
add_dir(tar, nested)
|
|
add_file(tar, nested + "/target.txt", b"deep")
|
|
long_target = "/workspace/" + "/".join(["deeply-nested-directory"] * 5) + "/target.txt"
|
|
add_symlink(tar, "workspace/long_link", long_target)
|
|
if external_symlink:
|
|
add_symlink(tar, "workspace/outside", "/usr/bin/python3")
|
|
buf.seek(0)
|
|
return buf
|
|
|
|
|
|
def test_strip_tar_member_prefix_rewrites_members_hydrate_refuses() -> None:
|
|
stripped = strip_tar_member_prefix(
|
|
_prefixed_workspace_archive(external_symlink=True),
|
|
prefix="workspace",
|
|
relativize_symlinks_under="/workspace",
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
members = {member.name: member for member in tar.getmembers()}
|
|
assert "dev.fifo" not in members
|
|
assert members["sub/abs_up"].issym()
|
|
assert members["sub/abs_up"].linkname == "../a.txt"
|
|
assert members["rel"].linkname == "a.txt"
|
|
assert members["double_slash"].linkname == "a.txt"
|
|
assert members["double_sep"].linkname == "a.txt"
|
|
# Components after the root prefix are kept verbatim; `..` is not collapsed.
|
|
# Depends on how `alias` resolves: left absolute for strict hydration to refuse.
|
|
assert members["abs_alias"].linkname == "/workspace/alias/../data.txt"
|
|
long_link = members["long_link"]
|
|
assert long_link.linkname == "/".join(["deeply-nested-directory"] * 5) + "/target.txt"
|
|
assert "linkpath" not in long_link.pax_headers or (
|
|
long_link.pax_headers["linkpath"] == long_link.linkname
|
|
)
|
|
# External absolute targets are left for hydrate's policy to decide.
|
|
assert members["outside"].linkname == "/usr/bin/python3"
|
|
|
|
|
|
def test_strip_tar_member_prefix_output_passes_strict_hydrate_validation(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
stripped = strip_tar_member_prefix(
|
|
_prefixed_workspace_archive(external_symlink=False, link_through_alias=False),
|
|
prefix="workspace",
|
|
relativize_symlinks_under=PurePosixPath("/workspace"),
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
validate_tarfile(tar, allow_external_symlink_targets=False)
|
|
safe_extract_tarfile(tar, root=tmp_path, allow_external_symlink_targets=False)
|
|
|
|
# Inspect the restored link metadata rather than reading through the links: the
|
|
# targets are POSIX paths that only a POSIX host resolves the way the sandbox does.
|
|
assert os.readlink(tmp_path / "sub" / "abs_up") == "../a.txt"
|
|
assert os.readlink(tmp_path / "alias") == "sub/deep"
|
|
assert (tmp_path / "sub" / "data.txt").read_bytes() == b"right"
|
|
assert not os.path.lexists(tmp_path / "dev.fifo")
|
|
|
|
|
|
def test_strip_tar_member_prefix_output_with_alias_link_is_refused_by_strict_hydrate() -> None:
|
|
"""The archive keeps `/workspace/alias/../data.txt` absolute, and the strict hydrate
|
|
validation is what refuses it: this rewrite never guesses through another link."""
|
|
stripped = strip_tar_member_prefix(
|
|
_prefixed_workspace_archive(external_symlink=False),
|
|
prefix="workspace",
|
|
relativize_symlinks_under=PurePosixPath("/workspace"),
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
assert tar.getmember("abs_alias").linkname == "/workspace/alias/../data.txt"
|
|
with pytest.raises(UnsafeTarMemberError, match="absolute symlink target not allowed"):
|
|
validate_tarfile(tar, allow_external_symlink_targets=False)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("members", "victim", "target"),
|
|
[
|
|
pytest.param(
|
|
(_dir("workspace/a"), _symlink("workspace/a/link", "..")),
|
|
"workspace/victim",
|
|
"/workspace/a/link/../tmp",
|
|
id="dot-dot after a link that climbs out",
|
|
),
|
|
pytest.param(
|
|
(_symlink("workspace/outside", "/usr"),),
|
|
"workspace/victim",
|
|
"/workspace/outside/../x",
|
|
id="dot-dot after an external link",
|
|
),
|
|
pytest.param(
|
|
(_file("workspace/secret", b"s"),),
|
|
"workspace/victim",
|
|
"/workspace/alias/../secret",
|
|
id="dot-dot through a component the archive does not create",
|
|
),
|
|
pytest.param(
|
|
(),
|
|
"workspace/victim",
|
|
"/workspace/missing.txt",
|
|
id="leaf the archive does not create",
|
|
),
|
|
pytest.param(
|
|
(_file("workspace/notes.txt", b"n"),),
|
|
"workspace/victim",
|
|
"/workspace/notes.txt/secret",
|
|
id="descends through a regular file",
|
|
),
|
|
pytest.param(
|
|
(_symlink("workspace/loop", "loop"),),
|
|
"workspace/victim",
|
|
"/workspace/loop/x",
|
|
id="descends through a symlink member",
|
|
),
|
|
pytest.param(
|
|
(_symlink("workspace/alias", "sub"), _dir("workspace/sub")),
|
|
"workspace/victim",
|
|
"/workspace/alias",
|
|
id="leaf is a symlink member",
|
|
),
|
|
pytest.param(
|
|
(_file("workspace/a.txt", b"a"),),
|
|
"workspace/victim",
|
|
"/workspace/a.txt/",
|
|
id="trailing separator after a regular file (ENOTDIR on the source)",
|
|
),
|
|
pytest.param(
|
|
(_dir("workspace/sub"),),
|
|
"workspace/victim",
|
|
"/workspace/sub/",
|
|
id="trailing separator after a directory",
|
|
),
|
|
pytest.param(
|
|
(_dir("workspace/sub"),),
|
|
"workspace/victim",
|
|
"/workspace/sub",
|
|
id="directory target",
|
|
),
|
|
pytest.param((), "workspace/victim", "/workspace", id="workspace root"),
|
|
pytest.param((), "workspace/victim", "/workspace/", id="workspace root with separator"),
|
|
pytest.param(
|
|
(_file("workspace/implied/deep/data.txt", b"d"),),
|
|
"workspace/victim",
|
|
"/workspace/implied/deep/data.txt",
|
|
id="directories only implied by a file path",
|
|
),
|
|
pytest.param(
|
|
(
|
|
_dir("workspace/a"),
|
|
_dir("workspace/sub"),
|
|
_file("workspace/data.txt", b"root"),
|
|
_symlink("workspace/a/link2", "../sub"),
|
|
_symlink("workspace/b", "a/link2"),
|
|
),
|
|
"workspace/sub/victim",
|
|
"/workspace/b/../data.txt",
|
|
id="would resolve inside, but only by interpreting two links",
|
|
),
|
|
],
|
|
)
|
|
def test_strip_tar_member_prefix_leaves_non_simple_targets_absolute(
|
|
members: tuple[bytes, ...], victim: str, target: str
|
|
) -> None:
|
|
"""Targets whose destination depends on another symlink, or that walk through a path
|
|
the archive does not establish as an ordinary directory, are not rewritten. They stay
|
|
absolute so the strict hydrate validation refuses them instead of a rewrite guessing."""
|
|
raw = _tar_bytes(_dir("workspace"), *members, _symlink(victim, target))
|
|
|
|
stripped = strip_tar_member_prefix(
|
|
io.BytesIO(raw), prefix="workspace", relativize_symlinks_under="/workspace"
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
name = victim.removeprefix("workspace/")
|
|
assert tar.getmember(name).linkname == target
|
|
with pytest.raises(UnsafeTarMemberError, match="absolute symlink target not allowed"):
|
|
validate_tarfile(tar, allow_external_symlink_targets=False)
|
|
|
|
|
|
def test_strip_tar_member_prefix_rebases_simple_targets_established_by_the_archive() -> None:
|
|
"""Every directory the relative target walks through is an explicit directory member
|
|
and the leaf is a regular file, so the rewrite is exact."""
|
|
raw = _tar_bytes(
|
|
_dir("workspace"),
|
|
_dir("workspace/sub"),
|
|
_dir("workspace/sub/deep"),
|
|
_file("workspace/sub/deep/data.txt", b"d"),
|
|
_dir("workspace/other"),
|
|
_symlink("workspace/other/victim", "/workspace/sub/deep/data.txt"),
|
|
)
|
|
|
|
stripped = strip_tar_member_prefix(
|
|
io.BytesIO(raw), prefix="workspace", relativize_symlinks_under="/workspace"
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
assert tar.getmember("other/victim").linkname == "../sub/deep/data.txt"
|
|
validate_tarfile(tar, allow_external_symlink_targets=False)
|
|
|
|
|
|
def test_strip_tar_member_prefix_leaves_link_under_unestablished_parent_absolute() -> None:
|
|
"""The link's own parent directory is only implied, so the `..` climb cannot be trusted
|
|
either: hydration could find a symlink there in the destination."""
|
|
raw = _tar_bytes(
|
|
_dir("workspace"),
|
|
_file("workspace/data.txt", b"d"),
|
|
_symlink("workspace/implied/victim", "/workspace/data.txt"),
|
|
)
|
|
|
|
stripped = strip_tar_member_prefix(
|
|
io.BytesIO(raw), prefix="workspace", relativize_symlinks_under="/workspace"
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
assert tar.getmember("implied/victim").linkname == "/workspace/data.txt"
|
|
|
|
|
|
def test_strip_tar_member_prefix_keeps_absolute_symlinks_without_a_root() -> None:
|
|
stripped = strip_tar_member_prefix(
|
|
_prefixed_workspace_archive(external_symlink=False), prefix="workspace"
|
|
)
|
|
|
|
with tarfile.open(fileobj=stripped, mode="r:*") as tar:
|
|
assert tar.getmember("sub/abs_up").linkname == "/workspace/a.txt"
|
|
|
|
|
|
def test_strip_tar_member_prefix_still_rejects_hardlink_members() -> None:
|
|
raw = _tar_bytes(
|
|
_dir("workspace"),
|
|
_file("workspace/a.txt", b"x"),
|
|
_hardlink("workspace/b.txt", "workspace/a.txt"),
|
|
)
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="hardlink member not allowed"):
|
|
strip_tar_member_prefix(io.BytesIO(raw), prefix="workspace")
|
|
|
|
|
|
def test_strip_tar_member_prefix_rewrites_pax_path_headers() -> None:
|
|
long_name = "workspace/" + ("a" * 120) + ".txt"
|
|
payload = b"payload"
|
|
raw = io.BytesIO()
|
|
with tarfile.open(fileobj=raw, mode="w", format=tarfile.PAX_FORMAT) as tar:
|
|
member = tarfile.TarInfo(long_name)
|
|
member.size = len(payload)
|
|
tar.addfile(member, io.BytesIO(payload))
|
|
raw.seek(0)
|
|
|
|
normalized = strip_tar_member_prefix(raw, prefix="workspace")
|
|
|
|
with tarfile.open(fileobj=normalized, mode="r:*") as tar:
|
|
[member] = tar.getmembers()
|
|
assert member.name == ("a" * 120) + ".txt"
|
|
assert member.pax_headers["path"] == ("a" * 120) + ".txt"
|
|
|
|
|
|
def test_safe_extract_tarfile_can_rehydrate_existing_leaf_symlink(tmp_path: Path) -> None:
|
|
raw = _tar_bytes(_symlink("link.txt", "/usr/local/bin/python3"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert os.readlink(tmp_path / "link.txt") == "/usr/local/bin/python3"
|
|
|
|
raw = _tar_bytes(_symlink("link.txt", "target-v2.txt"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert os.readlink(tmp_path / "link.txt") == "target-v2.txt"
|
|
|
|
|
|
def test_safe_extract_tarfile_rejects_external_symlink_target_in_strict_mode(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
raw = _tar_bytes(_symlink("link.txt", "/etc/passwd"))
|
|
|
|
with tarfile.open(fileobj=io.BytesIO(raw), mode="r:*") as tar:
|
|
with pytest.raises(UnsafeTarMemberError, match="absolute symlink target not allowed"):
|
|
safe_extract_tarfile(
|
|
tar,
|
|
root=tmp_path,
|
|
allow_external_symlink_targets=False,
|
|
)
|
|
|
|
|
|
def test_safe_extract_tarfile_can_replace_existing_leaf_file_with_symlink(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
raw = _tar_bytes(_file("link.txt", b"not a link"))
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
raw = _tar_bytes(_symlink("link.txt", "target.txt"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert os.readlink(tmp_path / "link.txt") == "target.txt"
|
|
|
|
|
|
def test_safe_extract_tarfile_can_replace_existing_leaf_symlink_with_file(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
raw = _tar_bytes(_symlink("python", "/usr/local/bin/python3"))
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
raw = _tar_bytes(_file("python", b"real file"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert (tmp_path / "python").read_bytes() == b"real file"
|
|
assert not (tmp_path / "python").is_symlink()
|
|
|
|
|
|
def test_safe_extract_tarfile_can_replace_existing_leaf_symlink_with_directory(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
raw = _tar_bytes(_symlink("bin", "/usr/local/bin"))
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
raw = _tar_bytes(_dir("bin"), _file("bin/python", b"real file"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert (tmp_path / "bin").is_dir()
|
|
assert not (tmp_path / "bin").is_symlink()
|
|
assert (tmp_path / "bin" / "python").read_bytes() == b"real file"
|
|
|
|
|
|
def test_safe_extract_tarfile_can_replace_existing_leaf_file_with_directory(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
raw = _tar_bytes(_file("bin", b"not a directory"))
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
raw = _tar_bytes(_dir("bin"), _file("bin/python", b"real file"))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
assert (tmp_path / "bin").is_dir()
|
|
assert (tmp_path / "bin" / "python").read_bytes() == b"real file"
|
|
|
|
|
|
def test_safe_extract_tarfile_rejects_existing_leaf_directory_for_symlink(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
(tmp_path / "link.txt").mkdir()
|
|
raw = _tar_bytes(_symlink("link.txt", "target.txt"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="destination directory already exists"):
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_members_under_archive_symlink() -> None:
|
|
raw = _tar_bytes(
|
|
_symlink("escape", "/tmp/outside"),
|
|
_file("escape/pwned.txt", b"pwned"),
|
|
)
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="descends through symlink"):
|
|
validate_tar_bytes(raw)
|
|
|
|
|
|
def test_validate_tar_bytes_can_reject_specific_symlink_path() -> None:
|
|
raw = _tar_bytes(_symlink("workspace", "/tmp/outside"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="symlink member not allowed: workspace"):
|
|
validate_tar_bytes(raw, reject_symlink_rel_paths={Path("workspace")})
|
|
|
|
|
|
def test_validate_tar_bytes_specific_symlink_rejection_normalizes_dot_prefix() -> None:
|
|
raw = _tar_bytes(_symlink("./workspace", "/tmp/outside"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="symlink member not allowed: workspace"):
|
|
validate_tar_bytes(raw, reject_symlink_rel_paths={"workspace"})
|
|
|
|
|
|
def test_validate_tar_bytes_specific_symlink_rejection_does_not_reject_children() -> None:
|
|
validate_tar_bytes(
|
|
_tar_bytes(_dir("workspace"), _symlink("workspace/link", "/tmp/outside")),
|
|
reject_symlink_rel_paths={"workspace"},
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"member",
|
|
[
|
|
_file("remote/data.txt"),
|
|
_symlink("remote/link", "../outside"),
|
|
_file("remote"),
|
|
],
|
|
)
|
|
def test_validate_tar_bytes_rejects_members_overlapping_protected_path(
|
|
member: _Member,
|
|
) -> None:
|
|
raw = _tar_bytes(member)
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="overlaps protected path: remote"):
|
|
validate_tar_bytes(raw, reject_rel_paths={"remote"})
|
|
|
|
|
|
def test_validate_tar_bytes_rejects_non_directory_ancestor_of_protected_path() -> None:
|
|
raw = _tar_bytes(_file("remote"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="overlaps protected path: remote/nested"):
|
|
validate_tar_bytes(raw, reject_rel_paths={"remote/nested"})
|
|
|
|
|
|
def test_validate_tar_bytes_allows_directory_ancestor_of_protected_path() -> None:
|
|
raw = _tar_bytes(_dir("remote"))
|
|
|
|
validate_tar_bytes(raw, reject_rel_paths={"remote/nested"})
|
|
|
|
|
|
def test_safe_extract_tarfile_rejects_preexisting_symlink_parent(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
outside = tmp_path / "outside"
|
|
outside.mkdir()
|
|
root = tmp_path / "root"
|
|
root.mkdir()
|
|
os.symlink(outside, root / "escape", target_is_directory=True)
|
|
raw = _tar_bytes(_file("escape/pwned.txt", b"pwned"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="path escapes root|symlink in parent path"):
|
|
_safe_extract(raw, root)
|
|
|
|
assert not (outside / "pwned.txt").exists()
|
|
|
|
|
|
def test_safe_extract_tarfile_rejects_symlink_under_preexisting_symlink_parent(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
outside = tmp_path / "outside"
|
|
outside.mkdir()
|
|
root = tmp_path / "root"
|
|
root.mkdir()
|
|
os.symlink(outside, root / "escape", target_is_directory=True)
|
|
raw = _tar_bytes(_symlink("escape/nested/link.txt", "target.txt"))
|
|
|
|
with pytest.raises(UnsafeTarMemberError, match="path escapes root|symlink in parent path"):
|
|
_safe_extract(raw, root)
|
|
|
|
assert not (outside / "nested").exists()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"member",
|
|
[
|
|
_hardlink("hardlink", "target.txt"),
|
|
_fifo("pipe"),
|
|
],
|
|
)
|
|
def test_validate_tar_bytes_rejects_unsupported_tar_member_types(
|
|
member: _Member,
|
|
) -> None:
|
|
with pytest.raises(UnsafeTarMemberError):
|
|
validate_tar_bytes(_tar_bytes(member))
|
|
|
|
|
|
def test_validate_tar_bytes_ignores_skipped_unsafe_member() -> None:
|
|
validate_tar_bytes(
|
|
_tar_bytes(_symlink(".runtime/escape", "/tmp/outside")),
|
|
skip_rel_paths=[Path(".runtime")],
|
|
)
|
|
|
|
|
|
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX file modes are Unix-specific")
|
|
@pytest.mark.parametrize(
|
|
("archived_mode", "expected_mode"),
|
|
[
|
|
pytest.param(0o755, 0o755, id="executable-script"),
|
|
pytest.param(0o644, 0o644, id="plain-file"),
|
|
pytest.param(0o600, 0o600, id="owner-only-file"),
|
|
pytest.param(0o700, 0o700, id="owner-only-executable"),
|
|
pytest.param(0o444, 0o644, id="read-only-file-stays-owner-writable"),
|
|
pytest.param(0o000, 0o600, id="unreadable-file-stays-owner-readable"),
|
|
pytest.param(0o777, 0o755, id="group-and-other-write-dropped"),
|
|
pytest.param(0o655, 0o644, id="execute-without-owner-execute-dropped"),
|
|
pytest.param(0o4755, 0o755, id="setuid-dropped"),
|
|
pytest.param(0o2755, 0o755, id="setgid-dropped"),
|
|
pytest.param(0o1755, 0o755, id="sticky-dropped"),
|
|
],
|
|
)
|
|
def test_safe_extract_tarfile_restores_regular_file_modes(
|
|
tmp_path: Path,
|
|
archived_mode: int,
|
|
expected_mode: int,
|
|
) -> None:
|
|
raw = _tar_bytes(_file("run.sh", b"#!/bin/sh\n", mode=archived_mode))
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
assert stat.S_IMODE((tmp_path / "run.sh").stat().st_mode) == expected_mode
|
|
|
|
|
|
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX file modes are Unix-specific")
|
|
def test_safe_extract_tarfile_keeps_workspace_scripts_executable(tmp_path: Path) -> None:
|
|
raw = _tar_bytes(
|
|
_dir("."),
|
|
_dir("./bin"),
|
|
_file("./bin/start", b"#!/bin/sh\necho hi\n", mode=0o755),
|
|
_file("./README.md", b"# readme\n", mode=0o644),
|
|
)
|
|
|
|
_safe_extract(raw, tmp_path)
|
|
|
|
assert os.access(tmp_path / "bin" / "start", os.X_OK)
|
|
assert not os.access(tmp_path / "README.md", os.X_OK)
|
|
|
|
|
|
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX file modes are Unix-specific")
|
|
def test_safe_extract_tarfile_restores_mode_when_replacing_an_existing_file(
|
|
tmp_path: Path,
|
|
) -> None:
|
|
_safe_extract(_tar_bytes(_file("run.sh", b"v1\n", mode=0o644)), tmp_path)
|
|
assert stat.S_IMODE((tmp_path / "run.sh").stat().st_mode) == 0o644
|
|
|
|
_safe_extract(_tar_bytes(_file("run.sh", b"v2\n", mode=0o755)), tmp_path)
|
|
|
|
assert (tmp_path / "run.sh").read_bytes() == b"v2\n"
|
|
assert stat.S_IMODE((tmp_path / "run.sh").stat().st_mode) == 0o755
|
|
|
|
|
|
class _FailingPayload:
|
|
"""A member payload that yields one chunk and then fails, like a truncated read."""
|
|
|
|
def __init__(self, chunk: bytes) -> None:
|
|
self._chunk: bytes | None = chunk
|
|
|
|
def read(self, size: int = -1) -> bytes:
|
|
if self._chunk is None:
|
|
raise OSError("payload stream failed")
|
|
chunk, self._chunk = self._chunk, None
|
|
return chunk
|
|
|
|
def close(self) -> None:
|
|
return None
|
|
|
|
|
|
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX file modes are Unix-specific")
|
|
def test_safe_extract_tarfile_keeps_a_partially_written_file_private(
|
|
tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
raw = _tar_bytes(_file("run.sh", b"#!/bin/sh\necho hi\n", mode=0o755))
|
|
|
|
with tarfile.open(fileobj=io.BytesIO(raw), mode="r:*") as tar:
|
|
monkeypatch.setattr(tar, "extractfile", lambda member: _FailingPayload(b"#!/bin/sh\n"))
|
|
|
|
with pytest.raises(OSError, match="payload stream failed"):
|
|
safe_extract_tarfile(tar, root=tmp_path)
|
|
|
|
dest = tmp_path / "run.sh"
|
|
assert dest.read_bytes() == b"#!/bin/sh\n"
|
|
assert stat.S_IMODE(dest.stat().st_mode) == 0o600
|
|
assert not os.access(dest, os.X_OK)
|