openai-agents-python/tests
Kazuhiro Sera 6c4e4a29cd
docs: Upgrade the GPT-Live demo to 1.1 (#6101)
feat: Upgrade the GPT-Live demo to 1.1

Demonstrate speech-triggered standby with a 20-second idle timeout and hosted web search in managed Responses. Add a responsive conversation timeline, call controls, and optional event diagnostics.
2026-11-20 06:11:03 +00:00
..
deferred_session_writes fix(sessions): persist deferred approval history across resumes (#4828) 2026-10-06 11:27:50 -07:00
docs docs: skip reference pages on Windows (#4761) 2026-09-07 23:12:41 +09:00
extensions fix: support opt-in JSON validation for Codex tool parameters (#6067) 2026-11-18 00:49:17 +00:00
fastapi feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
fixtures release: 0.25.0 (#5394) 2026-11-18 00:13:54 +00:00
mcp fix: allow concurrent Streamable HTTP requests on MCP v2 (#5641) 2026-11-09 06:42:47 +00:00
memory fix: preserve pending async calls in committed sessions (#6039) 2026-11-17 14:07:44 +00:00
model_settings fix(core): serialize only traced fields in ModelSettings.to_traceable_dict (#5003) 2026-09-20 12:20:10 +09:00
models fix: align model defaults with TS SDK (#6064) 2026-11-17 21:40:15 +00:00
realtime fix(realtime): honor explicit interruption with automatic VAD (#5427) 2026-10-26 22:07:02 +00:00
sandbox fix: sandbox memory selection across UTC offsets (#6066) 2026-11-18 00:49:23 +00:00
tracing fix: trace offered tools on Chat Completions generation spans (#5635) 2026-11-09 06:05:37 +00:00
utils feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
voice fix: use gpt-transcribe as the default voice STT model (#5276) 2026-10-01 21:46:03 +00:00
__init__.py Initial commit 2025-03-11 09:42:28 -07:00
_fake_workspace_paths.py feat: support sandbox extra path grants (#2920) 2026-04-17 15:26:51 -07:00
conftest.py fix(sandbox): add opt-in bounded workspace outbox reads (#5128) 2026-09-25 14:51:15 -07:00
model_test_helpers.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
README.md fix: require complete policy for new public TypedDicts (#4900) 2026-09-07 15:53:24 +00:00
test_agent_as_tool.py fix: preserve structured tool input in agent lifecycle hooks (#6037) 2026-11-16 22:43:17 +00:00
test_agent_as_tool_callback_validation.py fix(core): validate agent tool stream callbacks before execution (#5125) 2026-09-21 10:43:34 +09:00
test_agent_as_tool_output_guardrails.py fix(core): preserve guarded final outputs in agent tools (#5115) 2026-09-21 07:49:11 +09:00
test_agent_clone_shallow_copy.py docs: correct Agent.clone list attribute semantics (#4474) 2026-08-18 05:43:47 +09:00
test_agent_config.py fix: await async callable-object dynamic instructions (#3942) 2026-07-24 22:01:19 +00:00
test_agent_hooks.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_agent_instructions_signature.py fix: await async callable-object dynamic instructions (#3942) 2026-07-24 22:01:19 +00:00
test_agent_llm_hooks.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_agent_memory_leak.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_agent_prompt.py fix: align model defaults with TS SDK (#6064) 2026-11-17 21:40:15 +00:00
test_agent_runner.py fix: retain tool guardrail results after approval and resume (#5643) 2026-11-09 06:48:22 +00:00
test_agent_runner_streamed.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_agent_runner_sync.py fix(run): replace closed default loop in run_sync (#4102) 2026-08-02 12:22:12 +09:00
test_agent_scoped_approvals.py fix(core): scope function tool approvals to their owning agent (#5144) 2026-09-25 14:21:25 -07:00
test_agent_tool_enablement_snapshot.py fix(core): preserve tool identity during asynchronous enablement (#5136) 2026-09-21 15:06:14 +09:00
test_agent_tool_input.py docs: sync tool docs and add coverage for internal helper branches (#2485) 2026-02-13 22:43:09 -08:00
test_agent_tool_state.py fix: isolate RunState checkpoint tool decisions (#4413) 2026-08-14 20:50:23 +09:00
test_agent_tracing.py fix: trace approved tools inside the resumed agent span (#5294) 2026-10-04 18:02:03 +00:00
test_agents_logging.py fix: reuse the verbose stdout logging handler (#3957) 2026-07-25 23:51:30 +00:00
test_apply_diff.py fix(core): honor End of File hunks when the file ends in a newline (#4661) 2026-08-26 11:03:38 +09:00
test_apply_diff_helpers.py fix(core): honor End of File hunks when the file ends in a newline (#4661) 2026-08-26 11:03:38 +09:00
test_apply_patch_tool.py fix: bind tool approvals to concrete invocations (#4257) 2026-08-07 11:59:01 +00:00
test_asyncio_progress.py test: make async progress and trace-worker tests deterministic (#4875) 2026-09-05 18:11:38 +09:00
test_asyncio_stability_command.py perf: reduce async stability command startup overhead (#4874) 2026-09-05 18:11:15 +09:00
test_asyncio_tasks.py fix: stop native tool queues during function failure cleanup (#5446) 2026-10-27 19:22:01 +00:00
test_call_model_input_filter.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_call_model_input_filter_unit.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_cancel_streaming.py fix: preserve streamed cancellation and pending guardrail cleanup (#5224) 2026-09-28 10:36:13 -07:00
test_change_detection.py fix: restore release tests and refresh required readiness check (#5278) 2026-10-01 20:42:14 -07:00
test_check_optional_truthiness.py Fix falsy optional reference handling (#4305) 2026-08-08 17:55:23 +09:00
test_code_change_verification_policy.py fix: keep Codex verification for development sandboxed (#4508) 2026-08-18 11:46:40 +00:00
test_code_change_verification_runner.py fix(deps-dev): bump ruff from 0.9.2 to 0.16.8 (#5249) 2026-09-30 22:24:21 +00:00
test_computer_action.py fix: warn about unhandled computer safety checks (#5305) 2026-10-06 20:45:36 +00:00
test_computer_tool_lifecycle.py fix(core): resolve tools after agent start hooks (#5124) 2026-09-21 18:01:24 +09:00
test_config.py feat: support OpenAI Python 3 and HTTPX2 (#4380) 2026-08-13 12:09:43 +09:00
test_custom_tool.py fix: #3359 preserve local approval rejection reasons (#3360) 2026-05-11 22:32:46 +00:00
test_dapr_session_example.py docs: secure local Dapr database provisioning (#5122) 2026-09-21 10:55:58 +09:00
test_daytona_usaspending_example.py chore: improve automated example coverage and local service handling (#3297) 2026-05-09 16:17:49 +09:00
test_debug.py Fix a bug where the default behavior of loading the debug flag differs from the documentation (#1682) 2025-09-08 10:05:25 -04:00
test_decisions_examples.py docs: add native Decisions API integration recipes (#5357) 2026-10-13 05:56:48 +00:00
test_decorators.py fix(tools): annotate the bare tool guardrail decorator overloads (#4238) 2026-08-06 00:11:53 +00:00
test_default_tool_error_redaction.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_disabled_trace_provider_flush.py fix(tracing): flush buffered data after tracing is disabled (#4528) 2026-08-20 06:12:09 +00:00
test_disabled_trace_provider_shutdown.py fix(tracing): clean up processors after tracing is disabled (#4502) 2026-08-18 06:48:35 +09:00
test_doc_parsing.py Initial commit 2025-03-11 09:42:28 -07:00
test_error_logging_redaction.py fix: preserve caller-owned exceptions during sandbox redaction (#5367) 2026-10-13 20:36:17 +00:00
test_example_workflows.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_exception_exports.py test: improve coverage for stream and sandbox edge cases (#3730) 2026-07-04 10:51:32 +09:00
test_extension_filters.py fix: flatten restored handoff history summaries (#6041) 2026-11-16 22:45:29 +00:00
test_extra_headers.py fix: support cache-write usage across OpenAI Python versions (#3773) 2026-07-10 07:14:43 +09:00
test_financial_research_budget.py docs: cap financial research at 15 planned searches (#5140) 2026-09-21 16:36:52 +09:00
test_function_schema.py fix: Preserve parameter descriptions in Args-only docstrings (#5302) 2026-10-05 15:38:13 +00:00
test_function_schema_logger_restore.py fix(core): preserve Griffe logger inheritance (#4494) 2026-08-18 06:08:50 +09:00
test_function_tool.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_function_tool_approval_arguments.py fix(core): align conditional approvals with validated tool arguments (#5066) 2026-09-17 20:56:48 +09:00
test_function_tool_decorator.py feat: support current image generation tool options (#4932) 2026-09-09 10:26:52 +09:00
test_global_hooks.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_guardrails.py fix: report completed output guardrails when a tripwire fires (#6040) 2026-11-16 22:44:56 +00:00
test_handoff_failure_error_function.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_handoff_history_duplication.py fix: flatten restored handoff history summaries (#6041) 2026-11-16 22:45:29 +00:00
test_handoff_prompt.py feat: #636 Add human-in-the-loop (HITL) support (#2230) 2026-01-29 12:16:28 +09:00
test_handoff_tool.py fix: resolve agent tool name collisions consistently (#4137) 2026-08-03 04:54:49 +00:00
test_hitl_error_scenarios.py fix(deps-dev): bump ruff from 0.9.2 to 0.16.8 (#5249) 2026-09-30 22:24:21 +00:00
test_hitl_session_scenario.py fix: preserve wire aliases when reading Conversations session history (#6047) 2026-11-16 22:47:17 +00:00
test_hitl_utils.py feat: #636 Add human-in-the-loop (HITL) support (#2230) 2026-01-29 12:16:28 +09:00
test_hosted_mcp_connectors_example.py fix: summarize verbose connector output (#5203) 2026-09-27 14:09:09 -07:00
test_human_in_the_loop_server_example.py docs: demonstrate server-owned tool approval state (#5105) 2026-09-20 17:04:09 +09:00
test_image_generation_tool_reference.py feat: support current image generation tool options (#4932) 2026-09-09 10:26:52 +09:00
test_image_generation_tool_typing.py feat: support current image generation tool options (#4932) 2026-09-09 10:26:52 +09:00
test_integration_runner.py test: restrict default runs to OpenAI credentials (#5277) 2026-10-02 05:59:08 +00:00
test_invalid_final_output_handler.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_items_helpers.py fix: preserve wire aliases when replaying Responses items (#5638) 2026-11-09 06:33:28 +00:00
test_live_example.py docs: Upgrade the GPT-Live demo to 1.1 (#6101) 2026-11-20 06:11:03 +00:00
test_local_shell_tool.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_logprobs.py fix: support cache-write usage across OpenAI Python versions (#3773) 2026-07-10 07:14:43 +09:00
test_max_turns.py fix(core): redact blocked tool outputs from replay state (#4507) 2026-08-19 15:39:24 +09:00
test_mcp_tool_metadata.py test: add direct unit tests for _mcp_tool_metadata helpers (#3102) 2026-05-04 18:00:20 +09:00
test_model_error_logging_redaction.py fix(tracing): respect model-data logging redaction for record_model_error_on_span (#4496) 2026-08-18 06:20:08 +09:00
test_native_tool_failure_history.py fix: stop native tool queues during function failure cleanup (#5446) 2026-10-27 19:22:01 +00:00
test_oaiconv_resume_response_id.py fix(run): preserve last known response_id on conversation resume (#3245) 2026-05-09 02:42:05 +09:00
test_openai_provider_client_options.py fix(core/voice): honor OpenAI provider options over default clients (#4530) 2026-08-20 02:55:37 +00:00
test_output_guardrail_cancellation.py fix: await cancelled output guardrail tasks on tripwire (#3187) 2026-05-09 11:43:15 +09:00
test_output_tool.py fix(handoffs): enforce strict Pydantic validation when strict_json_schema=True (#3724) 2026-07-02 15:42:43 +09:00
test_pretty_print.py fix: escape terminal controls in result diagnostics (#5219) 2026-09-28 09:26:35 -07:00
test_process_model_response.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_programmatic_tool_calling.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_prompt_cache_key.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_provider_span_errors.py feat(core): add model call timeouts (#4428) 2026-08-16 11:15:36 +09:00
test_redis_session_example.py fix(examples): keep Redis connection details out of diagnostics (#5205) 2026-09-27 14:08:42 -07:00
test_release_automation.py fix: restore release tests and refresh required readiness check (#5278) 2026-10-01 20:42:14 -07:00
test_release_please.py ci: automate release candidate preparation and review (#5230) 2026-09-28 22:04:26 -07:00
test_release_provenance.py fix(ci): harden PyPI publishing and require manual release tags (#4726) 2026-08-28 10:40:38 +09:00
test_released_api_contract.py fix: use gpt-6-luna for session compaction by default (#5646) 2026-11-09 07:06:14 +00:00
test_repl.py fix: tool approval handling in the demo REPL (#6045) 2026-11-16 22:46:19 +00:00
test_repository_workflow_interfaces.py ci: run PyPI publish checks once on Python 3.14 (#6070) 2026-11-18 17:55:00 +00:00
test_responses.py fix: bind tool approvals to concrete invocations (#4257) 2026-08-07 11:59:01 +00:00
test_responses_tracing.py fix(deps): bump openai from 3.29.0 to 3.48.0 (#6054) 2026-11-17 06:58:58 -08:00
test_result_cast.py fix: use last_agent property in RunResultStreaming._create_error_details (#3967) 2026-07-27 00:19:44 +00:00
test_resume_tool_use_behavior.py fix: preserve completed tool results when resuming approvals (#5240) 2026-09-29 11:29:20 -07:00
test_run.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_run_config.py fix: align model defaults with TS SDK (#6064) 2026-11-17 21:40:15 +00:00
test_run_context_approvals.py fix(core): scope function tool approvals to their owning agent (#5144) 2026-09-25 14:21:25 -07:00
test_run_context_wrapper.py fix: honor exact call approval decisions (#4447) 2026-08-16 08:40:44 +09:00
test_run_error_details.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_run_examples_script.py feat(sandbox): remove Runloop provider (#5369) 2026-10-19 17:14:49 +00:00
test_run_hooks.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_run_impl_resume_paths.py fix: preserve tool outputs after callback failures (#5318) 2026-10-07 04:19:46 +00:00
test_run_internal_approvals.py test: improve coverage and organize test layout (#3085) 2026-05-02 15:40:54 +09:00
test_run_internal_error_handlers.py feat: add invalid final output recovery handler (#3736) 2026-07-06 15:25:54 +09:00
test_run_internal_items.py fix(core): strip created_by when replaying RunItems as input (#4568) 2026-08-22 22:18:42 +00:00
test_run_serial_tests.py fix: propagate the serial test runner's exit code on Windows (#4771) 2026-09-07 23:41:01 +09:00
test_run_state.py fix(deps): bump openai from 3.29.0 to 3.48.0 (#6054) 2026-11-17 06:58:58 -08:00
test_run_state_agent_identity.py refactor: extract RunState agent graph identity (#4876) 2026-09-05 18:11:52 +09:00
test_run_state_aliases.py fix: preserve function-call aliases across RunState restoration (#5358) 2026-10-13 04:33:56 +00:00
test_run_state_compatibility_corpus.py fix: preserve caller-owned exceptions during sandbox redaction (#5367) 2026-10-13 20:36:17 +00:00
test_run_state_hook_identity.py fix: preserve RunState agent ownership across hook state changes (#5362) 2026-10-13 19:34:17 +00:00
test_run_state_pending_input.py fix: preserve pending input history across streamed retries (#6042) 2026-11-17 01:01:56 +00:00
test_run_state_pydantic_context.py fix: preserve Pydantic run context snapshots (#5444) 2026-10-27 17:34:02 +00:00
test_run_step_execution.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_run_step_processing.py Sandbox Agents (#2889) 2026-04-15 10:00:40 -07:00
test_runner_guardrail_resume.py fix: preserve tool outputs after callback failures (#5318) 2026-10-07 04:19:46 +00:00
test_runner_model_provider_lifecycle.py fix(core): close model providers created by Runner (#4785) 2026-09-08 00:13:33 +09:00
test_runner_tracing_disabled.py fix: honor Runner tracing opt-out inside caller traces (#5296) 2026-10-04 19:30:52 +00:00
test_runtime_symmetry_contract.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_scripted_model.py feat(core): add model call timeouts (#4428) 2026-08-16 11:15:36 +09:00
test_scripted_sandbox.py fix(sandbox): scope exclusive Add File creation to UnixLocal (#4893) 2026-09-27 17:24:03 -07:00
test_server_conversation_tracker.py fix(core): resolve tools after agent start hooks (#5124) 2026-09-21 18:01:24 +09:00
test_shell_call_serialization.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_shell_examples.py docs: require interactive approval for host shell commands (#5107) 2026-09-21 06:29:13 +09:00
test_shell_tool.py fix: preserve empty structured shell results (#5423) 2026-10-26 21:36:30 +00:00
test_soft_cancel.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_source_compat_constructors.py fix(run): preserve completed tool guardrail results (#4180) 2026-08-05 09:21:55 +09:00
test_start_hook_tool_concurrency.py fix(core): resolve tools after agent start hooks (#5124) 2026-09-21 18:01:24 +09:00
test_start_hook_tools.py fix(core): resolve tools after agent start hooks (#5124) 2026-09-21 18:01:24 +09:00
test_stream_events.py test: make stream event ordering deterministic 2026-08-19 07:31:35 +09:00
test_stream_input_guardrail_timing.py fix: skip streamed model attempts after an input guardrail trips (#5644) 2026-11-09 06:48:26 +00:00
test_streamed_terminal_output_backfill.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_streaming_logging.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_streaming_tool_call_arguments.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_strict_schema.py fix: reject unsafe strict-schema ref siblings (#4356) 2026-08-11 11:25:19 +09:00
test_strict_schema_oneof.py fix(schema): normalize typeless strict object schemas (#4139) 2026-08-03 04:36:25 +00:00
test_test_environment.py fix(deps-dev): bump ruff from 0.9.2 to 0.16.8 (#5249) 2026-09-30 22:24:21 +00:00
test_tool_approval_call_id_reuse.py perf: avoid repeatedly fingerprinting completed tool calls (#5421) 2026-10-26 21:37:07 +00:00
test_tool_batch_failure_history.py fix: preserve completed tool outputs when a sibling fails (#5332) 2026-10-14 00:53:44 +00:00
test_tool_choice_reset.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_tool_context.py feat: consistently accept typed objects and dictionaries for SDK configuration (#3917) 2026-07-24 08:04:45 +09:00
test_tool_converter.py feat: add Responses API tool search support (#2610) 2026-03-06 19:23:38 +09:00
test_tool_custom_data.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_tool_exception_formatter.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_tool_failure_boundaries.py fix: preserve completed tool outputs when a sibling fails (#5332) 2026-10-14 00:53:44 +00:00
test_tool_guardrails.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_tool_identity.py test: add direct unit tests for _tool_identity helpers (#3101) 2026-05-04 17:59:43 +09:00
test_tool_metadata.py fix: show the GA computer alias in tracing (#2641) 2026-03-09 15:22:25 +09:00
test_tool_name_collision_policy.py fix(mcp): bind resumed MCP calls to their original recipients (#5119) 2026-09-21 14:13:23 +09:00
test_tool_origin.py feat: add scripted model test utilities (#4362) 2026-08-13 11:20:33 +09:00
test_tool_output_conversion.py Fix falsy optional reference handling (#4305) 2026-08-08 17:55:23 +09:00
test_tool_use_behavior.py fix(core): validate tool_use_behavior callback results (#5173) 2026-09-25 09:45:34 -07:00
test_tool_use_tracker.py Fix falsy optional reference handling (#4305) 2026-08-08 17:55:23 +09:00
test_trace_processor.py perf: bound trace collection normalization work (#5207) 2026-09-27 14:25:42 -07:00
test_trace_redaction_example.py docs: demonstrate fail-closed trace redaction (#5109) 2026-09-20 17:30:38 +09:00
test_tracing.py docs(tracing): fix custom span output example (#4196) 2026-08-05 14:19:06 +09:00
test_tracing_errors.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_tracing_errors_streamed.py fix: remove retry advice from the default tool error message (#5632) 2026-11-09 05:40:00 +00:00
test_tracing_provider_safe_debug.py Harden tracing shutdown logging against closed handlers (#2270) 2026-01-08 13:41:38 +09:00
test_transforms.py fix: #2951 warn for tool name character replacement (#2953) 2026-04-19 02:18:10 +00:00
test_update_rclone_pin.py fix: pin and verify rclone sandbox installs (#3911) 2026-07-22 23:00:17 +09:00
test_usage.py fix(deps): bump openai from 3.29.0 to 3.48.0 (#6054) 2026-11-17 06:58:58 -08:00
test_usaspending_setup_db.py feat: improve examples auto-run coverage and artifact handling (#3328) 2026-05-10 14:25:47 +09:00
test_visualization.py fix(visualization): preserve nodes with duplicate names (#4512) 2026-08-19 23:07:35 +09:00
testing_processor.py test: order test spans by start sequence, not by started_at alone (#4392) 2026-08-14 09:31:42 +00:00

Tests

Use uv for test commands. Run make sync for a fresh checkout, changed dependencies, or a dependency-resolution failure; do not repeat setup before every check.

Running tests

For provider-neutral agent workflow tests, prefer ScriptedModel from agents.testing instead of adding a new mock or fake Model. Use ScriptedRealtimeModel from agents.realtime.testing for Realtime session tests, the scripted utilities from agents.voice.testing for Voice pipeline tests, and scripted_sandbox_session() from agents.testing for deterministic Sandbox session calls. Keep a specialized test double only when the test specifically requires provider-wire conversion, malformed streams, controlled suspension or concurrency, or an exact cancellation or lifecycle boundary that the scripted utilities cannot preserve; document that boundary in the test.

make tests

make tests runs the shard-safe suite first with pytest-xdist using up to nine workers, then runs the tests marked serial after all xdist workers have exited. Set PYTEST_XDIST_AUTO_NUM_WORKERS to a positive integer to override the automatic worker count and cap. The serial runner limits collection to test files containing the literal pytest.mark.serial, so keep that literal marker in every file containing serial tests. For indirect or custom serial marker spellings, use uv run pytest -m serial to perform generic pytest collection.

The serial marker means that a test needs exclusive execution after every xdist worker exits, not merely ordered execution within one worker. Use it for shared external resources, process-wide state, or timing-sensitive lifecycle tests that have demonstrated interference under xdist. Tests that use their own subprocess, random port, or temporary directory do not need serial solely for that reason; prove them under xdist instead.

make tests-review omits tests marked review_optional. These slow subsystem-specific integration, subprocess, or multiprocessing checks remain mandatory in final make tests verification. This broad subset is not a replacement for the focused checks used during implementation review or for the final suite.

During iterative implementation review, run focused tests for the changed behavior and relevant subsystem boundaries, including applicable review_optional cases without filtering them away. Resolve uncertain coverage by tracing affected callers and dependencies and selecting the needed focused checks. Follow implementation-final-review for review sequencing; defer broad make tests-review, make tests, and repository-wide type checking until clean review. Then follow code-change-verification for the complete final SDK stack. Explicit requests to run a suite outside an implementation review remain supported.

make typecheck runs mypy and pyright concurrently. Mypy checks src, while Pyright checks the src and tests paths configured in pyrightconfig.json. Pyright uses four analysis threads by default; set PYRIGHT_THREADS to a positive integer to override the local thread count.

Performance and determinism

Tests should wait for observable state transitions rather than elapsed wall-clock time. Preserve the behavior and lifecycle branches under test when removing waits; a faster test is not equivalent if it replaces an active state with a completed state or bypasses the production finalization path.

Use these guidelines when adding or changing tests:

  • Use events, deterministic fakes, immediate exceptions, and narrowly scoped mocks instead of real sleeps or retry backoff when elapsed time is not the behavior under test.
  • Keep a real timeout or delay only when its duration semantics are the contract being tested. Use the smallest focused value that distinguishes the expected behavior.
  • Preserve active, completed, failure, cancellation, and cleanup coverage as applicable. Release blocked tasks and clean up sessions, processes, and other resources in finally blocks so failed assertions cannot hang the suite.
  • Parameterize cases that share the same setup, execution path, and assertions. Give each case a descriptive ID, and keep separate tests when their lifecycle or failure invariants differ.
  • Capture expected warnings in the narrowest test with the specific warning category and a stable message match. Do not hide unrelated warnings with a global filter.
  • Preserve subprocess isolation when import state, registration, shutdown, or interpreter lifecycle is under test. Instrument the exact side effect, such as construction or registration, instead of scanning the heap or waiting for it to occur.
  • Run independent read-only subprocess or filesystem probes with bounded concurrency when useful. Keep cases that mutate shared fixtures, scripts, environment, ports, or external services sequential.
  • Keep parallel tests shard-safe: avoid shared mutable global state, fixed writable paths, order dependence, and uncoordinated external resources. Mark a test serial only when isolation cannot express its required behavior.
  • Keep timing and scheduler patches local to the test context, and continue exercising the production decision, retry, finalization, or cleanup path rather than replacing it wholesale.

Measure performance changes with both focused and broad runs:

uv run pytest tests/path/to/test_file.py --durations=10
make tests-parallel

Compare test counts, skips, warnings, assertions, and lifecycle coverage as well as elapsed time. Full-suite wall-clock results depend on host load and worker scheduling, so treat repeated focused measurements as the stronger evidence for an individual optimization. Run the repository's required verification stack after the final test changes.

Release compatibility unit tests must exercise policy and validation logic with explicit constructed modules instead of inspecting the current checkout's shared import state. The prospective release-contract job validates the current source checkout once in a dedicated Python process, and the packaged integration profiles validate real wheel, sdist, optional-extra, and platform surfaces in isolated environments. Keep the combined serial focused runtime of release compatibility unit tests below 1.5 seconds and each case below 100 milliseconds in normal conditions. Tests that build or install distributions, isolate imports in new interpreters, access the network, start containers, or require external services belong in integration_tests/ instead. Run the concrete local Docker-backed contracts with make integration-tests-containers; ordinary make tests must not pull container images. The released API manifest permits compatible suffix additions and records enum member construction from Enum.__new__ so CPython's version-dependent enum metaclass signature is not treated as an SDK change. Historical RunState fixtures must be produced by the recorded historical writer rather than by editing a current payload's schema version; the corpus README documents the explicit canonical-reader exception for schema versions that never had a corresponding writer.

The released API manifest is a rolling latest-release contract. After a release PR has updated pyproject.toml, check out the clean release branch locally and run make update-released-api-contract VERSION=<version> before requesting final review. The command first rejects any incompatibility with the committed contract, then freezes the candidate's current top-level exports, every inspectable top-level class or function signature and execution kind, and every newly exported SDK-owned inspectable class or function in a tracked public submodule. Qualified submodule callables remain tracked in later releases. Callable contracts include Pydantic model field names and defaults plus the binding, signature, and execution kind of each public callable member declared directly on an exported class or inherited from an SDK-owned base class. Exact Python functions are classified as synchronous functions, generators, coroutines, or asynchronous generators, and decorated functions use their caller-visible standard inspect.signature() contract. Methods declared only by third-party base classes and arbitrary descriptors remain outside the contract. Before regeneration, add newly documented class properties or factory-result properties to public_properties, selected public TypedDict fields to public_typed_dicts, newly intended cross-module import identities to canonical_imports, and optional module/export declarations to modules in tests/fixtures/released_api_contract_policy.json; those policy decisions are deliberately not inferred from implementation modules. The generator records each selected TypedDict field's requiredness and declared annotation without enrolling arbitrary TypedDict classes or fields. For a newly exported top-level TypedDict, add a public_typed_dicts entry for module agents that lists every field, including inherited fields. The existing prospective-release-contract CI job rejects missing entries or omitted fields on the implementation PR, before release preparation. Previously released exports retain their curated coverage; this check does not recursively discover nested types. It merges the curated policy into the prospective and released contracts and freezes declared unsupported platforms so validation remains portable. The v0.19.4 baseline includes base-package submodule paths, canonical identities, and documented result properties used by its shipped docs and examples; optional-extra and experimental paths remain outside this base contract. After rebasing the release branch, run make check-released-api-contract VERSION=<version> and regenerate only when it reports drift. No GitHub workflow imports candidate code with write credentials for this update.

Snapshots

We use inline-snapshots for some tests. If your code adds new snapshot tests or breaks existing ones, you can fix/create them. After fixing/creating snapshots, run make tests again to verify the tests pass.

Fixing snapshots

make snapshots-fix

Creating snapshots

make snapshots-create