diff --git a/.claude/skills/playwright-roll/SKILL.md b/.claude/skills/playwright-roll/SKILL.md index ece920353..9a6adbd03 100644 --- a/.claude/skills/playwright-roll/SKILL.md +++ b/.claude/skills/playwright-roll/SKILL.md @@ -1,23 +1,285 @@ --- name: playwright-roll -description: Roll Playwright Python to a new version +description: Roll Playwright Python to a new driver version. Walks the upstream `docs/src/api/` commit range, ports each public-API change, suppresses the rest in `expected_api_mismatch.txt`, regenerates the typed surface, and adds tests. --- -Help the user roll to a new version of Playwright. -../../../ROLLING.md contains general instructions and scripts. +# Rolling Playwright Python -Start with updating the version and generating the API to see the state of things. +The goal of a roll is to move `driver_version` in `setup.py` to a new release, port every public API change introduced upstream during that interval, and suppress the rest, so that `./scripts/update_api.sh` runs clean and the test suite still passes. -Afterwards, work through the list of changes that need to be backported. -You can find a list of pull requests that might need to be taking into account in the issue titled "Backport changes". -Work through them one-by-one and check off the items that you have handled. -Not all of them will be relevant, some might have partially been reverted, etc. - so feel free to check with the upstream release branch. +The previous human-facing summary lives in `../../../ROLLING.md`. This skill is the operational playbook — read it end to end before starting. -Rolling includes: -- updating client implementation to match changes in the upstream JS implementation (see ../playwright/packages/playwright-core/src/client) -- adding a couple of new tests to verify new/changed functionality +## Mental model -## Tips & Tricks -- Project checkouts are in the parent directory (`../`). -- when updating checkboxes, store the issue content into /tmp and edit it there, then update the issue based on the file -- use the "gh" cli to interact with GitHub +The Python port is hand-written code in `playwright/_impl/`, plus a generator (`scripts/generate_*.py`, `scripts/documentation_provider.py`) that: + +1. introspects the Python `_impl` classes via `inspect`, +2. emits typed wrapper classes into `playwright/{async,sync}_api/_generated.py`, and +3. diffs the introspected surface against `playwright/driver/package/api.json` (downloaded inside the new driver wheel). + +Anything in `api.json` that is missing or differently typed in `_impl/` causes generation to fail. Three resolutions: + +- **PORT** — the new API is intended for Python (no `langs.only` filter, or `langs.only` includes `"python"`). Implement it in `_impl/`. +- **MISMATCH** — the API genuinely exists for Python but is shaped differently (a callback signature uses unions, a kwarg uses a legacy name, etc.) and there's a justified reason to keep the divergence. Add a precise line to `scripts/expected_api_mismatch.txt` with a comment explaining *why*. +- **N/A** — the commit only touches docs, has `* langs: js` (or any other filter that excludes Python), is server-side, Electron-only, or was reverted later in the same release. No action. + +The upstream documentation source of truth is `docs/src/api/*.md` in the playwright repo. Every `## method:` / `## property:` / `## event:` / `### option:` / `### param:` block has an optional `* langs: js` (or `js, python`, etc.) filter. The Python doclint resolves these into `langs` fields on each member of `api.json`. **An empty `langs: {}` means "all languages including Python" — *implement it*, don't suppress it.** + +> **The mistake the 1.59 roll made twice over:** classifying things as "internal tooling, N/A for Python" based on the *name* of the API (Screencast, Debugger, pickLocator, clearConsoleMessages, artifactsDir, …). Almost all of those had empty `langs: {}` in `api.json` and were real Python APIs. Sounding tooling-y is not a `langs` filter. **The `langs` field on the member in `api.json` is the only authoritative signal.** When in doubt, dump it (see "Verifying classifications" below). + +## Pre-flight + +You will need two checkouts in the parent directory: +- `~/code/playwright-python` — this repo. +- `~/code/playwright` — the upstream playwright monorepo (used read-only for diffing). + +Bring upstream up to date and ensure release branches/tags are present: + +```sh +git -C ~/code/playwright fetch --tags +git -C ~/code/playwright fetch origin 'release-*:release-*' +``` + +There is sometimes no `vX.Y.0` tag for the latest release (the bots cut release branches first and tag later). Anchor on commits, not tags — see "Identify the commit range" below. + +## Process + +### 1. Set up the env + +`CONTRIBUTING.md` covers this. Notes from past rolls: + +- The repo says Python 3.9 is required, but 3.9+ works. If `python3.9` isn't available, use `python3` (3.12 is fine). +- If `python3-venv` is missing system-wide, use `uv venv env` instead, then `uv pip install --python env/bin/python --upgrade pip`. Don't try to `apt install` — sudo is denied in the harness. +- Always activate the venv before any `pip`, `pytest`, `mypy`, or `pre-commit` invocation. + +### 2. Bump the driver and download it + +```sh +# Edit setup.py +driver_version = "" # e.g. "1.59.1" + +source env/bin/activate +python -m build --wheel # downloads the new driver from cdn.playwright.dev +playwright install chromium # NOT --with-deps; sudo is denied +``` + +The wheel build prints `Fetching https://cdn.playwright.dev/builds/driver/playwright--linux.zip` and unpacks the driver under `playwright/driver/package/`. From this point, `playwright/driver/package/api.json` reflects the new release. + +### 3. Identify the commit range + +The diff range is "every commit on the new release branch since the previous release was cut". Anchor commits: + +- **Previous release end**: the `chore: bump version to vX.Y.0-next` commit on `main`. That commit is the first commit *after* the previous release (X.Y-1) was cut. Use its parent (`~1`) as the lower bound. + ```sh + git -C ~/code/playwright log --all --grep="bump version to v" --oneline | head + ``` +- **New release end**: the tip of `release-` (or the matching tag if it exists). + +Save the commit list, oldest first, scoped to `docs/src/api/`: + +```sh +git -C ~/code/playwright log ~1..release- --oneline --reverse -- docs/src/api > /tmp/roll--commits.md +``` + +A normal roll yields 50–100 commits. If you see 0 or thousands, the range is wrong. + +Format the file as a markdown checklist and add the standard preamble (status legend, where to look up `api.json` etc.) — see the file from the 1.58→1.59 roll for the template. + +### 4. Walk the commit list + +For each commit, in chronological order: + +```sh +git -C ~/code/playwright show -- docs/src/api/ +``` + +Look for: +- `## (async )?method:` / `## property:` / `## event:` additions or removals; +- `* langs: ...` lines on those blocks; +- `### param:` / `### option:` additions or removals; +- new `class-X.md` files (whole new classes — usually `langs: js`); +- type changes in `- returns:` lines. + +Classify and act. + +#### Verifying classifications (do this before suppressing anything) + +Before tagging anything as MISMATCH or N/A based on appearance, dump the actual `langs` from `api.json`: + +```python +import json +data = json.load(open("playwright/driver/package/api.json")) +classes = {c["name"]: c for c in data} +for cls_name in ["Page", "BrowserContext", "Screencast", "Debugger"]: + cls = classes.get(cls_name) + if not cls: + continue + print(f"\n{cls_name}: cls_langs={cls.get('langs', {})}") + for m in cls["members"]: + print(f" {m['name']} kind={m.get('kind')} langs={m.get('langs', {})}") +``` + +For options/params nested inside an `Object`-typed arg, walk one level deeper: + +```python +for a in member.get("args", []): + if a["name"] == "options": + for prop in a.get("type", {}).get("properties", []): + print(prop["name"], prop.get("langs", {})) +``` + +A few rules of thumb that catch most "actually a PORT" cases: +- If the *containing class* has empty `langs: {}` and the *member* has empty `langs: {}`, it's for Python — implement it. +- If the member is empty but a single *option* has `langs: js`, the method is for Python and you only skip that option (e.g. `Screencast.start.size` is `langs: js` while `Screencast.start` itself isn't). +- If you're about to add three or more `Method not implemented:` entries for the same class, stop — you almost certainly need to implement the class. + +#### PORT + +Implement the change in `playwright/_impl/.py`. Use the upstream JS implementation as a reference: `~/code/playwright/packages/playwright-core/src/client/.ts`. Translate idioms: + +| Upstream JS | Python | +|---|---| +| `async foo(): Promise` | `async def foo(self) -> X:` | +| `foo(): X` (sync getter, no args, no body) | `@property def foo(self) -> X:` (the doc generator treats argument-less sync getters as properties — see `documentation_provider.py:133`. If you make it a method instead, you'll get a "Method vs property mismatch" error.) | +| `await this._channel.foo({ a, b })` | `await self._channel.send("foo", None, locals_to_params(locals()))` | +| `(await this._channel.foo()).value` | `await self._channel.send("foo", None)` (Python's `send()` auto-unwraps single-key responses; only call `send_return_as_dict` when the protocol returns multiple keys.) | +| `(await this._channel.foo()).artifact` (multi-key, may be empty) | `result = await self._channel.send_return_as_dict("foo", None); (result or {}).get("artifact")` — `send_return_as_dict` returns **`None`** (not `{}`) when the protocol response carries no fields. | +| `try { ... } catch (e) { if (isTargetClosedError(e)) return; throw e; }` | `try: ...; except Exception as e: if is_target_closed_error(e): return; raise` (import from `playwright._impl._errors`) | +| Inline `[Object]` return like `{endpoint: string}` | A `TypedDict` in `playwright/_impl/_api_structures.py` — *not* `Dict[str, str]`. The doc generator serializes TypedDicts as `{field: type, ...}` via `get_type_hints` and that matches the inline-object form exactly. See `RemoteAddr`, `BrowserBindResult`, `DebuggerPausedDetails`. | +| `binary` event/return field | The Python channel layer hands you a base64 string. Decode with `base64.b64decode(value)` before exposing as `bytes`. See `Screencast._dispatch_frame`. | + +When implementing a new ChannelOwner subclass (one constructed by the protocol with `(parent, type, guid, initializer)`): +1. Register it in `playwright/_impl/_object_factory.py:create_remote_object` — otherwise the guid resolves to `DummyObject` and downstream code breaks mysteriously. +2. Import it and add it to `generated_types` in `scripts/generate_api.py`, plus add a `XxxImpl` import in the `header` string. + +When implementing a non-ChannelOwner wrapper class (a plain class that holds a Page/Context reference, like `Screencast`, `Clock`): +- Set `self._loop = parent._loop` and `self._dispatcher_fiber = parent._dispatcher_fiber` in `__init__`. The generated `AsyncBase`/`SyncBase` wrappers read these; missing them gives `AttributeError: 'X' object has no attribute '_loop'` at first use. + +When adding a new TypedDict in `_api_structures.py`: +- Add it to the `from playwright._impl._api_structures import …` line in `scripts/generate_api.py` so the generator can resolve it as a forward reference in type hints. +- Re-export it from both `playwright/async_api/__init__.py` and `playwright/sync_api/__init__.py`: assignment line plus an entry in `__all__`. Same pattern as `ViewportSize`, `RemoteAddr`. + +If the new API was previously suppressed in `expected_api_mismatch.txt`, **remove that line** when implementing it. + +If a doc rename involves a *positional* parameter (no default, before any `*`), users almost certainly call it positionally — you can rename freely. The 1.59 `BrowserType.connect.wsEndpoint` → `endpoint` is the canonical example. Don't suppress this kind of rename; just rename in `_impl/`. **Important corollary:** when docs rename a param, the wire-protocol field usually also changed in `packages/protocol/src/protocol.yml` and the server-side dispatcher in `packages/playwright-core/src/server/dispatchers/*Dispatcher.ts`. If so, you must also update the channel-send dict key (e.g. `{"wsEndpoint": …}` → `{"endpoint": …}`). A "Parameter not documented" suppression for a renamed param is a code smell hiding a wire-protocol bug. + +#### MISMATCH + +A MISMATCH is a *justified, durable* divergence between the docs and the Python surface. Use it sparingly — most apparent mismatches turn out to be PORTs you skipped. Legitimate examples in the current `expected_api_mismatch.txt`: + +- Hidden internal kwargs (`Browser.new_context(default_browser_type=)`). +- Callback signatures where Python explicitly unions one-arg and two-arg variants but the docs document only the canonical form (`Page.route(handler=)`, `WebSocketRoute.on_close(handler=)`). + +Add a precise line to `scripts/expected_api_mismatch.txt` with a `# comment` group header explaining *why* the divergence is intentional. The exact wording comes from the generator's error message. Examples: + +``` +# One vs two arguments in the callback, Python explicitly unions. +Parameter type mismatch in Page.route(handler=): documented as Callable[[Route, Request], Union[Any, Any]], code has Union[Callable[[Route, Request], Any], Callable[[Route], Any]] +``` + +The generator removes lines from `expected_api_mismatch.txt` that no longer match an error. If you see "No longer there: …" in the script's stderr, delete that line. + +**Do not suppress** these — they're PORTs in disguise: + +- "Internal tooling" classes/methods whose `langs` field is empty (`Screencast.*`, `Debugger.*`, `Page.pick_locator`, `BrowserContext.debugger`, `Browser.bind/unbind`, `Page.{clear,console}_*`). The 1.59 roll suppressed all of these initially, then had to walk every one back. Verify `langs` first. +- A renamed positional parameter (`Parameter not documented: X.y(old_name=)` + `Parameter not implemented: X.y(new_name=)`). Just rename in `_impl/` and update the channel-send dict key. +- A `Parameter type mismatch in X.y(return=): documented as {field: T}, code has Dict[str, T]`. Use a TypedDict. + +#### N/A + +Common N/A flavors: +- Whole new class with `langs: js` (Disposable, Inspector/Screencast, Debugger, Overlay). +- Members with `langs: js` (most "tooling" / MCP / agentic features). +- Doc-only edits (typo fixes, "Improve `not` property sections", etc.). +- Reverts that cancel an earlier add in the same range (always check the rest of the range before porting something that gets reverted). +- Java/C# `langs:` blocks. +- Electron-only changes (`docs/src/api/class-electron.md`). + +Tick the box in `/tmp/roll--commits.md` with one line: `[x] — : `. + +### 5. Regenerate + +```sh +./scripts/update_api.sh +``` + +The script does, in order: +1. `git checkout HEAD -- playwright/{async,sync}_api/_generated.py` (resets to last committed), +2. runs `scripts/generate_{sync,async}_api.py` which dumps to `.x` then renames into place, +3. invokes `pre-commit run --files` on the generated files. + +Failure modes and fixes: + +| Symptom | Cause | Fix | +|---|---|---| +| `Method not implemented: X.y` | `api.json` documents `X.y`, no Python impl exists. | PORT it, or add a MISMATCH line. | +| `Parameter not implemented: X.y(z=)` | New parameter on existing method. | Add the kwarg in `_impl/`, or MISMATCH. | +| `Method vs property mismatch: X.y` | You implemented as a method but the doc treats it as a property (sync, no args, has return type). | Add `@property` in `_impl/`. | +| `Method not documented: X.y` | Python has it but `api.json` doesn't. | The upstream removed the API; remove from `_impl/` and from `_generated.py` callers. | +| `Parameter type mismatch in X.y(z=): documented as ..., code has ...` | Type signature doesn't line up. | Match the type in `_impl/`, or MISMATCH it for known historical divergences. | +| `pyright … reportInconsistentOverload` (single-event class) | A class gained its first event in `api.json`, so the generator emits one `Literal[…]` overload + impl, which pyright wants ≥2 of. | The generator already handles this — `documentation_provider.print_events` emits a second `@typing.overload` with `event: str` plus a generic impl. If you regress this, see how 1.59 handled `CDPSession` getting a `close` event. | +| `pre-commit` keeps reformatting `_generated.py` on each run | First run after regen always reformats once; rerun until idle. | `pre-commit run --all-files` to settle. | +| `Parameter not documented: X.y(z=)` | Python has a kwarg the docs don't mention (e.g. legacy name from a doc rename). | If the param is positional with no default, just rename it in `_impl/`. Check `protocol.yml` and the server dispatcher — if the wire field renamed too, also update the channel-send dict key. Only MISMATCH for genuinely hidden internal kwargs (`default_browser_type`). | +| `KeyError: 'templates'` deep in `inner_serialize_doc_type` | A `Promise\|X` union upstream collapsed to a bare `Promise` with no templates in `api.json`. | `documentation_provider.inner_serialize_doc_type` should treat that as `Any` (`if "templates" not in type: return "Any"`). | +| `"void" is not defined (reportUndefinedVariable)` in generated event handlers | `api.json` has a `void`-typed event payload that the serializer left as the literal string `"void"`. | `documentation_provider.inner_serialize_doc_type` should map `"void"` to `"None"` alongside `"null"`. | +| `AttributeError: 'X' object has no attribute '_loop'` (or `_dispatcher_fiber`) at first use of a new wrapper class | The non-ChannelOwner wrapper isn't initializing the fields the generated `AsyncBase`/`SyncBase` reads. | In the wrapper's `__init__`, set `self._loop = parent._loop` and `self._dispatcher_fiber = parent._dispatcher_fiber`. | +| `'NoneType' object has no attribute 'get'` after `send_return_as_dict` | Method's protocol response carries no fields and `send_return_as_dict` returned `None`. | `(result or {}).get(...)`. | +| Frame/buffer payload arrives as a `str` instead of `bytes` | Protocol `binary` fields cross the wire as base64. | `base64.b64decode(value)` in the impl before exposing. | + +After the script settles, run `pre-commit run --all-files` once more to confirm everything is idle. + +### 6. Add tests + +For each PORT, add one async test and a matching sync test. Conventions: + +- Tests go in the existing topic file (`test_page_network_request.py`, `test_browsercontext.py`, `test_dialog.py`, …) — don't create new files unless there's no obvious home. +- Use `from playwright.async_api import …`, **not** `from playwright._impl._page import Page` (the impl class doesn't have the public wrappers like `expect_console_message`). +- For event-info objects, `await message_info.value` (it's an `async` property). +- Don't write tests that hang the page (e.g. `page.evaluate(... fetch slow ...)` followed by `page.close()` from a fixture) — the request task gets a `TargetClosedError`. Use `page.on("event", handler)` to capture state at event time instead. +- `playwright install chromium` (no `--with-deps`) is sufficient for the test suite under sandbox. + +### 7. Update existing high-touch artifacts + +- `setup.py`: already done in step 2. +- `README.md`: gets the chromium/firefox/webkit version table updated automatically by `scripts/update_versions.py` (called from `update_api.sh`). Don't edit by hand. +- The "Backport changes" tracking issue on GitHub (filed by `microsoft-playwright-automation`) is the *intent* tracker, but it's frequently out of sync with what's actually been ported. Treat it as a starting point, not the source of truth — the `docs/src/api/` commit walk is authoritative. + +### 8. Final verification + +```sh +pre-commit run --all-files +mypy playwright # 2 pre-existing errors in _json_pipe.py and _artifact.py are unrelated +pytest --browser chromium tests/async/ tests/sync/ +``` + +Then a smoke regression on a few neighboring suites (`tests/async/test_browser*.py`, `test_cdp_session.py`, `test_tracing.py`, `test_dialog.py`, `test_page_*.py`) to make sure nothing inherent to the port shifted. + +## Reference: `expected_api_mismatch.txt` line forms + +Exact strings the generator emits (and that this file must contain to suppress): + +``` +Method not implemented: . +Parameter not implemented: .(=) +Parameter not documented: .(=) +Method vs property mismatch: . +Method not documented: . +Parameter type mismatch in .(=): documented as , code has +``` + +Class names use the upstream PascalCase (`BrowserContext`, `BrowserType`); method/param names are converted to `snake_case` matching the Python surface. + +## Tips & gotchas + +- **`langs.only` is your filter — and the only filter.** Don't classify by name (`Screencast`, `Debugger`, `pickLocator`) or by intuition ("looks like internal tooling"). Always check `langs` in `api.json`. The 1.59 roll cost two extra audit passes by trusting names over `langs`. +- **Audit your own classifications a second time.** After the first walk through the commit range, before opening the PR, re-read every line in `expected_api_mismatch.txt` you added during this roll and ask "is this divergence justified, or did I skip a port?" Run the `langs`-dump snippet on each suspicious entry. The 1.59 roll's first PR had ~20 wrong suppressions; the second pass cut them to 0. +- **A cluster of suppressions on the same class is a smell.** If you're about to add five `Method not implemented: Foo.*` lines, you're almost certainly looking at a class that needs to be implemented. Implement the whole thing once and the suppressions disappear. +- **Watch for revert pairs in the same range.** 1.59 added and reverted `Browser.isRemote` (#39613 / #39620) inside the same release. Walking chronologically lets you skip the add when you see the revert later. +- **Watch for rename-revert pairs.** 1.59 had `Locator.normalize` → `Locator.toCode` (#39648) → `Locator.normalize` (#39754). Final state wins; only port the last. +- **Doc renames almost always include a wire-protocol rename.** Whenever you see `### param: X.y.oldName` → `### param: X.y.newName` in a doc commit, also `git -C ~/code/playwright show -- packages/protocol/src/protocol.yml` and the corresponding `*Dispatcher.ts` file. If the wire field changed too, the channel-send dict key in `_impl/` must change. Suppressing the doc-side mismatch is hiding a real bug — the previous Python code is silently sending an unknown field that the new server ignores. +- **TypedDicts beat `Dict[str, X]` for any structured return.** When the docs describe a return as `[Object]` with named fields (or even `[Object=Foo]`), define a `TypedDict` in `_api_structures.py`, re-export from both public `__init__.py` files, and use it. Zero runtime cost (it's still a `dict`), and the doc generator's type comparator matches by structure via `get_type_hints`. +- **Positional renames are free.** A param with no default before any `*` separator is positional-or-keyword in Python, but realistic call sites pass it positionally. Renaming such a param doesn't break callers. +- **The "Backport changes" GitHub issue can be misleading.** In the 1.59 roll its checkboxes were all marked `[x]` with annotations like "✅ IMPLEMENTED", but several of those features had not actually been merged into the Python port. Trust the `docs/src/api/` walk over the issue. +- **`api.json` may carry doclint quirks.** 1.59 hit two: `Promise|X` collapsed to a bare `Promise` with no `templates`, and `void`-typed events serialized as the literal string `"void"`. Both are upstream artifacts; patch `inner_serialize_doc_type` to handle them rather than fighting the api.json side. +- **Don't edit `_generated.py` to fix lint or typing.** Fix `_impl/`, `documentation_provider.py`, or `expected_api_mismatch.txt` instead. Hand-editing the generated file is reverted on the next regen. +- **`/tmp/roll--commits.md` is a working artifact, not a deliverable.** Don't commit it. The commit message and PR description are where the audit summary belongs. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 65ba1f433..eb06caff0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -50,7 +50,18 @@ jobs: os: [ubuntu-latest, windows-latest, macos-latest] python-version: ['3.9', '3.10'] browser: [chromium, firefox, webkit] + exclude: + # WebKit on standard macOS-latest (currently macos-15-arm64) is unstable; + # upstream pins paid macos-15-xlarge for cross-browser webkit too. + - os: macos-latest + browser: webkit include: + - os: macos-15-xlarge + python-version: '3.9' + browser: webkit + - os: macos-15-xlarge + python-version: '3.10' + browser: webkit - os: windows-latest python-version: '3.11' browser: chromium @@ -160,8 +171,15 @@ jobs: strategy: fail-fast: false matrix: - os: [ubuntu-22.04, macos-13, windows-2022] + os: [ubuntu-22.04, macos-latest, windows-2022] runs-on: ${{ matrix.os }} + defaults: + run: + # `setup-miniconda` activates the env only for login shells; using + # `bash -el` (recommended by the action) ensures `conda` and the + # installed `conda-build` are on PATH on every OS, including Windows + # where the default shell is pwsh and skips the activation hooks. + shell: bash -el {0} steps: - uses: actions/checkout@v5 with: @@ -169,11 +187,11 @@ jobs: - name: Get conda uses: conda-incubator/setup-miniconda@v3 with: - python-version: 3.9 + python-version: '3.12' channels: conda-forge miniconda-version: latest - name: Prepare - run: conda install conda-build conda-verify + run: conda install -n base "conda-build>=26" conda-verify - name: Build run: conda build . diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index c6e71028a..e9a7048c5 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -31,11 +31,11 @@ jobs: - name: Get conda uses: conda-incubator/setup-miniconda@v3 with: - python-version: 3.9 + python-version: '3.12' channels: conda-forge miniconda-version: latest - name: Prepare - run: conda install anaconda-client conda-build conda-verify + run: conda install -n base anaconda-client "conda-build>=26" conda-verify - name: Build and Upload env: ANACONDA_API_TOKEN: ${{ secrets.ANACONDA_API_TOKEN }} diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..1cbaf3e1a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,98 @@ +# CLAUDE.md + +Guidance for Claude when working in this repository. + +## What this is + +Python bindings for [Playwright](https://playwright.dev). The Python client talks JSON over a pipe to the Node-based driver bundled in `playwright/driver/`. The pipe protocol is defined upstream in `packages/protocol/src/protocol.yml`. + +## Layout + +- `playwright/_impl/` — hand-written client implementation (one module per object: `_browser.py`, `_page.py`, `_locator.py`, `_network.py`, etc.). Edit these to add or change behavior. +- `playwright/async_api/_generated.py`, `playwright/sync_api/_generated.py` — **auto-generated**. Never edit by hand; rerun `./scripts/update_api.sh` after changing `_impl/` or the driver. +- `scripts/generate_api.py`, `scripts/generate_async_api.py`, `scripts/generate_sync_api.py`, `scripts/documentation_provider.py` — codegen and validation. They diff the Python implementation against the driver's `playwright/driver/package/api.json` and abort if either side is out of sync. +- `scripts/expected_api_mismatch.txt` — explicit allowlist of "documented in JS, not in Python" or "named differently in Python" gaps. Lines that no longer apply must be removed. +- `tests/async/`, `tests/sync/` — pytest suites. Most new tests are added to the async file with a sync mirror. +- `setup.py` — `driver_version = "X.Y.Z"` is the source of truth for which driver build is downloaded from `cdn.playwright.dev`. +- `ROLLING.md`, `CONTRIBUTING.md` — human-facing setup and roll docs. + +## Setup + +`CONTRIBUTING.md` has the full sequence. The short version: + +```sh +python3 -m venv env && source env/bin/activate +pip install --upgrade pip +pip install -r local-requirements.txt +pip install -e . +python -m build --wheel # downloads the driver listed in setup.py +pre-commit install +``` + +If the system lacks `python3-venv`, `uv venv env` is an acceptable substitute (then `uv pip install --python env/bin/python --upgrade pip`). + +## Common commands + +- Regenerate `_generated.py`: `./scripts/update_api.sh` (runs codegen + pre-commit on the generated files). +- Lint everything: `pre-commit run --all-files`. +- Type-check: `mypy playwright`. +- Run tests: `pytest --browser chromium [-k name]`. Browsers are installed via `playwright install chromium` (do **not** use `--with-deps`, which requires sudo). + +When changing public API, edit `_impl/`, then run `./scripts/update_api.sh`. The script regenerates `_generated.py` and validates against the driver's `api.json`. If validation fails, fix the mismatch in `_impl/`, in `expected_api_mismatch.txt`, or in `documentation_provider.py` — not by hand-editing `_generated.py`. + +## Rolling Playwright to a new version + +This is the recurring high-stakes task. Use the dedicated skill: + +→ **[`.claude/skills/playwright-roll/SKILL.md`](.claude/skills/playwright-roll/SKILL.md)** + +It documents the full process: the upstream commit-range diff over `docs/src/api/`, how to classify each commit (PORT / MISMATCH / N/A), how to handle the `langs:` filter, the recurring failure modes, and the tests/sync-mirroring conventions. + +## Working on PRs + +- Never post comments, replies, or reviews on GitHub PRs/issues under my account without my explicit approval. Draft the proposed text and wait for me to approve before sending. + +## House style + +- Don't hand-edit generated files. +- Don't add `# type: ignore` or modify `_generated.py` to silence pyright; fix the source of the mismatch. +- New public methods on impl classes need a sync test mirror under `tests/sync/`. +- Keep `expected_api_mismatch.txt` minimal — every entry needs a one-line rationale comment above it. +- Prefer `locals_to_params(locals())` for forwarding optional kwargs to channel sends, matching the rest of the codebase. + +## Commit Convention + +Before committing, run `mypy playwright` and fix errors. + +Semantic commit messages: `label(scope): description` + +Labels: `fix`, `feat`, `chore`, `docs`, `test`, `devops` + +```bash +git checkout -b fix-12345 +# ... make changes ... +git add +git commit -m "$(cat <<'EOF' +fix(asyncio): do not deadlock in atexit handler + +Fixes: https://github.com/microsoft/playwright-python/issues/12345 +EOF +)" +git push origin fix-12345 +gh pr create --repo microsoft/playwright-python --head username:fix-12345 \ + --title "fix(asyncio): do not deadlock in atexit handler" \ + --body "$(cat <<'EOF' +## Summary +- + +Fixes https://github.com/microsoft/playwright-python/issues/12345 +EOF +)" +``` + +Never add Co-Authored-By agents in commit message. +Never add "Generated with" in commit message. +Never add test plan to PR description. Keep PR description short — a few bullet points at most. +Branch naming for issue fixes: `fix-` + +**Never `git push` without an explicit instruction to push.** Applies even when a PR is already open for the branch — additional commits are immediately visible to reviewers. Commit locally, report what was committed, and wait. Only push when the user's message contains "push", "upload", "create PR", "ship it", or equivalent. diff --git a/README.md b/README.md index c5e2d70b0..f0a4fc423 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ Playwright is a Python library to automate [Chromium](https://www.chromium.org/H | | Linux | macOS | Windows | | :--- | :---: | :---: | :---: | -| Chromium 145.0.7632.6 | ✅ | ✅ | ✅ | -| WebKit 26.0 | ✅ | ✅ | ✅ | -| Firefox 146.0.1 | ✅ | ✅ | ✅ | +| Chromium 147.0.7727.15 | ✅ | ✅ | ✅ | +| WebKit 26.4 | ✅ | ✅ | ✅ | +| Firefox 148.0.2 | ✅ | ✅ | ✅ | ## Documentation diff --git a/local-requirements.txt b/local-requirements.txt index 8a72b5745..5735ae2a0 100644 --- a/local-requirements.txt +++ b/local-requirements.txt @@ -1,3 +1,4 @@ +asyncio-atexit==1.0.1 autobahn==23.1.2 black==25.1.0 build==1.3.0 diff --git a/playwright/_impl/_api_structures.py b/playwright/_impl/_api_structures.py index c0d0ee442..256b59435 100644 --- a/playwright/_impl/_api_structures.py +++ b/playwright/_impl/_api_structures.py @@ -166,6 +166,10 @@ class RemoteAddr(TypedDict): port: int +class BrowserBindResult(TypedDict): + endpoint: str + + class SecurityDetails(TypedDict): issuer: Optional[str] protocol: Optional[str] @@ -311,3 +315,18 @@ class TracingGroupLocation(TypedDict): file: str line: Optional[int] column: Optional[int] + + +class DebuggerLocation(TypedDict): + file: str + line: Optional[int] + column: Optional[int] + + +class DebuggerPausedDetails(TypedDict): + location: DebuggerLocation + title: str + + +class ScreencastFrame(TypedDict): + data: bytes diff --git a/playwright/_impl/_browser.py b/playwright/_impl/_browser.py index 5a9a87450..6454f8c3f 100644 --- a/playwright/_impl/_browser.py +++ b/playwright/_impl/_browser.py @@ -27,6 +27,7 @@ ) from playwright._impl._api_structures import ( + BrowserBindResult, ClientCertificate, Geolocation, HttpCredentials, @@ -247,6 +248,20 @@ def version(self) -> str: async def new_browser_cdp_session(self) -> CDPSession: return from_channel(await self._channel.send("newBrowserCDPSession", None)) + async def bind( + self, + title: str, + workspaceDir: str = None, + host: str = None, + port: int = None, + ) -> BrowserBindResult: + return await self._channel.send_return_as_dict( + "startServer", None, locals_to_params(locals()) + ) + + async def unbind(self) -> None: + await self._channel.send("stopServer", None) + async def start_tracing( self, page: Page = None, diff --git a/playwright/_impl/_browser_context.py b/playwright/_impl/_browser_context.py index f56564a27..6839d7c7f 100644 --- a/playwright/_impl/_browser_context.py +++ b/playwright/_impl/_browser_context.py @@ -46,7 +46,9 @@ from_nullable_channel, ) from playwright._impl._console_message import ConsoleMessage +from playwright._impl._debugger import Debugger from playwright._impl._dialog import Dialog +from playwright._impl._disposable import Disposable, DisposableStub from playwright._impl._errors import Error, TargetClosedError from playwright._impl._event_context_manager import EventContextManagerImpl from playwright._impl._fetch import APIRequestContext @@ -122,8 +124,10 @@ def __init__( self._base_url: Optional[str] = self._options.get("baseURL") self._videos_dir: Optional[str] = self._options.get("recordVideo") self._tracing = cast(Tracing, from_channel(initializer["tracing"])) + self._debugger: Debugger = cast(Debugger, from_channel(initializer["debugger"])) self._har_recorders: Dict[str, HarRecordingMetadata] = {} self._request: APIRequestContext = from_channel(initializer["requestContext"]) + self._request._timeout_settings = self._timeout_settings self._clock = Clock(self) self._channel.on( "bindingCall", @@ -391,16 +395,18 @@ async def set_offline(self, offline: bool) -> None: async def add_init_script( self, script: str = None, path: Union[str, Path] = None - ) -> None: + ) -> Disposable: if path: script = (await async_readfile(path)).decode() if not isinstance(script, str): raise Error("Either path or script parameter must be specified") - await self._channel.send("addInitScript", None, dict(source=script)) + return from_channel( + await self._channel.send("addInitScript", None, dict(source=script)) + ) async def expose_binding( self, name: str, callback: Callable, handle: bool = None - ) -> None: + ) -> Disposable: for page in self._pages: if name in page._bindings: raise Error( @@ -409,16 +415,18 @@ async def expose_binding( if name in self._bindings: raise Error(f'Function "{name}" has been already registered') self._bindings[name] = callback - await self._channel.send( - "exposeBinding", None, dict(name=name, needsHandle=handle or False) + return from_channel( + await self._channel.send( + "exposeBinding", None, dict(name=name, needsHandle=handle or False) + ) ) - async def expose_function(self, name: str, callback: Callable) -> None: - await self.expose_binding(name, lambda source, *args: callback(*args)) + async def expose_function(self, name: str, callback: Callable) -> Disposable: + return await self.expose_binding(name, lambda source, *args: callback(*args)) async def route( self, url: URLMatch, handler: RouteHandlerCallback, times: int = None - ) -> None: + ) -> DisposableStub: self._routes.insert( 0, RouteHandler( @@ -430,6 +438,7 @@ async def route( ), ) await self._update_interception_patterns() + return DisposableStub(lambda: self.unroute(url, handler), self) async def unroute( self, url: URLMatch, handler: Optional[RouteHandlerCallback] = None @@ -581,6 +590,9 @@ def _on_close(self) -> None: self._tracing._reset_stack_counter() self.emit(BrowserContext.Events.Close, self) + def is_closed(self) -> bool: + return self._closing_or_closed + async def close(self, reason: str = None) -> None: if self._closing_or_closed: return @@ -626,6 +638,15 @@ async def storage_state( await async_writefile(path, json.dumps(result)) return result + async def set_storage_state( + self, storageState: Union[StorageState, str, Path] + ) -> None: + if isinstance(storageState, (str, Path)): + state = json.loads(await async_readfile(storageState)) + else: + state = storageState + await self._channel.send("setStorageState", None, {"storageState": state}) + def _effective_close_reason(self) -> Optional[str]: if self._close_reason: return self._close_reason @@ -752,6 +773,10 @@ async def new_cdp_session(self, page: Union[Page, Frame]) -> CDPSession: def tracing(self) -> Tracing: return self._tracing + @property + def debugger(self) -> Debugger: + return self._debugger + @property def request(self) -> "APIRequestContext": return self._request diff --git a/playwright/_impl/_browser_type.py b/playwright/_impl/_browser_type.py index 3aef7dd6d..88ea5fc52 100644 --- a/playwright/_impl/_browser_type.py +++ b/playwright/_impl/_browser_type.py @@ -86,6 +86,7 @@ async def launch( downloadsPath: Union[str, Path] = None, slowMo: float = None, tracesDir: Union[pathlib.Path, str] = None, + artifactsDir: Union[pathlib.Path, str] = None, chromiumSandbox: bool = None, firefoxUserPrefs: Dict[str, Union[str, float, bool]] = None, ) -> Browser: @@ -143,6 +144,7 @@ async def launch_persistent_context( contrast: Contrast = None, acceptDownloads: bool = None, tracesDir: Union[pathlib.Path, str] = None, + artifactsDir: Union[pathlib.Path, str] = None, chromiumSandbox: bool = None, firefoxUserPrefs: Dict[str, Union[str, float, bool]] = None, recordHarPath: Union[Path, str] = None, @@ -213,7 +215,7 @@ async def connect_over_cdp( async def connect( self, - wsEndpoint: str, + endpoint: str, timeout: float = None, slowMo: float = None, headers: Dict[str, str] = None, @@ -229,7 +231,7 @@ async def connect( "connect", None, { - "wsEndpoint": wsEndpoint, + "endpoint": endpoint, "headers": headers, "slowMo": slowMo, "timeout": timeout if timeout is not None else 0, @@ -355,9 +357,13 @@ def normalize_launch_params(params: Dict) -> None: if params["ignoreDefaultArgs"] is True: params["ignoreAllDefaultArgs"] = True del params["ignoreDefaultArgs"] + elif params["ignoreDefaultArgs"] is False: + del params["ignoreDefaultArgs"] if "executablePath" in params: params["executablePath"] = str(Path(params["executablePath"])) if "downloadsPath" in params: params["downloadsPath"] = str(Path(params["downloadsPath"])) if "tracesDir" in params: params["tracesDir"] = str(Path(params["tracesDir"])) + if "artifactsDir" in params: + params["artifactsDir"] = str(Path(params["artifactsDir"])) diff --git a/playwright/_impl/_cdp_session.py b/playwright/_impl/_cdp_session.py index 95e65c57a..07d291ae2 100644 --- a/playwright/_impl/_cdp_session.py +++ b/playwright/_impl/_cdp_session.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. +from types import SimpleNamespace from typing import Any, Dict from playwright._impl._connection import ChannelOwner @@ -19,14 +20,21 @@ class CDPSession(ChannelOwner): + Events = SimpleNamespace( + Event="event", + Close="close", + ) + def __init__( self, parent: ChannelOwner, type: str, guid: str, initializer: Dict ) -> None: super().__init__(parent, type, guid, initializer) self._channel.on("event", lambda params: self._on_event(params)) + self._channel.on("close", lambda _: self.emit(CDPSession.Events.Close, self)) def _on_event(self, params: Any) -> None: self.emit(params["method"], params.get("params")) + self.emit(CDPSession.Events.Event, params) async def send(self, method: str, params: Dict = None) -> Dict: return await self._channel.send("send", None, locals_to_params(locals())) diff --git a/playwright/_impl/_console_message.py b/playwright/_impl/_console_message.py index f37e3dd4d..d98901d34 100644 --- a/playwright/_impl/_console_message.py +++ b/playwright/_impl/_console_message.py @@ -76,6 +76,10 @@ def args(self) -> List[JSHandle]: def location(self) -> SourceLocation: return self._event["location"] + @property + def timestamp(self) -> float: + return self._event["timestamp"] + @property def page(self) -> Optional["Page"]: return self._page diff --git a/playwright/_impl/_debugger.py b/playwright/_impl/_debugger.py new file mode 100644 index 000000000..36e4bd989 --- /dev/null +++ b/playwright/_impl/_debugger.py @@ -0,0 +1,54 @@ +# Copyright (c) Microsoft Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from types import SimpleNamespace +from typing import Any, Dict, Optional + +from playwright._impl._api_structures import DebuggerLocation, DebuggerPausedDetails +from playwright._impl._connection import ChannelOwner + + +class Debugger(ChannelOwner): + Events = SimpleNamespace( + PausedStateChanged="pausedstatechanged", + ) + + def __init__( + self, parent: ChannelOwner, type: str, guid: str, initializer: Dict + ) -> None: + super().__init__(parent, type, guid, initializer) + self._paused_details: Optional[DebuggerPausedDetails] = None + self._channel.on( + "pausedStateChanged", lambda params: self._on_paused_state_changed(params) + ) + + def _on_paused_state_changed(self, params: Dict[str, Any]) -> None: + self._paused_details = params.get("pausedDetails") + self.emit(Debugger.Events.PausedStateChanged) + + async def request_pause(self) -> None: + await self._channel.send("requestPause", None) + + async def resume(self) -> None: + await self._channel.send("resume", None) + + async def next(self) -> None: + await self._channel.send("next", None) + + async def run_to(self, location: DebuggerLocation) -> None: + await self._channel.send("runTo", None, {"location": location}) + + @property + def paused_details(self) -> Optional[DebuggerPausedDetails]: + return self._paused_details diff --git a/playwright/_impl/_dialog.py b/playwright/_impl/_dialog.py index 226e703b9..f6750e396 100644 --- a/playwright/_impl/_dialog.py +++ b/playwright/_impl/_dialog.py @@ -15,6 +15,7 @@ from typing import TYPE_CHECKING, Dict, Optional from playwright._impl._connection import ChannelOwner, from_nullable_channel +from playwright._impl._errors import is_target_closed_error from playwright._impl._helper import locals_to_params if TYPE_CHECKING: # pragma: no cover @@ -51,7 +52,12 @@ async def accept(self, promptText: str = None) -> None: await self._channel.send("accept", None, locals_to_params(locals())) async def dismiss(self) -> None: - await self._channel.send( - "dismiss", - None, - ) + try: + await self._channel.send( + "dismiss", + None, + ) + except Exception as e: + if is_target_closed_error(e): + return + raise diff --git a/playwright/_impl/_disposable.py b/playwright/_impl/_disposable.py new file mode 100644 index 000000000..c0b7e85a1 --- /dev/null +++ b/playwright/_impl/_disposable.py @@ -0,0 +1,93 @@ +# Copyright (c) Microsoft Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import inspect +import traceback +from typing import Awaitable, Callable, Dict + +import greenlet + +from playwright._impl._connection import ChannelOwner +from playwright._impl._errors import Error, is_target_closed_error + + +class Disposable(ChannelOwner): + def __init__( + self, parent: ChannelOwner, type: str, guid: str, initializer: Dict + ) -> None: + super().__init__(parent, type, guid, initializer) + + async def dispose(self) -> None: + try: + await self._channel.send( + "dispose", + None, + ) + except Exception as e: + if not is_target_closed_error(e): + raise e + + async def close(self) -> None: + await self.dispose() + + def __repr__(self) -> str: + return "" + + +class DisposableStub: + def __init__( + self, + dispose_fn: Callable[[], Awaitable[None]], + parent: ChannelOwner, + ) -> None: + self._dispose_fn = dispose_fn + self._loop = parent._loop + self._dispatcher_fiber = parent._dispatcher_fiber + + async def dispose(self) -> None: + await self._dispose_fn() + + async def __aenter__(self) -> "DisposableStub": + return self + + async def __aexit__(self, *args: object) -> None: + await self.dispose() + + def __enter__(self) -> "DisposableStub": + return self + + def __exit__(self, *args: object) -> None: + self._sync(self.dispose()) + + def _sync(self, coro: object) -> object: + __tracebackhide__ = True + if self._loop.is_closed(): + coro.close() # type: ignore + raise Error("Event loop is closed! Is Playwright already stopped?") + g_self = greenlet.getcurrent() + task = self._loop.create_task(coro) # type: ignore + setattr(task, "__pw_stack__", inspect.stack(0)) + setattr(task, "__pw_stack_trace__", traceback.extract_stack(limit=10)) + task.add_done_callback(lambda _: g_self.switch()) + while not task.done(): + self._dispatcher_fiber.switch() # type: ignore + asyncio._set_running_loop(self._loop) + return task.result() + + async def close(self) -> None: + await self.dispose() + + def __repr__(self) -> str: + return "" diff --git a/playwright/_impl/_helper.py b/playwright/_impl/_helper.py index 1d7e4f67b..dc0a2479d 100644 --- a/playwright/_impl/_helper.py +++ b/playwright/_impl/_helper.py @@ -22,6 +22,7 @@ from typing import ( TYPE_CHECKING, Any, + Awaitable, Callable, Dict, List, @@ -54,8 +55,12 @@ from playwright._impl._network import Request, Response, Route, WebSocketRoute URLMatch = Union[str, Pattern[str], Callable[[str], bool]] -URLMatchRequest = Union[str, Pattern[str], Callable[["Request"], bool]] -URLMatchResponse = Union[str, Pattern[str], Callable[["Response"], bool]] +URLMatchRequest = Union[ + str, Pattern[str], Callable[["Request"], Union[bool, Awaitable[bool]]] +] +URLMatchResponse = Union[ + str, Pattern[str], Callable[["Response"], Union[bool, Awaitable[bool]]] +] RouteHandlerCallback = Union[ Callable[["Route"], Any], Callable[["Route", "Request"], Any] ] diff --git a/playwright/_impl/_local_utils.py b/playwright/_impl/_local_utils.py index c2d2d3fca..cd8fd8343 100644 --- a/playwright/_impl/_local_utils.py +++ b/playwright/_impl/_local_utils.py @@ -62,7 +62,9 @@ async def har_unzip(self, zipFile: str, harFile: str) -> None: params = locals_to_params(locals()) await self._channel.send("harUnzip", None, params) - async def tracing_started(self, tracesDir: Optional[str], traceName: str) -> str: + async def tracing_started( + self, tracesDir: Optional[str], traceName: str, live: bool = False + ) -> str: params = locals_to_params(locals()) return await self._channel.send("tracingStarted", None, params) diff --git a/playwright/_impl/_locator.py b/playwright/_impl/_locator.py index 2e6a7abed..5f1b8f29a 100644 --- a/playwright/_impl/_locator.py +++ b/playwright/_impl/_locator.py @@ -564,7 +564,12 @@ async def screenshot( ), ) - async def aria_snapshot(self, timeout: float = None) -> str: + async def aria_snapshot( + self, + timeout: float = None, + depth: int = None, + mode: Literal["ai", "default"] = None, + ) -> str: return await self._frame._channel.send( "ariaSnapshot", self._frame._timeout, @@ -574,6 +579,14 @@ async def aria_snapshot(self, timeout: float = None) -> str: }, ) + async def normalize(self) -> "Locator": + result = await self._frame._channel.send( + "resolveSelector", + None, + {"selector": self._selector}, + ) + return Locator(self._frame, result) + async def scroll_into_view_if_needed( self, timeout: float = None, diff --git a/playwright/_impl/_network.py b/playwright/_impl/_network.py index 9f2c29f6e..06bf88267 100644 --- a/playwright/_impl/_network.py +++ b/playwright/_impl/_network.py @@ -154,6 +154,7 @@ def __init__( self._fallback_overrides: SerializedFallbackOverrides = ( SerializedFallbackOverrides() ) + self._response: Optional["Response"] = None def __repr__(self) -> str: return f"" @@ -243,6 +244,10 @@ async def response(self) -> Optional["Response"]: ) ) + @property + def existing_response(self) -> Optional["Response"]: + return self._response + @property def frame(self) -> "Frame": if not self._initializer.get("frame"): @@ -799,6 +804,7 @@ def __init__( ) -> None: super().__init__(parent, type, guid, initializer) self._request: Request = from_channel(self._initializer["request"]) + self._request._response = self timing = self._initializer["timing"] self._request._timing["startTime"] = timing["startTime"] self._request._timing["domainLookupStart"] = timing["domainLookupStart"] @@ -881,6 +887,12 @@ async def security_details(self) -> Optional[SecurityDetails]: None, ) + async def http_version(self) -> str: + return await self._channel.send( + "httpVersion", + None, + ) + async def finished(self) -> None: async def on_finished() -> None: await self._request._target_closed_future() diff --git a/playwright/_impl/_object_factory.py b/playwright/_impl/_object_factory.py index b44009bc3..7911ddb30 100644 --- a/playwright/_impl/_object_factory.py +++ b/playwright/_impl/_object_factory.py @@ -20,7 +20,9 @@ from playwright._impl._browser_type import BrowserType from playwright._impl._cdp_session import CDPSession from playwright._impl._connection import ChannelOwner +from playwright._impl._debugger import Debugger from playwright._impl._dialog import Dialog +from playwright._impl._disposable import Disposable from playwright._impl._element_handle import ElementHandle from playwright._impl._fetch import APIRequestContext from playwright._impl._frame import Frame @@ -64,8 +66,12 @@ def create_remote_object( return BrowserContext(parent, type, guid, initializer) if type == "CDPSession": return CDPSession(parent, type, guid, initializer) + if type == "Debugger": + return Debugger(parent, type, guid, initializer) if type == "Dialog": return Dialog(parent, type, guid, initializer) + if type == "Disposable": + return Disposable(parent, type, guid, initializer) if type == "ElementHandle": return ElementHandle(parent, type, guid, initializer) if type == "Frame": diff --git a/playwright/_impl/_page.py b/playwright/_impl/_page.py index 1f05a9048..5a8444624 100644 --- a/playwright/_impl/_page.py +++ b/playwright/_impl/_page.py @@ -22,6 +22,7 @@ from typing import ( TYPE_CHECKING, Any, + Awaitable, Callable, Dict, List, @@ -49,6 +50,7 @@ from_nullable_channel, ) from playwright._impl._console_message import ConsoleMessage +from playwright._impl._disposable import Disposable, DisposableStub from playwright._impl._download import Download from playwright._impl._element_handle import ElementHandle, determine_screenshot_type from playwright._impl._errors import Error, TargetClosedError, is_target_closed_error @@ -98,6 +100,7 @@ WebSocketRouteHandler, serialize_headers, ) +from playwright._impl._screencast import Screencast from playwright._impl._video import Video from playwright._impl._waiter import Waiter @@ -175,7 +178,11 @@ def __init__( self._timeout_settings: TimeoutSettings = TimeoutSettings( self._browser_context._timeout_settings ) - self._video: Optional[Video] = None + self._video: Video = Video( + self, + cast(Optional[Artifact], from_nullable_channel(initializer.get("video"))), + ) + self._screencast: Screencast = Screencast(self) self._opener = cast("Page", from_nullable_channel(initializer.get("opener"))) self._close_reason: Optional[str] = None self._close_was_called = False @@ -224,7 +231,6 @@ def __init__( self._on_web_socket_route(from_channel(params["webSocketRoute"])) ), ) - self._channel.on("video", lambda params: self._on_video(params)) self._channel.on("viewportSizeChanged", self._on_viewport_size_changed) self._channel.on( "webSocket", @@ -356,10 +362,6 @@ def _on_download(self, params: Any) -> None: Page.Events.Download, Download(self, url, suggested_filename, artifact) ) - def _on_video(self, params: Any) -> None: - artifact = from_channel(params["artifact"]) - self._force_video()._artifact_ready(artifact) - def _on_viewport_size_changed(self, params: Any) -> None: self._viewport_size = params["viewportSize"] @@ -501,12 +503,12 @@ async def add_style_tag( ) -> ElementHandle: return await self._main_frame.add_style_tag(**locals_to_params(locals())) - async def expose_function(self, name: str, callback: Callable) -> None: - await self.expose_binding(name, lambda source, *args: callback(*args)) + async def expose_function(self, name: str, callback: Callable) -> Disposable: + return await self.expose_binding(name, lambda source, *args: callback(*args)) async def expose_binding( self, name: str, callback: Callable, handle: bool = None - ) -> None: + ) -> Disposable: if name in self._bindings: raise Error(f'Function "{name}" has been already registered') if name in self._browser_context._bindings: @@ -514,10 +516,12 @@ async def expose_binding( f'Function "{name}" has been already registered in the browser context' ) self._bindings[name] = callback - await self._channel.send( - "exposeBinding", - None, - dict(name=name, needsHandle=handle or False), + return from_channel( + await self._channel.send( + "exposeBinding", + None, + dict(name=name, needsHandle=handle or False), + ) ) async def set_extra_http_headers(self, headers: Dict[str, str]) -> None: @@ -661,18 +665,20 @@ async def bring_to_front(self) -> None: async def add_init_script( self, script: str = None, path: Union[str, Path] = None - ) -> None: + ) -> Disposable: if path: script = add_source_url_to_script( (await async_readfile(path)).decode(), path ) if not isinstance(script, str): raise Error("Either path or script parameter must be specified") - await self._channel.send("addInitScript", None, dict(source=script)) + return from_channel( + await self._channel.send("addInitScript", None, dict(source=script)) + ) async def route( self, url: URLMatch, handler: RouteHandlerCallback, times: int = None - ) -> None: + ) -> DisposableStub: self._routes.insert( 0, RouteHandler( @@ -684,6 +690,7 @@ async def route( ), ) await self._update_interception_patterns() + return DisposableStub(lambda: self.unroute(url, handler), self) async def unroute( self, url: URLMatch, handler: Optional[RouteHandlerCallback] = None @@ -823,6 +830,18 @@ async def screenshot( async def title(self) -> str: return await self._main_frame.title() + async def aria_snapshot( + self, + timeout: float = None, + depth: int = None, + mode: Literal["ai", "default"] = None, + ) -> str: + return await self._main_frame._channel.send( + "ariaSnapshot", + self._main_frame._timeout, + locals_to_params(locals()), + ) + async def close(self, runBeforeUnload: bool = None, reason: str = None) -> None: self._close_reason = reason self._close_was_called = True @@ -1168,21 +1187,17 @@ async def pdf( await async_writefile(path, decoded_binary) return decoded_binary - def _force_video(self) -> Video: - if not self._video: - self._video = Video(self) + @property + def video(self) -> Optional[Video]: + # Video is only exposed when the page actually produced a recording artifact. + # The initializer carries the artifact; if absent, no video was recorded. + if not self._video._artifact: + return None return self._video @property - def video( - self, - ) -> Optional[Video]: - # Note: we are creating Video object lazily, because we do not know - # BrowserContextOptions when constructing the page - it is assigned - # too late during launchPersistentContext. - if not self._browser_context._videos_dir: - return None - return self._force_video() + def screencast(self) -> Screencast: + return self._screencast def _close_error_with_reason(self) -> TargetClosedError: return TargetClosedError( @@ -1264,7 +1279,7 @@ def expect_request( urlOrPredicate: URLMatchRequest, timeout: float = None, ) -> EventContextManagerImpl[Request]: - def my_predicate(request: Request) -> bool: + def my_predicate(request: Request) -> Union[bool, Awaitable[bool]]: if not callable(urlOrPredicate): return url_matches( self._browser_context._base_url, @@ -1296,7 +1311,7 @@ def expect_response( urlOrPredicate: URLMatchResponse, timeout: float = None, ) -> EventContextManagerImpl[Response]: - def my_predicate(request: Response) -> bool: + def my_predicate(request: Response) -> Union[bool, Awaitable[bool]]: if not callable(urlOrPredicate): return url_matches( self._browser_context._base_url, @@ -1435,8 +1450,12 @@ async def requests(self) -> List[Request]: request_objects = await self._channel.send("requests", None) return [from_channel(r) for r in request_objects] - async def console_messages(self) -> List[ConsoleMessage]: - message_dicts = await self._channel.send("consoleMessages", None) + async def console_messages( + self, filter: Literal["all", "since-navigation"] = None + ) -> List[ConsoleMessage]: + message_dicts = await self._channel.send( + "consoleMessages", None, locals_to_params(locals()) + ) return [ ConsoleMessage( {**event, "page": self._channel}, self._loop, self._dispatcher_fiber @@ -1444,10 +1463,27 @@ async def console_messages(self) -> List[ConsoleMessage]: for event in message_dicts ] - async def page_errors(self) -> List[Error]: - error_objects = await self._channel.send("pageErrors", None) + async def page_errors( + self, filter: Literal["all", "since-navigation"] = None + ) -> List[Error]: + error_objects = await self._channel.send( + "pageErrors", None, locals_to_params(locals()) + ) return [parse_error(error["error"]) for error in error_objects] + async def clear_console_messages(self) -> None: + await self._channel.send("clearConsoleMessages", None) + + async def clear_page_errors(self) -> None: + await self._channel.send("clearPageErrors", None) + + async def pick_locator(self) -> "Locator": + selector = await self._channel.send("pickLocator", None, {}) + return self.locator(selector) + + async def cancel_pick_locator(self) -> None: + await self._channel.send("cancelPickLocator", None, {}) + class Worker(ChannelOwner): Events = SimpleNamespace(Close="close", Console="console") diff --git a/playwright/_impl/_screencast.py b/playwright/_impl/_screencast.py new file mode 100644 index 000000000..1f1da3c4f --- /dev/null +++ b/playwright/_impl/_screencast.py @@ -0,0 +1,140 @@ +# Copyright (c) Microsoft Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +from pathlib import Path +from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Union + +from playwright._impl._api_structures import ScreencastFrame +from playwright._impl._artifact import Artifact +from playwright._impl._connection import from_nullable_channel +from playwright._impl._disposable import DisposableStub +from playwright._impl._errors import Error +from playwright._impl._helper import locals_to_params + +if TYPE_CHECKING: # pragma: no cover + from playwright._impl._page import Page + + +ScreencastFrameCallback = Callable[[ScreencastFrame], Any] +ScreencastPosition = Literal[ + "bottom", + "bottom-left", + "bottom-right", + "top", + "top-left", + "top-right", +] + + +class Screencast: + def __init__(self, page: "Page") -> None: + self._page = page + self._loop = page._loop + self._dispatcher_fiber = page._dispatcher_fiber + self._started = False + self._save_path: Optional[Union[str, Path]] = None + self._on_frame: Optional[ScreencastFrameCallback] = None + self._artifact: Optional[Artifact] = None + page._channel.on("screencastFrame", lambda params: self._dispatch_frame(params)) + + def _dispatch_frame(self, params: dict) -> None: + if not self._on_frame: + return + data = params["data"] + if isinstance(data, str): + data = base64.b64decode(data) + result = self._on_frame({"data": data}) + if hasattr(result, "__await__"): + self._page._loop.create_task(result) + + async def start( + self, + onFrame: ScreencastFrameCallback = None, + path: Union[str, Path] = None, + quality: int = None, + ) -> DisposableStub: + if self._started: + raise Error("Screencast is already started") + self._started = True + self._on_frame = onFrame + result = await self._page._channel.send_return_as_dict( + "screencastStart", + None, + { + "quality": quality, + "sendFrames": bool(onFrame), + "record": bool(path), + }, + ) + artifact_channel = (result or {}).get("artifact") + if artifact_channel: + self._artifact = from_nullable_channel(artifact_channel) + self._save_path = path + return DisposableStub(lambda: self.stop(), self._page) + + async def stop(self) -> None: + self._started = False + self._on_frame = None + await self._page._channel.send("screencastStop", None) + if self._save_path and self._artifact: + await self._artifact.save_as(self._save_path) + self._artifact = None + self._save_path = None + + async def show_actions( + self, + duration: float = None, + position: ScreencastPosition = None, + fontSize: int = None, + ) -> DisposableStub: + await self._page._channel.send( + "screencastShowActions", None, locals_to_params(locals()) + ) + return DisposableStub(lambda: self.hide_actions(), self._page) + + async def hide_actions(self) -> None: + await self._page._channel.send("screencastHideActions", None) + + async def show_overlay(self, html: str, duration: float = None) -> DisposableStub: + result = await self._page._channel.send_return_as_dict( + "screencastShowOverlay", None, locals_to_params(locals()) + ) + overlay_id = (result or {}).get("id") + return DisposableStub( + lambda: self._page._channel.send( + "screencastRemoveOverlay", None, {"id": overlay_id} + ), + self._page, + ) + + async def show_chapter( + self, + title: str, + description: str = None, + duration: float = None, + ) -> None: + await self._page._channel.send( + "screencastChapter", None, locals_to_params(locals()) + ) + + async def show_overlays(self) -> None: + await self._page._channel.send( + "screencastSetOverlayVisible", None, {"visible": True} + ) + + async def hide_overlays(self) -> None: + await self._page._channel.send( + "screencastSetOverlayVisible", None, {"visible": False} + ) diff --git a/playwright/_impl/_tracing.py b/playwright/_impl/_tracing.py index bbc6ec35e..2798b89d9 100644 --- a/playwright/_impl/_tracing.py +++ b/playwright/_impl/_tracing.py @@ -18,6 +18,7 @@ from playwright._impl._api_structures import TracingGroupLocation from playwright._impl._artifact import Artifact from playwright._impl._connection import ChannelOwner, from_nullable_channel +from playwright._impl._disposable import DisposableStub from playwright._impl._helper import locals_to_params @@ -27,6 +28,7 @@ def __init__( ) -> None: super().__init__(parent, type, guid, initializer) self._include_sources: bool = False + self._is_live: bool = False self._stacks_id: Optional[str] = None self._is_tracing: bool = False self._traces_dir: Optional[str] = None @@ -38,11 +40,22 @@ async def start( snapshots: bool = None, screenshots: bool = None, sources: bool = None, + live: bool = None, ) -> None: params = locals_to_params(locals()) self._include_sources = bool(sources) + self._is_live = bool(live) - await self._channel.send("tracingStart", None, params) + await self._channel.send( + "tracingStart", + None, + { + "name": name, + "snapshots": snapshots, + "screenshots": screenshots, + "live": live, + }, + ) trace_name = await self._channel.send( "tracingStartChunk", None, {"title": title, "name": name} ) @@ -58,7 +71,7 @@ async def _start_collecting_stacks(self, trace_name: str) -> None: self._is_tracing = True self._connection.set_is_tracing(True) self._stacks_id = await self._connection.local_utils.tracing_started( - self._traces_dir, trace_name + self._traces_dir, trace_name, self._is_live ) async def stop_chunk(self, path: Union[pathlib.Path, str] = None) -> None: @@ -136,8 +149,11 @@ def _reset_stack_counter(self) -> None: self._is_tracing = False self._connection.set_is_tracing(False) - async def group(self, name: str, location: TracingGroupLocation = None) -> None: + async def group( + self, name: str, location: TracingGroupLocation = None + ) -> DisposableStub: await self._channel.send("tracingGroup", None, locals_to_params(locals())) + return DisposableStub(lambda: self.group_end(), self) async def group_end(self) -> None: await self._channel.send( diff --git a/playwright/_impl/_transport.py b/playwright/_impl/_transport.py index 2ca84d459..3cc029e18 100644 --- a/playwright/_impl/_transport.py +++ b/playwright/_impl/_transport.py @@ -137,38 +137,44 @@ async def connect(self) -> None: async def run(self) -> None: assert self._proc.stdout assert self._proc.stdin - while not self._stopped: - try: - buffer = await self._proc.stdout.readexactly(4) - if self._stopped: - break - length = int.from_bytes(buffer, byteorder="little", signed=False) - buffer = bytes(0) - while length: - to_read = min(length, 32768) - data = await self._proc.stdout.readexactly(to_read) + try: + while not self._stopped: + try: + buffer = await self._proc.stdout.readexactly(4) + if self._stopped: + break + length = int.from_bytes(buffer, byteorder="little", signed=False) + buffer = bytes(0) + while length: + to_read = min(length, 32768) + data = await self._proc.stdout.readexactly(to_read) + if self._stopped: + break + length -= to_read + if len(buffer): + buffer = buffer + data + else: + buffer = data if self._stopped: break - length -= to_read - if len(buffer): - buffer = buffer + data - else: - buffer = data - if self._stopped: - break - obj = self.deserialize_message(buffer) - self.on_message(obj) - except asyncio.IncompleteReadError: - if not self._stopped: - self.on_error_future.set_exception( - Exception("Connection closed while reading from the driver") - ) - break - await asyncio.sleep(0) - - await self._proc.communicate() - self._stopped_future.set_result(None) + obj = self.deserialize_message(buffer) + self.on_message(obj) + except asyncio.IncompleteReadError: + if not self._stopped: + self.on_error_future.set_exception( + Exception("Connection closed while reading from the driver") + ) + break + await asyncio.sleep(0) + + await self._proc.communicate() + finally: + # Release waiters on wait_until_stopped() even if this task was + # cancelled before reaching the end (e.g. by asyncio.run()'s + # task-cancellation phase that runs before asyncio-atexit hooks). + if not self._stopped_future.done(): + self._stopped_future.set_result(None) def send(self, message: Dict) -> None: assert self._output diff --git a/playwright/_impl/_video.py b/playwright/_impl/_video.py index 68dedf6f8..e991803e3 100644 --- a/playwright/_impl/_video.py +++ b/playwright/_impl/_video.py @@ -13,7 +13,7 @@ # limitations under the License. import pathlib -from typing import TYPE_CHECKING, Union +from typing import TYPE_CHECKING, Optional, Union from playwright._impl._artifact import Artifact from playwright._impl._helper import Error @@ -23,49 +23,34 @@ class Video: - def __init__(self, page: "Page") -> None: + def __init__(self, page: "Page", artifact: Optional[Artifact]) -> None: self._loop = page._loop self._dispatcher_fiber = page._dispatcher_fiber self._page = page - self._artifact_future = page._loop.create_future() - if page.is_closed(): - self._page_closed() - else: - page.on("close", lambda page: self._page_closed()) + self._is_remote = page._connection.is_remote + self._artifact = artifact def __repr__(self) -> str: return f"