diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..87e1bd0c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,137 @@ +# AGENTS.md - Coding Agent Guidelines + +Guidelines for AI coding agents working on `actor-scraper`, Apify's monorepo of generic scraper Actors. + +## Start here: pick the right playbook + +Read this file first for context, then the principles that apply to your task. They state constraints and the reasoning behind them, not steps to follow — work the steps out yourself. + +| Your task | Read | +| ------------------------------------------ | ---------------------------------------------------------------------------- | +| Fixing a bug, regression, or failing test | [docs/agents/fixing-bugs.md](docs/agents/fixing-bugs.md) | +| Implementing a feature or new input option | [docs/agents/implementing-features.md](docs/agents/implementing-features.md) | + +Both playbooks share one rule that overrides convenience: **Crawlee is ours — we maintain it, so investigate it, fix it there by preference, and never route around it.** Details in each playbook. + +## What this repo is + +A monorepo of the **generic scrapers**: Apify's configure-don't-code Actors, where the user supplies a start URL, some crawl settings, and a `pageFunction` — a JavaScript function as a string — and the Actor runs it against every page. + +| Package | Published as | Engine | +| ------------------------------------------- | ---------------------------- | ------------------------------------------------------ | +| `packages/actor-scraper/web-scraper` | `apify/web-scraper` | Puppeteer, `pageFunction` runs **in the browser page** | +| `packages/actor-scraper/puppeteer-scraper` | `apify/puppeteer-scraper` | Puppeteer, `pageFunction` runs in Node | +| `packages/actor-scraper/playwright-scraper` | `apify/playwright-scraper` | Playwright | +| `packages/actor-scraper/cheerio-scraper` | `apify/cheerio-scraper` | Raw HTTP + Cheerio | +| `packages/actor-scraper/jsdom-scraper` | `apify/jsdom-scraper` | Raw HTTP + jsdom | +| `packages/actor-scraper/camoufox-scraper` | `apify/camoufox-scraper` | Camoufox (anti-detection Firefox) | +| `packages/actor-scraper/sitemap-scraper` | `apify/sitemap-extractor` | HTTP HEAD over sitemaps | +| `packages/scraper-tools` | `@apify/scraper-tools` (npm) | Shared runtime for all of the above | + +Three consequences worth internalizing before you change anything: + +- **`pageFunction` and its context object are a public API.** Thousands of user-written scripts run against them. Every property on the context (`$`, `page`, `request`, `log`, `globalStore`, `skipLinks`, `enqueueRequest`, `saveSnapshot`, …) is something someone's script already calls. Removing, renaming, or retyping one breaks live scrapers silently. +- **`INPUT_SCHEMA.json` is a public API too**, for the same reason, plus saved tasks on the platform reference option names directly. +- **These Actors are old, heavily used, and full of deliberate backward compatibility.** `Apify` aliased to `Actor` on the context, `RequestQueueV2`, string-eval'd hooks — those are not oversights. Do not "modernize" them as a side effect of another change. + +## Crawlee + +The scrapers are thin wrappers around [Crawlee](https://crawlee.dev/), which **Apify maintains — we own it**. This is not a third-party black box; treat it as an upstream part of this codebase. When a change could be made either here or in Crawlee, Crawlee is the preferred home for it. + +- **Establish what Crawlee already does before writing crawling, queueing, session, proxy, retry, or fingerprinting code.** +- **Check before building anything Crawlee-shaped**, including closed issues and merged PRs. It may already exist in a newer build, be implemented and unreleased, or have an agreed design worth following instead of inventing your own. Note which major it landed in. +- **Extend rather than reimplement**, reaching for the cheapest mechanism that works: configuration, then hooks (how the existing Actors add nearly everything, including the user's own string-supplied hooks), then subclassing, then an upstream change. +- **If Crawlee can't express what you need, that's an upstream change, not a local workaround.** +- **Betas are normal, not a risk.** Build on the beta that has what you need. Never bump a Crawlee **major** — crossing the v3/v4 line is a migration project with its own decision, never a step inside a feature. +- **Never** fork or vendor Crawlee into `src/`, monkey-patch it, reimplement a queue or session pool or retry policy it already provides, or reach into private and `@internal` fields to get at something unexported. Export it upstream instead. + +# Implementing features + +Read [AGENTS.md](../../AGENTS.md) first for repo and platform context. These are the principles for new features, new input options, and behavior changes — not a procedure. Work out the steps yourself; these are the constraints the result has to satisfy. + +## The public surface is additive-only + +**`INPUT_SCHEMA.json`** is a public API — saved tasks and integrations reference option names directly. Add; never rename, retype, or repurpose. Default to existing behavior so current tasks are unaffected. It's loaded and validated at runtime, so a mismatch between schema and code is a runtime failure, not a compile error. `title` and `description` render in the input form, so they're user-facing copy. When the same capability lands in more than one Actor it must have the same name and semantics in both — users move between these Actors. + +**The `pageFunction` context** in [packages/scraper-tools/src/context.ts](../../packages/scraper-tools/src/context.ts) is the most sensitive surface in the repo: it's the entire API user scripts get. Additive only — never remove, rename, or retype an existing property, because something out there calls it. Things users shouldn't touch are hidden behind Symbols on purpose, not merely undocumented. A new property should exist on every engine's context, or be clearly engine-specific in the way `$` and `page` already are; silently present-on-some is the worst outcome for users. For `web-scraper` it also has to exist on the browser side in [bundle.browser.ts](../../packages/actor-scraper/web-scraper/src/internals/bundle.browser.ts), and anything non-serializable needs a bridge — plan for that before designing the API, not after. + +Report what you changed, which Actors and packages it affects, what you verified, and anything you deliberately left out. Don't commit or push. + +## Fixing bugs + +Read [AGENTS.md](../../AGENTS.md) first for repo and platform context. These are the principles for bug fixes, regressions, and failing tests — not a procedure. Work out the steps yourself; these are the constraints the result has to satisfy. + +## Smallest diff wins + +A bug fix is not an opportunity to improve the code around it. No refactoring, renaming, reorganizing, extracting helpers, reformatting untouched code, or "while I'm here" cleanups. + +## Know your blast radius before you edit + +A fix in an Actor's own `crawler_setup.ts` affects one Actor. A fix in `packages/scraper-tools` ships to all seven and to `@apify/scraper-tools` on npm, so it needs correspondingly higher confidence and a check that the other Actors don't depend on the current behavior. Say which one you're doing when you report. + +Report what the root cause was, what you changed, which Actors are affected, and what you verified. If something still fails, say so with the output. Don't commit or push. + +## Repository layout + +- `packages/actor-scraper/*` — one directory per Actor. Each holds `src/main.ts` (a one-liner calling `runActor`), `src/internals/crawler_setup.ts` (all the wiring), `src/internals/consts.ts` (`Input` type and enums), `INPUT_SCHEMA.json`, `Dockerfile`, `README.md`. +- `packages/scraper-tools/src` — the shared runtime: + - [run_actor.ts](packages/scraper-tools/src/run_actor.ts) — the `Actor.main()` wrapper every Actor's entry point calls + - [context.ts](packages/scraper-tools/src/context.ts) — the `Context` class exposed to `pageFunction`; private members are hidden behind Symbols on purpose + - [tools.ts](packages/scraper-tools/src/tools.ts) — `evalFunctionOrThrow` / `evalFunctionArrayOrThrow` (user code compiled via `node:vm`), input validation, request metadata + - [browser_tools.ts](packages/scraper-tools/src/browser_tools.ts) — snapshots and browser-side plumbing + - [consts.ts](packages/scraper-tools/src/consts.ts) — shared constants such as `META_KEY` +- `test/scraper-tools/` — unit tests (vitest) +- `test/e2e/` — end-to-end tests; each `test/e2e/scrapers//test.mjs` runs a real Actor against real pages +- `scripts/` — build copy step and the platform build trigger +- `docs/agents/` — these playbooks + +`web-scraper` is the odd one out: [src/internals/bundle.browser.ts](packages/actor-scraper/web-scraper/src/internals/bundle.browser.ts) is compiled and injected into the page, so its `pageFunction` and context live in the browser, with a bridge back to Node. Changes there need to be made on both sides. + +## Commands + +Package manager is **pnpm** (version pinned in `packageManager`). Builds go through turbo; do not use npm or yarn. + +```bash +pnpm build # turbo run build across all packages +pnpm test # vitest unit tests +pnpm test:e2e # end-to-end scraper tests (real runs, slow) +pnpm tsc-check-tests # type-check the test project +pnpm lint # oxlint --type-aware +pnpm lint:fix # oxlint --type-aware --fix +pnpm format # oxfmt +pnpm format:check # oxfmt --check +pnpm clean # turbo run clean +``` + +To run a single Actor locally, from its package directory: + +```bash +pnpm start +``` + +Notes: + +- Linting is **oxlint**, formatting is **oxfmt** — not ESLint or Prettier. Husky plus lint-staged runs both on commit. +- Commit messages must follow conventional commits (commitlint enforces it); release notes are generated from them. +- Vitest aliases `@apify/scraper-tools` to `packages/scraper-tools/src`, so unit tests see your source changes without a build. E2E tests do not — build first. + +### Testing + +```bash +pnpm exec vitest run test/scraper-tools/tools.test.ts # single unit test file +pnpm exec vitest run -t "test name" # filter by name +node test/e2e/runScraperTests.mjs # all e2e tests +``` + +### Errors and logging + +- Use `log` from `@apify/log` (already re-exported onto the `pageFunction` context). +- Errors thrown out of a `pageFunction` are surfaced per-request by Crawlee's retry machinery — do not swallow them in the wrapper. +- Reserve fatal errors for genuinely unrecoverable setup problems, such as invalid input. + +## Working agreements + +- **Do not commit or push unless explicitly asked.** Leave changes in the working tree and report what you changed. +- This is a **public, Apache-2.0 repository** with external contributors — see [CONTRIBUTING.md](CONTRIBUTING.md). Code and comments are read by people outside Apify. +- Actor `README.md` files are the store pages users read; they are user-facing copy, not internal notes. `docs/*.md` is owned by a documentation code owner ([.github/CODEOWNERS](.github/CODEOWNERS)). +- A change that touches `packages/scraper-tools` touches **every** Actor. Say so explicitly when reporting it.