qtap: Improve bailout, refactor browser launch, fix Safari multi file

Major:

* qtap,server: Streamline error handling and signal handling to handle
  various kinds of errors that previously went unhandled, bypassing
  teardown, or caused us to get stuck waiting for nothing.

  In particular, "Could not open <file>" is now propagated to a
  top-level error (e.g. run throws/rejects) rather than a test-level
  "bail". This means we emit "error" instead of "finish" and
  "clientresult", and other tests are cancelled.

  With this in place, other uncaught errors we can handle other errors
  better as well, such as errors from reporters.

* qtap: Handle uncaught errors from reporters, which otherwise
  cause emit() to throw. Provide a dedicated EventEmitter proxy for
  each reporter, to attribute the reporter in the error message.
  This also protects our EventEmitter object from a reporter tampering
  with "emit" or "on". The proxy only implements "on" because there
  is no use case for reporters calling "emit", and it is simpler this
  way.

* tap-finished: Add "bailout" as finish signal, and fix stuckness
  when calling end().

* server: Separate browser launch from TAP consumption.

  Starting the tapParser right away inside the launch attempt meant
  we threw an exception for browser connect timeout, we have no way
  to produce a finalResult / clientresult event. This is why bailout
  was separate, but this is messy to deal with in reporters and also
  makes partial results hard to transfer.

  Instead, handle our bailout the same way as TAP bailout: As part
  of the clientresult event, not as a separate event. Upstream tap-parser
  emits finalResult#bailout, we can do the same in clientresult.

  For this to work we need to make tap-finished trigger on bailout,
  which it didn't until now.

  The benefit of this is also that there is no longer any risk of
  multiple browser attempts communicating over the same TAP stream
  and producing duplicate or conflicting results because we do not
  attach the TAP stream until after the browser has connected.

  For additional disambiguation we also mint the clientId during the
  attempt, instead of sharing it across retries.

* browsers: Fix safari() bug when running multiple test files.

* reporter: Flesh out more of the default reporter,
  and refine the events we emit to ease the work of reporters.

  * Remove separate "bail" event to simplify reporter code.
    Make this part of "finish" and "clientresult" events instead.
  * Emit all "assert" events, not just the first.

Minor:

* browsers: Rename Headless Firefox => Firefox Headless.
* client: Set `window.qunit_config_reporters_html = false`.
* client: Add basic visual `<pre>` output in debug mode.
* server: Change clientId format from "client_1" to "client_S1_C1"
  to make verbose logs more useful. This makes it obvious which
  clients are for the same testFile, and thus ControlServer.
* qtap: De-duplicate browsers and test files.

Author: Krinkle

API.md

@@ -155,70 +155,77 @@ async function mybrowser (url, signals) {
 }
 ```
 
-## QTap basic events
+## QTap summary events
 
-### Event: `'error'`
+These are emitted at most once for the run overall.
 
-* `error <Error|string>`
+### Event: `'clients'`
 
-### Event: `'finish'`
+The `clients` event conveys which browsers are being started, and which tests will be run. It is emitted as soon as QTap has validated the parameters. Each client is a browser process that runs one test suite. For example, if you run 2 test suites in 3 different browsers, there will be 6 clients.
+
+* `event.clients {Object<string,Object>}` Keyed by clientId
+  * `clientId {string}` An identifier unique within the current qtap process (e.g. `client_123`).
+  * `testFile {string}` Relative file path or URL (e.g. `test/index.html` or `http://localhost/test/`).
+  * `browserName {string}` Browser name, as specified in config or CLI (e.g. `firefox`).
+  * `displayName {string}` Browser pretty name (e.g. "Headless Firefox").
 
-Summary event that is ideal for when you run one test suite in one browser, or if you otherwise don't need a break down of results by client.
+### Event: `'error'`
+
+* `error {Error|string}`
 
-* `event.ok <boolean>` Aggregate status of each client's results. If any failed, this is false.
-* `event.exitCode <number>` Suggested exit code, 0 for success, 1 for failed.
-* `event.total <number>` Aggregated from `result` events.
-* `event.passed <number>` Aggregated from `result` events.
-* `event.failed <number>` Aggregated from `result` events.
-* `event.skips <array>` Carried from the first client that failed, or empty.
-* `event.todos <array>` Carried from the first client that failed, or empty.
-* `event.failures <array>` Carried from the first client that failed, or empty.
-* `event.bailout <false|string>` Carried from the first client that failed, or false.
+### Event: `'finish'`
 
-## QTap reporter events
+Summary event based on the `clientresult` events. This is mutually exclusive with `error`.
 
-A client will emit each of these events only once, except `consoleerror` which may be emitted any number of times.
+* `event.ok {boolean}` Aggregate status of each client's results. If any failed or bailed, this is false.
+* `event.exitCode {number}` Suggested exit code, 0 for success, 1 for failed or bailed.
+* `event.total {number}` Aggregated from `clientresult` events.
+* `event.passed {number}` Aggregated from `clientresult` events.
+* `event.failed {number}` Aggregated from `clientresult` events.
 
-### Event: `'client'`
+## QTap client events
 
-The `client` event is emitted when a client is created. A client is a dedicated browser instance that runs one test suite. For example, if you run 2 test suites in 3 different browsers, there will be 6 clients.
+These are emitted once per client, except `clientconsole` and `clientassert` which may be emitted many times by a client during a test run.
 
-* `event.clientId <string>` An identifier unique within the current qtap process (e.g. `client_123`).
-* `event.testFile <string>` Relative file path or URL (e.g. `test/index.html` or `http://localhost/test/`).
-* `event.browserName <string>` Browser name, as specified in config or CLI (e.g. `firefox`).
-* `event.displayName <string>` Browser pretty name, (e.g. "Headless Firefox").
+### Event: `'clientonline'`
 
-### Event: `'online'`
+The `clientonline` event is emitted when a browser has successfully started and opened the test file. If a browser fails to start or connect, then the `error` event is emitted instead.
 
-The `online` event is emitted when a browser has successfully started and opened the test file. If a browser fails to connect, a `bail` event is emitted instead.
+* `event.clientId {string}`
+* `event.testFile {string}`
+* `event.browserName {string}`
+* `event.displayName {string}`
 
-* `event.clientId <string>`
+### Event: `'clientresult'`
 
-### Event: `'result'`
+The `clientresult` event is emitted when a browser has completed a test run. This includes if it bailed mid-run, such as when a test run times out.
 
-The `result` event is emitted when a browser has completed a test run. This is mutually exclusive with the `bail` event.
+* `event.clientId {string}`
+* `event.ok {boolean}`
+* `event.total {number}`
+* `event.passed {number}`
+* `event.failed {number}`
+* `event.skips {array}` Details about skipped tests (count as passed).
+* `event.todos {array}` Details about todo tests (count as passed).
+* `event.failures {array}` Details about failed tests.
+* `event.bailout {false|string}`
 
-* `event.clientId <string>`
-* `event.ok <boolean>`
-* `event.total <number>`
-* `event.passed <number>`
-* `event.failed <number>`
-* `event.skips <array>` Details about skipped tests (count as passed).
-* `event.todos <array>` Details about todo tests (count as passed).
-* `event.failures <array>` Details about failed tests.
+### Event: `'clientconsole'`
 
-### Event: `'bail'`
+The `clientconsole` event relays any warnings and uncaught errors from the browser console. These are for debug purposes only, and do not cause a test run to fail per-se. A complete and successful test run, may nonetheless print warnings or errors to the console.
 
-The `bail` event is emitted when a browser was unable to start or complete a test run.
+Note that test frameworks such as QUnit may catch global errors during a test
 
-* `event.clientId <string>`
-* `event.reason <string>`
+It is recommended that reporters only display console errors if a test run failed (i.e. there was a failed test result, or an uncaught error).
 
-### Event: `'consoleerror'`
+* `event.clientId {string}`
+* `event.message {string}`
 
-The `consoleerror` event relays any warning or error messages from the browser console. These are for debug purposes only, and do not indicate that a test has failed. A complete and successful test run, may nonetheless print warnings or errors to the console.
+### Event: `'assert'`
 
-It is recommended that reporters only display console errors if a test run failed (i.e. there was a failed test result, or the cilent bailed).
+The `assert` event describes a single test result (whether passing or failing). This can be used by reporters to indicate activity, display the name of a test in real-time, or to convey failures early.
 
-* `event.clientId <string>`
-* `event.message <string>`
+* `event.clientId {string}`
+* `event.ok {boolean}`
+* `event.fullname {string}`
+* `event.diag {undefined|Object}`

ARCHITECTURE.md

@@ -436,7 +436,7 @@ Safari has long resisted the temptation to offer a reasonable command-line inter
   {"value":null}
   ```
 
-  This addresses all previous concerns, and seems to be the best as of 2025. The only downside is that it requires a bit more code to setup (find available port, and perform various HTTP requests).
+  This addresses all previous concerns, and seems to work best as of 2025. The only downside is that it requires more code to set up (find available port, and perform various HTTP requests).
 
   - https://webkit.org/blog/6900/webdriver-support-in-safari-10/
   - https://developer.apple.com/documentation/webkit/macos-webdriver-commands-for-safari-12-and-later

bin/qtap.js

@@ -1,6 +1,8 @@
 #!/usr/bin/env node
 'use strict';
 
+import util from 'node:util';
+
 import { program, InvalidArgumentError } from 'commander';
 import qtap from '../src/qtap.js';
 
@@ -36,8 +38,8 @@ program
       + 'This is ignored when testing by URL instead of file.'
   )
   .option('--timeout <number>',
-    'The maximum timeout of any single test.\n'
-      + 'The test is stopped if it takes longer than this between results.',
+    'The maximum duration of a single unit test.\n'
+      + 'The test is stopped if the browser is idle longer than this between results.',
     function (val) {
       const num = Number(val);
       if (num < 0 || !Number.isFinite(num)) {
@@ -58,7 +60,11 @@ program
     },
     60
   )
-  .option('-r, --reporter <reporter>', 'One of "minimal", "dynamic", or "none".', 'minimal')
+  .option('-r, --reporter <reporter>',
+    'Set one or more reporters.\n'
+      + 'Available: "minimal", "dynamic", "none", or a custom reporter.',
+    'minimal'
+  )
   .option('-w, --watch', 'Watch files for changes and re-run the test suite.')
   .option('-d, --debug', 'Enable debug mode. This keeps the browser open,\n'
       + 'and for local browsers it will launch visibly instead of headless.')
@@ -90,7 +96,37 @@ if (opts.version) {
     });
     process.exit(result.exitCode);
   } catch (e) {
-    console.error(e);
+    console.log(
+      '\n'
+      + util.styleText('bgRedBright',
+        util.styleText('redBright', '__')
+        + util.styleText(['whiteBright', 'bold'], 'ERROR')
+        + util.styleText('redBright', '__')
+      )
+      + '\n'
+    );
+    if (e instanceof qtap.QTapError && e.qtapClient) {
+      console.error(util.styleText('grey',
+        `Bail out from ${e.qtapClient.testFile} in ${e.qtapClient.browser}:`
+      ));
+    }
+    if (e instanceof qtap.QTapError) {
+      // Omit internal stack trace for QTapError, not relevant to end-users,
+      // by printing `e.toString()` instead of `e`.
+      //
+      // Omit internal "BrowserStopSignal" prefix from messages.
+      const message = e.name === 'BrowserStopSignal' ? e.message : e.toString();
+      // Bold first line, grey the rest
+      // "foo"               > "<b>foo</b>"
+      // "foo \n bar \n baz" > "<b>foo</b> \n <grey>bar \n baz</grey>"
+      const formatted = message
+        .replace(/^.+?$/m, (m) => util.styleText('bold', m))
+        .replace(/\n(^.+)$/ms, (m, rest) => '\n' + util.styleText('grey', rest));
+      console.error(formatted);
+    } else {
+      // Print including full stack trace
+      console.error(e);
+    }
     process.exit(1);
   }
 }

eslint.config.js

@@ -25,14 +25,15 @@ export default [
     }
   },
   {
-    files: ['test/*.js'],
+    files: ['test/**/*.js'],
     languageOptions: {
       globals: {
         QUnit: 'readonly'
       }
     },
     ...qunitRecommended,
     rules: {
+      'no-var': 'off',
       'object-property-newline': 'off',
     }
   },

package.json

@@ -26,15 +26,15 @@
     "commander": "12.1.0",
     "tap-parser": "18.0.0",
     "which": "5.0.0",
-    "yaml": "^2.4.1"
+    "yaml": "^2.8.1"
   },
   "devDependencies": {
     "@types/node": "22.10.5",
     "@types/which": "3.0.4",
     "eslint": "~8.57.1",
     "eslint-config-semistandard": "~17.0.0",
     "eslint-plugin-qunit": "^8.1.2",
-    "qunit": "2.24.1",
+    "qunit": "2.24.3",
     "semistandard": "~17.0.0",
     "typescript": "5.7.3"
   },

src/browsers.js

@@ -118,7 +118,7 @@ async function firefox (url, signals, logger, debugMode) {
   const profileDir = LocalBrowser.makeTempDir(signals, logger);
   const args = [url, '-profile', profileDir, '-no-remote', '-wait-for-browser'];
   if (!debugMode) {
-    firefox.displayName = 'Headless Firefox';
+    firefox.displayName = 'Firefox Headless';
     args.push('-headless');
   }
 
@@ -164,7 +164,7 @@ firefox.displayName = 'Firefox';
 function makeChromium (displayName, getPaths) {
   /** @type {Browser} - https://github.com/microsoft/TypeScript/issues/22063 */
   const chromium = async function (url, signals, logger, debugMode) {
-    chromium.displayName = debugMode ? displayName : `Headless ${displayName}`;
+    chromium.displayName = debugMode ? displayName : `${displayName} Headless`;
     // https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md
     const dataDir = LocalBrowser.makeTempDir(signals, logger);
     const args = [
@@ -215,8 +215,6 @@ const detect = async function (url, signals, logger, debugMode) {
 };
 
 export default {
-  LocalBrowser,
-
   detect,
   firefox,
   chrome,

src/client.cjs

@@ -4,10 +4,12 @@
 function qtapClientHead () {
   // Support QUnit 2.24+: Enable TAP reporter, declaratively.
   window.qunit_config_reporters_tap = true;
+  window.qunit_config_reporters_html = false;
 
   // Cache references to original methods, to avoid getting trapped by mocks (e.g. Sinon)
   var setTimeout = window.setTimeout;
   var XMLHttpRequest = window.XMLHttpRequest;
+  var createTextNode = document.createTextNode && document.createTextNode.bind && document.createTextNode.bind(document);
 
   // Support IE 9: console.log.apply is undefined.
   // Don't bother with Function.apply.call. Skip super call instead.
@@ -67,6 +69,7 @@ function qtapClientHead () {
   function createBufferedWrite (url) {
     var buffer = '';
     var isSending = false;
+    var debugElement = false;
     function send () {
       var body = buffer;
       buffer = '';
@@ -81,8 +84,16 @@ function qtapClientHead () {
       };
       xhr.open('POST', url, true);
       xhr.send(body);
+
+      // Optimization: Only check this once, during the first send
+      if (debugElement === false) {
+        debugElement = document.getElementById('__qtap_debug_element') || null;
+      }
+      if (debugElement) {
+        debugElement.appendChild(createTextNode(body));
+      }
     }
-    return function write (str) {
+    return function writeTap (str) {
       buffer += str + '\n';
       if (!isSending) {
         isSending = true;