fix(runner): add mutex to testAndSet to prevent race condition

[usernametooshort]

Summary

Fixes a race condition in testAndSet where the read-check-then-write sequence is not atomic.

Problem

testAndSet performs:

1. seen(k) - check if key exists
2. setSeen(k) - set the key

Without synchronization, two concurrent goroutines can both pass the seen() check before either calls setSeen(), causing duplicate processing of the same target.

Impact

This race occurs in processTargets() where multiple goroutines call testAndSet() for discovered TLS SubjectAN/SubjectCN values (lines 1692, 1697, 1706, 1746, 1751).

When the race triggers:

• Same target gets processed multiple times
• Duplicate output entries
• Wasted HTTP requests and resources

Fix

Add a sync.Mutex to the Runner struct to serialize testAndSet operations.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced the robustness of the deduplication system to ensure consistent behavior during concurrent operations.

• Tests

  • Added comprehensive unit tests for the deduplication method, including concurrent execution scenarios to validate correctness.

[neo-by-projectdiscovery-dev]

Neo - PR Security Review

No security issues found

Highlights

• Added unit tests for testAndSet function (basic behavior validation)
• Added concurrent race condition test with 100 goroutines to verify mutex correctness

_{Comment @pdneo help for available commands. · Open in Neo}

[coderabbitai]

Walkthrough

Added a mutex field to the Runner struct and wrapped the deduplication logic in testAndSet with lock/unlock operations to ensure atomic checking and setting of seen state during concurrent execution.

Changes

Cohort / File(s)	Summary
Synchronization Primitive runner/runner.go	Added seenMux sync.Mutex field to Runner struct and wrapped the key deduplication sequence (trimming, validation, checking, and setting seen state) with mutex locking to ensure thread-safe atomic operations.
Concurrent Testing runner/runner_test.go	Added TestRunner_testAndSet for basic functionality (first insertion, duplicates, whitespace handling) and TestRunner_testAndSet_concurrent spawning 100 goroutines to verify exactly one caller succeeds while others fail, validating the mutex-protected logic under concurrent access.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Poem

🐰 A mutex now guards the seen's sacred gate,
No race conditions shall seal the fate,
One hundred hops all synchronized true—
Where only the first shall break on through! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and clearly describes the main change: adding a mutex to the testAndSet method to prevent race conditions, which is the core fix in the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Mzack9999]

The PR should point to the dev branch

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

runner/runner.go (1)

1705-1714: ⚠️ Potential issue | 🟠 Major

Don't disable CSPProbe on the shared scan options.

processItem commonly passes a shared *ScanOptions into process, so this write is visible to sibling goroutines. The first response that reaches this branch can turn CSP expansion off for unrelated in-flight targets, and the read/write on scanopts.CSPProbe itself races. Clone the options for the recursive follow-up and clear the flag on that clone only.

Suggested fix

 if scanopts.CSPProbe && result.CSPData != nil {
-	scanopts.CSPProbe = false
+	childScanopts := scanopts.Clone()
+	childScanopts.CSPProbe = false
 	domains := result.CSPData.Domains
 	domains = append(domains, result.CSPData.Fqdns...)
 	for _, tt := range domains {
 		if !r.testAndSet(tt) {
 			continue
 		}
-		r.process(tt, wg, hp, protocol, scanopts, output)
+		r.process(tt, wg, hp, protocol, childScanopts, output)
 	}
 }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@runner/runner.go` around lines 1705 - 1714, The code currently clears the
shared flag scanopts.CSPProbe before spawning recursive work, causing a data
race and turning off CSP expansion for other goroutines; instead, create a
shallow copy of the ScanOptions (e.g., newOpts := *scanopts; newOpts.CSPProbe =
false) and pass &newOpts into r.process so only the recursive follow-up sees the
flag cleared, leaving the original scanopts unmodified; update the branch
handling CSP expansion (the block that reads scanopts.CSPProbe and calls
r.process) to use the cloned options when invoking r.process.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@runner/runner.go`:
- Around line 1705-1714: The code currently clears the shared flag
scanopts.CSPProbe before spawning recursive work, causing a data race and
turning off CSP expansion for other goroutines; instead, create a shallow copy
of the ScanOptions (e.g., newOpts := *scanopts; newOpts.CSPProbe = false) and
pass &newOpts into r.process so only the recursive follow-up sees the flag
cleared, leaving the original scanopts unmodified; update the branch handling
CSP expansion (the block that reads scanopts.CSPProbe and calls r.process) to
use the cloned options when invoking r.process.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cb4fbdaf-34fe-4960-9a4f-c1fc2dbf4afc

📥 Commits

Reviewing files that changed from the base of the PR and between ecef472 and 2e901d0.

📒 Files selected for processing (2)

• runner/runner.go
• runner/runner_test.go