fix: update sitemap URL extraction to deal with different type of sit…

[saimsajidirl]

…emap urls

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[biswajeetdev]

Trying multiple candidate URLs is the right fix — sitemap_index.xml and sitemap-index.xml are common patterns that the single-URL approach was silently missing.

Two small observations:

1. Function name
clean_default_sitemap_urls implies it is cleaning/normalising a URL, but it is actually generating a list of candidates. Something like default_sitemap_candidates or get_default_sitemap_urls would be clearer to a future reader.

2. Early return on first successful fetch
The function returns [sm_url] on the first URL that fetch_content returns a truthy value for. Depending on how fetch_content behaves, some hosts serve a 200 with an HTML error page at /sitemap_index.xml — those would be treated as a found sitemap and short-circuit the search before reaching the real sitemap.xml. If fetch_content validates content-type or XML structure this is not an issue, but worth a comment if it only checks HTTP status.

[saimsajidirl]

Good catch on both points.

1. I agree the current name is a bit misleading. The function started as a URL normalization helper and evolved into generating candidate sitemap URLs. Renaming it to something like get_default_sitemap_urls or default_sitemap_candidates would make the intent much clearer.

2. That's a fair concern. Right now the early return assumes that a successful fetch_content indicates a valid sitemap. If fetch_content only validates the HTTP response and not the actual content, an HTML error page served with a 200 could indeed cause a false positive and prevent checking the remaining candidates. In our case, sitemap validation happens downstream, but I'll add a comment to make that assumption explicit (or move the validation closer to the fetch logic if it makes sense).