arXiv fetch is unreliable

[TimidRobot]

Description

The arXiv fetch script pulls license information from two unreliable fields:

quantifying/scripts/1-fetch/arxiv_fetch.py

Lines 353 to 369 in 6482d63

	def extract_license_info(entry):
	"""
	Extract CC license information from ArXiv paper entry.
	Checks rights field first, then summary field for license patterns.
	Returns normalized license identifier or "Unknown".
	"""
	# checking through the rights field first then summary
	if hasattr(entry, "rights") and entry.rights:
	license_info = normalize_license_text(entry.rights)
	if license_info != "Unknown":
	return license_info
	if hasattr(entry, "summary") and entry.summary:
	license_info = normalize_license_text(entry.summary)
	if license_info != "Unknown":
	return license_info
	return "Unknown"

1. rights
   • ‼️ Does not exist. There is no <rights> entry in 5.2. Details of Atom Results Returned
2. summary
   • Relies on a text match. However, the summary may talk about any number of legal tools without the paper itself using any of those legal tools.

Reproduction

Add a logging statement to extract_license_info() function or examine the schema.

For example, a result for the query [ALL:](all:"CC BY") includes the following summary for [2008.00774v3] Elsevier OA CC-By Corpus:

We introduce the Elsevier OA CC-BY corpus. This is the first open corpus of
Scientific Research papers which has a representative sample from across
scientific disciplines. This corpus not only includes the full text of the
article, but also the metadata of the documents, along with the bibliographic
information for each reference.

Here we can see the paper is about CC BY works, but the paper is licensed arXiv.org - Non-exclusive license to distribute, not CC BY.

Expectation

License information should be reliable.

Near as I can tell, the API does not provide licensing information. Licensing details would have to be scraped from each article's web page. Unfortunately, the structured RDF data on each article's web page doesn't include licensing information.

Web scraping is not currently in scope for this project.

Additional context

• License and copyright - arXiv info

Resolution

• I would be interested in resolving this bug.

[jessbryte]

A possible solution would be to search each url in the license_identifier in using open search aggregator and return the identifier and the total results.

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
   1000
</opensearch:totalResults>

[Opsmithe]

@TimidRobot. After analyzing this issue, I've identified a fundamental solution that directly addresses the reliability concerns raised. The legacy ArXiv API endpoint (http://export.arxiv.org/api/query) indeed lacks structured license metadata, forcing the current implementation to rely on unreliable text pattern matching in summary field. This produces false positives when papers discuss CC licenses rather than use them. To fix this, I have migrated the script to ArXiv's OAI-PMH API (https://oaipmh.arxiv.org/oai) which provides Direct XML field with structured license. I believe this helps to eliminate false positives through precise license detection.

Key Findings

About OAI-PMH API Search Capabilities:

The API OAI-PMH API uses the following metadata Exposure Methods
(https://info.arxiv.org/help/oa/index.html)

1. arXivRaw (Internal Format)
   For our CC license quantification, arXivRaw could provide richer temporal data but doesn't solve the core limitation of incomplete license metadata population.

2. arXivOld (Legacy Format)
   This offers a simplified structure and a License field available, but a flat author list (not structured) and minimal metadata. For our overall goal of showing CC license adoption trends, this format is not entirely suitable

3. oai_dc (Dublin Core)
   This Standard bibliographic format but No license field which is a major limitation for our use case

4. arXiv (ArXiv Format)
   This format offers a structured license field, with detailed author information, categories, dates, abstracts. Overall I have adopted this format in the new implementation of the arxiv_fetch.py script for the following reasons

• Structured license field - Direct access to <license> element
• Structured authors - Individual <author> elements with keyname/forenames
• Complete metadata - <categories>,< dates>,<abstracts>, <doi>,<journal refs>
• Current format - Actively maintained and updated
• Reasonable size - Not as heavy as arXivRaw that also has with version history

Available Search/Filter Options:

1. Date-based harvesting: for example from=2024-01-01&until=2024-01-02
2. Category-based sets: physics, math, cs, etc.
3. Metadata formats.

This approach should give us some structured, reliable license data instead of text pattern matching, but requires harvesting larger datasets and filtering client-side. This datasource has about 2,877,361 papers as of November 6, 2025. Estimated Full Dataset Size: ~8.5 GB of XML metadata for all 2.8M papers, 3,500 records per page (based on resumption token) and ~822 API calls needed to harvest everything. Using the earliest date (2005-09-16) would create a 100x larger processing job with diminishing returns for CC license detection. The API provides higher quality data but requires a harvest-and-filter approach rather than targeted searching.

Implementation

I have considered the following approach to get concrete licenses

1. Initial harvest by year/category instead of full dataset
2. Focus on recent papers (last 5-10 years) where CC licensing is more common
3. The script uses extract_license_from_xml() which:
   • Parses XML with ET.fromstring(record_xml)
   • Extracts from structured field: root.find(".//{http://arxiv.org/OAI/arXiv/}license")
   • Maps URLs to standardized identifiers via LICENSE_MAPPING()
   • Only processes papers with actual CC licenses (filters out ArXiv's default license)

Key API information.

• API Key: No API key required
• Large dataset size: 2.8M+ records require significant processing time
• Rate limiting: 3-second delays between requests recommended
• Version handling: Only latest version exposed (unless using arXivRaw format)

Running the script

using the --years-back 10 args to give a broader distribution, the script was tested using the given command

 pipenv run python scripts/1-fetch/arxiv_fetch.py --limit 200 --years-back 4 --enable-save

Accurate Data: From 200 papers sampled (2022-2025), it found:

• CC BY 4.0: 143 papers (71.5%)
• CC BY-NC-ND 4.0: 32 papers (16%)
• CC BY-NC-SA 4.0: 12 papers (6%)
• CC BY-SA 4.0: 5 papers (2.5%)
• CC0 1.0: 8 papers (4%)

[Opsmithe]

A possible solution would be to search each url in the license_identifier in using open search aggregator and return the identifier and the total results.

<opensearch:totalResults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
   1000
</opensearch:totalResults>

Hi @jessbryte. I analyzed your approach here, and whilst I think that this is also good and introduces dynamism with OpenSearch, I think for our use case here, the current approach serves the purpose, returning more than 90% of the possible licenses mapped using the static method I used.

Of course, there are broad implications of using OpenSearch, but considering the network overhead, single point of failure with OpenSearch, and several other factors, I believe the static approach is more suitable for this project for the following reasons:

• Network latency for each license lookup would significantly slow down the fetch process
• OpenSearch service downtime would break the entire data collection pipeline
• Additional complexity in error handling, retries, and fallback mechanisms
• Dependency on external infrastructure that may not always be available
• The current static mapping already handles 90%+ of encountered licenses effectively
• ArXiv license URLs follow predictable patterns and don't change frequently
• No risk of incorrect mappings from potentially stale or inconsistent search data
• Full control over license identifier standards and formats

reliability and simplicity often outweigh the theoretical benefits of dynamic systems. The current method proves effective in practice, and I believe we should optimize for maintainability and robustness rather than introducing additional complexity for marginal gains

That said, I appreciate the suggestion and would be open to revisiting this approach if we encounter significant numbers of unmapped licenses in future data collection or if otherwise indulged by @TimidRobot