⚡️ Speed up function _is_inside_unrequested_special_dir by 532%

[codeflash-ai]

📄 532% (5.32x) speedup for _is_inside_unrequested_special_dir in src/datasets/data_files.py

⏱️ Runtime : 4.11 milliseconds → 650 microseconds (best of 6 runs)

📝 Explanation and details

The optimized code achieves a 532% speedup (4.11ms → 650μs) by eliminating the heavyweight PurePath object creation overhead and replacing it with lightweight string operations.

Key Performance Optimizations

1. Eliminated PurePath Object Construction
The original code creates two PurePath objects per function call, which involves significant parsing and validation overhead. The optimized version uses simple string operations (rfind, split) that operate directly on the input strings without any object instantiation.

2. Replaced List Comprehensions with Manual Counting
Instead of building intermediate lists with list comprehensions ([part for part in ... if part.startswith("__")]), the optimized code counts special directories in-place. This avoids memory allocation for temporary lists and the associated iteration overhead.

3. Direct String Manipulation

• Uses rfind('/') to locate the last separator (O(n) scan but no parsing)
• Uses string slicing to extract parent paths
• Uses split('/') only when needed, then iterates once to count

Why This Works

The original implementation's bottleneck (as shown in line profiler) is spending 98.8% of time in just two lines creating PurePath objects. The optimized version:

• Avoids PurePath constructor overhead entirely
• Performs minimal string operations
• Counts directly without intermediate data structures

Impact on Workloads

Based on function_references, this function is called from resolve_pattern() in a hot path - it filters every matched file path in glob results. The speedup directly benefits:

• Large codebases with many files to scan
• Patterns matching hundreds/thousands of files
• Directory structures with deep nesting (as shown in tests with 50-100 nested directories)

Test Case Performance

The optimization excels across all test scenarios:

• Simple paths: 500-700% faster (e.g., __pycache__/b.txt cases)
• Deep nesting: 400-500% faster (50-100 directory levels)
• Bulk operations: 590-656% faster (multiple file scans)
• Edge cases: 900%+ faster (single-level paths with no parent)

The optimization is particularly effective for the common case of scanning many normal directories (no special dirs), where it achieves 600-1000% speedup by quickly determining both counts are zero.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 37 Passed
🌀 Generated Regression Tests	✅ 371 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage	100.0%

⚙️ Click to see Existing Unit Tests

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
test_data_files.py::test_is_inside_unrequested_special_dir	54.7μs	9.84μs	456%✅
test_data_files.py::test_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir	53.8μs	9.29μs	478%✅

🌀 Click to see Generated Regression Tests

import pytest  # used for our unit tests
from src.datasets.data_files import _is_inside_unrequested_special_dir

def test_docstring_examples():
    # Verify the examples provided in the function docstring behave exactly as documented.
    # Example 1: special dir present in path but pattern is a global glob -> should be considered "inside unrequested special dir"
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") # 24.3μs -> 3.83μs (534% faster)

    # Example 2: pattern matches the file but does not explicitly mention special dir -> still treated as unrequested
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") # 11.0μs -> 1.64μs (569% faster)

    # Example 3: pattern explicitly mentions the special directory -> not inside an unrequested special dir
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") # 9.05μs -> 1.34μs (577% faster)

    # Example 4: pattern uses a token that itself starts with "__", this counts as explicitly requesting it -> not ignored
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") # 9.26μs -> 1.16μs (701% faster)

def test_no_special_dirs_returns_false():
    # When neither path nor pattern contain special directories, lengths of special-dir-lists are both zero -> False
    codeflash_output = _is_inside_unrequested_special_dir("dir/subdir/file.txt", "**") # 21.2μs -> 3.31μs (541% faster)
    # Also when pattern explicitly requests a normal directory, still no special dirs -> False
    codeflash_output = _is_inside_unrequested_special_dir("dir/subdir/file.txt", "dir/subdir/*") # 11.1μs -> 1.88μs (491% faster)

def test_single_special_dir_differences():
    # Path has one special directory; pattern does not -> should return True
    codeflash_output = _is_inside_unrequested_special_dir("__special__/file.txt", "*/file.txt") # 21.3μs -> 3.48μs (513% faster)

    # Path has one special directory; pattern explicitly includes same -> should return False
    codeflash_output = _is_inside_unrequested_special_dir("__special__/file.txt", "__special__/*") # 10.1μs -> 1.56μs (546% faster)

    # Path has one special directory; pattern includes a different special directory -> counts differ (1 vs 1?),
    # but here pattern has a special dir at parent position as well - test both to ensure logic is purely count-based.
    codeflash_output = _is_inside_unrequested_special_dir("__a__/file.txt", "__b__/file.txt") # 9.84μs -> 1.39μs (607% faster)

def test_multiple_nested_special_directories():
    # Path contains two nested special directories in its parent
    path = "__a__/__b__/c.txt"

    # Pattern explicitly includes both special directories in its parent -> counts equal -> False
    codeflash_output = _is_inside_unrequested_special_dir(path, "__a__/__b__/*") # 23.4μs -> 3.81μs (513% faster)

    # Pattern only includes one of the special directories -> counts unequal (2 vs 1) -> True
    codeflash_output = _is_inside_unrequested_special_dir(path, "__a__/*") # 10.9μs -> 1.71μs (539% faster)

    # Pattern includes none -> counts unequal (2 vs 0) -> True
    codeflash_output = _is_inside_unrequested_special_dir(path, "**") # 8.95μs -> 1.15μs (676% faster)

def test_special_name_variations_are_detected_by_prefix():
    # Any directory name that starts with "__" should be treated as special.
    # Here "__pycache__inner" starts with "__" and should therefore be counted.
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__inner/b.txt", "**") # 21.5μs -> 3.12μs (588% faster)
    # If pattern explicitly mentions the same shape of directory name, it counts as requested -> False
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__inner/b.txt", "__pycache__inner/*") # 11.0μs -> 1.70μs (549% faster)

def test_matched_rel_path_without_parent_parts():
    # If matched_rel_path is just a directory name (no parent), .parent.parts will be empty -> 0 special dirs
    # Example: matched_rel_path is "__pycache__" (no deeper file) so parent is '.' -> no special dirs detected
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__", "**") # 17.2μs -> 1.71μs (903% faster)

    # If pattern would have a special parent but matched_rel_path has none (above) -> counts differ -> True
    codeflash_output = _is_inside_unrequested_special_dir("file_at_root.txt", "__special__/*") # 11.3μs -> 2.13μs (432% faster)

def test_absolute_paths_and_roots():
    # Absolute paths include a root part; PurePath.parts contains the root, but it won't start with "__".
    # Path and pattern both absolute and explicitly include special -> should return False
    codeflash_output = _is_inside_unrequested_special_dir("/project/__s__/file.txt", "/project/__s__/*") # 23.1μs -> 4.37μs (428% faster)

    # Path absolute contains special but pattern does not mention special -> True
    codeflash_output = _is_inside_unrequested_special_dir("/project/__s__/file.txt", "/project/*") # 11.7μs -> 1.86μs (530% faster)

    # Pattern absolute contains special, path does not -> True
    codeflash_output = _is_inside_unrequested_special_dir("/project/normal/file.txt", "/project/__s__/*") # 10.5μs -> 1.74μs (502% faster)

def test_pattern_tokens_starting_with_double_underscore_count():
    # Pattern tokens such as "__*" (wildcards but with the "__" prefix) start with "__" and thus are considered explicit requests.
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") # 22.2μs -> 3.37μs (557% faster)

    # Similarly, pattern token "__something" should be counted
    codeflash_output = _is_inside_unrequested_special_dir("__x__/a.txt", "__something/a.txt") # 11.8μs -> 1.59μs (641% faster)

def test_large_scale_varied_counts_consistency():
    # Large-scale, deterministic (no randomness) exercise of many path/pattern combinations.
    # We create up to 300 cases with varying counts of special directories inserted deterministically.
    results = []
    cases = 300  # keep under 1000 as requested
    for i in range(cases):
        # Construct a path with (i % 5) special directories in the parent and a filename
        special_count_path = i % 5
        parts_path = ["__a__"] * special_count_path + [f"file{i}.txt"]
        matched = "/".join(parts_path)

        # Construct a pattern with ((i+1) % 5) special directories in the parent and the same filename
        special_count_pattern = (i + 1) % 5
        parts_pattern = ["__a__"] * special_count_pattern + [f"file{i}.txt"]
        pattern = "/".join(parts_pattern)

        # Expected is True iff the counts differ
        expected = (special_count_path != special_count_pattern)

        # Append actual boolean result for later aggregated assertions
        codeflash_output = _is_inside_unrequested_special_dir(matched, pattern); actual = codeflash_output # 2.62ms -> 406μs (544% faster)
        results.append(actual)

    # Sanity check on aggregate values: count how many True results we got.
    # This should equal number of i in range(cases) where i%5 != (i+1)%5.
    expected_true_count = sum(1 for i in range(cases) if (i % 5) != ((i + 1) % 5))

def test_edge_case_similar_names_not_starting_with_double_underscore():
    # Names that contain "__" but not at the start should NOT be treated as special by this function.
    # E.g., "prefix__suffix" does not start with "__" and so should not be counted.
    codeflash_output = _is_inside_unrequested_special_dir("prefix__suffix/file.txt", "**") # 20.3μs -> 2.86μs (610% faster)

    # If pattern explicitly contains a leading "__" element but path does not start with "__", counts differ -> True
    codeflash_output = _is_inside_unrequested_special_dir("prefix__suffix/file.txt", "__special__/file.txt") # 11.0μs -> 1.77μs (520% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from pathlib import PurePath

# imports
import pytest
from src.datasets.data_files import _is_inside_unrequested_special_dir

def test_basic_pycache_with_wildcard():
    """Test that __pycache__ is ignored when using ** pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "**"); result = codeflash_output # 21.3μs -> 3.07μs (594% faster)

def test_basic_pycache_with_generic_pattern():
    """Test that __pycache__ is ignored with */b.txt pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt"); result = codeflash_output # 22.0μs -> 3.51μs (525% faster)

def test_explicit_pycache_in_pattern():
    """Test that __pycache__ is NOT ignored when explicitly mentioned in pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*"); result = codeflash_output # 21.0μs -> 3.34μs (527% faster)

def test_explicit_wildcard_pycache_pattern():
    """Test that __pycache__ is NOT ignored when using __*/* pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*"); result = codeflash_output # 21.8μs -> 3.26μs (569% faster)

def test_normal_directory_no_special_dirs():
    """Test normal directory without special directories returns False."""
    codeflash_output = _is_inside_unrequested_special_dir("data/file.txt", "**"); result = codeflash_output # 20.7μs -> 2.69μs (671% faster)

def test_normal_directory_with_wildcard_pattern():
    """Test normal directory matching with explicit pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("data/file.txt", "data/*"); result = codeflash_output # 20.4μs -> 3.12μs (554% faster)

def test_single_level_normal_file():
    """Test single-level path without special directories."""
    codeflash_output = _is_inside_unrequested_special_dir("file.txt", "**"); result = codeflash_output # 18.1μs -> 1.63μs (1009% faster)

def test_nested_normal_directories():
    """Test nested normal directories with matching pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("a/b/c/file.txt", "a/b/c/*"); result = codeflash_output # 20.4μs -> 3.69μs (453% faster)

def test_multiple_special_dirs_in_path():
    """Test path with multiple special directories."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/__special__/file.txt", "**"); result = codeflash_output # 20.9μs -> 3.30μs (533% faster)

def test_multiple_special_dirs_all_explicit_in_pattern():
    """Test multiple special directories all explicitly in pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/__special__/file.txt", "__pycache__/__special__/*"); result = codeflash_output # 21.3μs -> 3.98μs (435% faster)

def test_multiple_special_dirs_partial_explicit_in_pattern():
    """Test multiple special directories with only one explicit in pattern."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/__special__/file.txt", "__pycache__/**"); result = codeflash_output # 21.7μs -> 3.82μs (470% faster)

def test_special_dir_at_different_levels():
    """Test special directory not as immediate parent."""
    codeflash_output = _is_inside_unrequested_special_dir("__special__/normal/file.txt", "**"); result = codeflash_output # 21.3μs -> 3.40μs (525% faster)

def test_special_dir_explicit_with_wildcard():
    """Test special directory explicitly mentioned with wildcard."""
    codeflash_output = _is_inside_unrequested_special_dir("__special__/file.txt", "__special__/**"); result = codeflash_output # 20.9μs -> 3.48μs (501% faster)

def test_pattern_with_special_dir_not_in_path():
    """Test pattern contains special dir that's not in the path."""
    codeflash_output = _is_inside_unrequested_special_dir("data/file.txt", "__special__/**"); result = codeflash_output # 21.6μs -> 3.32μs (550% faster)

def test_empty_parent_path():
    """Test when path has no parent directory."""
    codeflash_output = _is_inside_unrequested_special_dir("file.txt", "file.txt"); result = codeflash_output # 16.6μs -> 1.68μs (888% faster)

def test_root_level_special_dir():
    """Test special directory at root level."""
    codeflash_output = _is_inside_unrequested_special_dir("__init__/module.py", "**"); result = codeflash_output # 20.7μs -> 2.99μs (594% faster)

def test_double_underscore_in_middle():
    """Test directory name with double underscore not at start."""
    codeflash_output = _is_inside_unrequested_special_dir("my__dir/file.txt", "**"); result = codeflash_output # 20.2μs -> 2.84μs (610% faster)

def test_single_underscore_directory():
    """Test directory with single underscore is not treated as special."""
    codeflash_output = _is_inside_unrequested_special_dir("_private/file.txt", "**"); result = codeflash_output # 20.6μs -> 2.80μs (635% faster)

def test_three_underscores_directory():
    """Test directory with three underscores at start is special."""
    codeflash_output = _is_inside_unrequested_special_dir("___special/file.txt", "**"); result = codeflash_output # 20.3μs -> 2.97μs (583% faster)

def test_complex_pattern_with_multiple_wildcards():
    """Test complex pattern with multiple levels of wildcards."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/a/b/file.txt", "**/*/file.txt"); result = codeflash_output # 22.1μs -> 4.16μs (430% faster)

def test_pattern_matches_special_dir_name():
    """Test pattern that matches the special directory name."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/file.txt", "__pycache__/file.txt"); result = codeflash_output # 20.2μs -> 3.24μs (524% faster)

def test_very_deep_nesting_no_special():
    """Test very deep nesting without special directories."""
    codeflash_output = _is_inside_unrequested_special_dir("a/b/c/d/e/f/g/h/file.txt", "**"); result = codeflash_output # 22.3μs -> 3.87μs (476% faster)

def test_very_deep_nesting_one_special():
    """Test very deep nesting with one special directory."""
    codeflash_output = _is_inside_unrequested_special_dir("a/b/__special__/d/e/f/g/h/file.txt", "**"); result = codeflash_output # 22.7μs -> 3.87μs (488% faster)

def test_special_dir_with_numbers():
    """Test special directory with numbers."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__123/file.txt", "**"); result = codeflash_output # 20.8μs -> 3.04μs (583% faster)

def test_special_dir_with_underscores_only():
    """Test directory that is only underscores."""
    codeflash_output = _is_inside_unrequested_special_dir("____/file.txt", "**"); result = codeflash_output # 20.2μs -> 2.92μs (590% faster)

def test_pattern_with_exact_match_and_special_dir():
    """Test exact pattern match with special directory in path."""
    codeflash_output = _is_inside_unrequested_special_dir("__pycache__/file.txt", "__pycache__/file.txt"); result = codeflash_output # 20.5μs -> 3.37μs (506% faster)

def test_pattern_parent_different_from_path_parent():
    """Test when pattern parent doesn't match path parent structure."""
    codeflash_output = _is_inside_unrequested_special_dir("__special__/a/b/file.txt", "__special__/c/**"); result = codeflash_output # 22.8μs -> 3.96μs (477% faster)

def test_deeply_nested_path_with_multiple_special_dirs():
    """Test deeply nested path with multiple special directories scattered throughout."""
    path = "__cache__/" + "/".join([f"dir{i}" for i in range(50)]) + "/file.txt"
    pattern = "**"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 54.4μs -> 8.71μs (525% faster)

def test_long_path_no_special_dirs():
    """Test long path without special directories."""
    path = "/".join([f"dir{i}" for i in range(100)]) + "/file.txt"
    pattern = "**"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 76.8μs -> 14.5μs (431% faster)

def test_long_pattern_with_explicit_special_dirs():
    """Test long pattern explicitly mentioning special directories."""
    path = "__pycache__/" + "/".join([f"dir{i}" for i in range(50)]) + "/file.txt"
    pattern = "__pycache__/" + "/".join([f"dir{i}" for i in range(50)]) + "/*"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 61.9μs -> 14.3μs (334% faster)

def test_many_special_dirs_all_matching_pattern():
    """Test many special directories all explicitly in pattern."""
    special_parts = "/".join([f"__special{i}__" for i in range(20)])
    path = special_parts + "/file.txt"
    pattern = special_parts + "/*"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 38.0μs -> 8.57μs (344% faster)

def test_many_special_dirs_partial_in_pattern():
    """Test many special directories with only some in pattern."""
    special_parts = "/".join([f"__special{i}__" for i in range(20)])
    path = special_parts + "/file.txt"
    pattern = "/".join([f"__special{i}__" for i in range(10)]) + "/**"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 36.2μs -> 7.33μs (394% faster)

def test_alternating_special_and_normal_dirs_large():
    """Test alternating special and normal directories at scale."""
    parts = []
    for i in range(30):
        if i % 2 == 0:
            parts.append(f"__dir{i}__")
        else:
            parts.append(f"dir{i}")
    path = "/".join(parts) + "/file.txt"
    pattern = "**"
    codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 37.1μs -> 7.19μs (417% faster)

def test_performance_bulk_normal_paths():
    """Test performance with multiple normal paths."""
    test_cases = [
        ("data/file.txt", "**"),
        ("src/module/code.py", "src/**"),
        ("tests/unit/test_module.py", "tests/**/test_*.py"),
        ("docs/readme.md", "docs/*"),
        ("config/settings.json", "config/**"),
    ]
    for path, pattern in test_cases:
        codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 60.2μs -> 8.72μs (590% faster)

def test_performance_bulk_special_paths():
    """Test performance with multiple special paths."""
    test_cases = [
        ("__pycache__/file.txt", "**"),
        ("__special__/module.py", "**"),
        ("__init__/code.py", "**/**.py"),
        ("__test__/unittest.py", "**"),
        ("__config__/settings.json", "**"),
    ]
    for path, pattern in test_cases:
        codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 56.7μs -> 7.50μs (656% faster)

def test_performance_bulk_explicit_special_paths():
    """Test performance with multiple explicitly mentioned special paths."""
    test_cases = [
        ("__pycache__/file.txt", "__pycache__/*"),
        ("__special__/module.py", "__special__/*"),
        ("__init__/code.py", "__init__/**"),
        ("__test__/unittest.py", "__test__/*"),
        ("__config__/settings.json", "__config__/**"),
    ]
    for path, pattern in test_cases:
        codeflash_output = _is_inside_unrequested_special_dir(path, pattern); result = codeflash_output # 58.8μs -> 8.37μs (603% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

To edit these changes git checkout codeflash/optimize-_is_inside_unrequested_special_dir-mlcly82u and push.