10 list comprehension bottleneck sandbox

.github/workflows/ci.yml

@@ -46,7 +46,12 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install setuptools --upgrade
-          pip install ./
+          pip install cython numpy  # Install build dependencies first
+          pip install -e .
+
+      - name: Verify installation
+        run: |
+          python verify_installation.py
 
       - name: Run tests
         run: |
@@ -77,9 +82,15 @@ jobs:
       run: |
         source venv/bin/activate
         python -m pip install --upgrade pip
-        pip install ./
+        pip install cython numpy  # Install build dependencies first
+        pip install -e .
+
+    - name: Verify installation
+      run: |
+        source venv/bin/activate
+        python verify_installation.py
 
     - name: Run tests
       run: |
         source venv/bin/activate
-        python -m unittest discover -s tests
+        python -m unittest discover -s tests

.gitignore

@@ -172,3 +172,12 @@ cython_debug/
 
 # Ignore vscode settings
 .vscode/
+
+*prof
+
+# Add to .gitignore
+*.c
+*.html
+*.so
+*.pyd
+build/

CONTRIBUTING.md

@@ -26,6 +26,26 @@ pre-commit install # this will install a pre-commit hook into the git repo
 You can also/alternatively run `pre-commit run` (changes only) or
 `pre-commit run --all-files` to check even without installing the hook.
 
+## Building Cython Extensions (Optional but Recommended)
+
+For optimal performance during development, build the Cython extensions:
+
+```bash
+# Install Cython if not already installed
+pip install cython
+
+# Build the extensions
+./build_cython.sh
+
+# Or manually
+python setup_cython.py build_ext --inplace
+```
+
+**When to rebuild:**
+- After modifying any `.pyx` files (e.g., `src/ModularCirc/HelperRoutines.pyx`)
+- After pulling changes that include `.pyx` modifications
+- When switching between branches with different Cython code
+
 # Testing
 
 This repo uses `unittest` for testing. You can run locally the tests by running the following command:

CYTHON_README.md

@@ -0,0 +1,112 @@
+# Cythonizing ModularCirc HelperRoutines
+
+This directory contains a Cythonized version of the `HelperRoutines` module for improved performance.
+
+## Quick Start (Automatic Build)
+
+**The Cython extension is now built automatically during package installation!**
+
+```bash
+# Install with Cython support (requires cython and numpy)
+pip install cython numpy
+pip install -e .
+```
+
+The setup will automatically detect Cython and build the extension. If Cython is not available or the build fails, the package will fall back to the Numba implementation.
+
+## Manual Build (Development)
+
+For quick rebuilds during development without reinstalling the entire package:
+
+```bash
+bash build_cython.sh
+```
+
+Or manually:
+
+```bash
+python setup_cython.py build_ext --inplace
+```
+
+## Prerequisites
+
+- Python >= 3.10
+- Cython >= 3.0
+- NumPy >= 1.20
+- C compiler (gcc, clang, or MSVC)
+
+## Using the Cython Module
+
+### Option 1: Automatic (Recommended)
+
+Add this at the top of the script:
+```python
+# Try to import Cython version, fall back to Numba if unavailable
+try:
+    from ModularCirc.HelperRoutines import resistor_model_flow
+    print("Using Cythonized HelperRoutines")
+except ImportError:
+    # Fall back to current Numba implementation
+    pass
+```
+
+### Option 2: Manual Import (where function hasn't been cythonised)
+
+In any module that uses HelperRoutines:
+
+```python
+try:
+    from ModularCirc.HelperRoutines.HelperRoutinesCython import resistor_model_flow, chamber_volume_rate_change
+except ImportError:
+    from ModularCirc.HelperRoutines.HelperRoutines import resistor_model_flow, chamber_volume_rate_change
+```
+
+## Performance Benefits
+
+Cython provides:
+- **No JIT compilation overhead**: Functions are pre-compiled to machine code
+- **Faster setup time**: No Numba compilation delays on first run
+- **C-level performance**: Direct C math library calls (`sqrt`, `exp`, `log`, etc.)
+- **Type safety**: Compile-time type checking
+- **GIL release**: Many functions use `nogil` for better multi-threading potential
+
+Expected improvements:
+- **Setup time**: Near-instant (no JIT compilation)
+- **Runtime**: Comparable to or faster than Numba (0-15% improvement typical)
+- **Memory**: Slightly lower memory footprint
+
+## Debugging
+
+If compilation fails, check:
+
+1. **Compiler availability**: Ensure you have a C compiler (gcc, clang, or MSVC)
+2. **NumPy headers**: Make sure NumPy is installed: `pip install numpy`
+3. **Cython version**: Use Cython >= 0.29: `pip install --upgrade cython`
+
+View detailed annotation (optimization opportunities):
+
+```bash
+# After building, check the generated HTML file
+open src/ModularCirc/HelperRoutines.html
+```
+
+## Cleanup
+
+To remove compiled artifacts:
+
+```bash
+# Remove compiled extensions
+rm -f src/ModularCirc/HelperRoutines/HelperRoutinesCython*.so
+rm -f src/ModularCirc/HelperRoutines/HelperRoutinesCython*.pyd
+rm -f src/ModularCirc/HelperRoutines/HelperRoutines.c
+
+# Remove build artifacts
+rm -rf build/
+```
+
+## Notes
+
+- The `.pyx` file maintains API compatibility with the original `.py` file
+- All Numba `@nb.njit` decorators are replaced with Cython equivalents
+- Type annotations use Cython's static typing for maximum performance
+- Functions marked `nogil` can run without the Python GIL, enabling true parallelism

INSTALLATION_OPTIONS.md

@@ -0,0 +1,132 @@
+# ModularCirc Installation Options
+
+ModularCirc supports two performance backends for the `HelperRoutines` module:
+
+1. **Cython** (C-compiled, faster startup, recommended for production)
+2. **Numba** (JIT-compiled, slower startup, easier development)
+
+## Installation Methods
+
+### Option 1: Install with Cython (Recommended)
+
+For best performance with faster startup times:
+
+```bash
+# Install with Cython dependencies
+pip install -e .[performance]
+
+# Build the Cython extension
+python setup.py build_ext --inplace
+```
+
+### Option 2: Install with Numba only
+
+For development or if you encounter Cython build issues:
+
+```bash
+# Set environment variable to skip Cython build
+export MODULARCIRC_USE_CYTHON=0
+
+# Install package
+pip install -e .
+```
+
+Or install normally - if Cython is not available, it will automatically fall back to Numba.
+
+## Runtime Configuration
+
+You can control which implementation is used at runtime with environment variables:
+
+### Force Numba implementation
+
+Even if Cython is installed, you can force the use of Numba:
+
+```bash
+export MODULARCIRC_FORCE_NUMBA=1
+python your_script.py
+```
+
+Or in Python:
+
+```python
+import os
+os.environ['MODULARCIRC_FORCE_NUMBA'] = '1'
+
+import ModularCirc  # Will use Numba
+```
+
+### Enable verbose output
+
+To see which implementation is being used:
+
+```bash
+export MODULARCIRC_VERBOSE=1
+python your_script.py
+```
+
+Or:
+
+```python
+import os
+os.environ['MODULARCIRC_VERBOSE'] = '1'
+
+import ModularCirc
+# Will print: "✓ Using Cythonized HelperRoutines" or "⚠ Using Numba HelperRoutines"
+```
+
+### Check which implementation is active
+
+In your Python code:
+
+```python
+from ModularCirc.HelperRoutines import USING_CYTHON
+
+if USING_CYTHON:
+    print("Using fast Cython implementation")
+else:
+    print("Using Numba implementation")
+```
+
+## Environment Variables Summary
+
+| Variable | Values | Default | Description |
+|----------|--------|---------|-------------|
+| `MODULARCIRC_USE_CYTHON` | 0 or 1 | 1 | Controls whether to build Cython extension during installation |
+| `MODULARCIRC_FORCE_NUMBA` | 0 or 1 | 0 | Forces use of Numba implementation at runtime |
+| `MODULARCIRC_VERBOSE` | 0 or 1 | 0 | Enables verbose output about which implementation is used |
+
+## Performance Comparison
+
+- **Cython**: No JIT compilation overhead, faster startup (~2-3x faster first run)
+- **Numba**: JIT compilation on first use, slightly slower startup but similar runtime performance
+
+For production use or when running many small simulations, Cython is recommended.
+For development or when C compiler is not available, Numba is a good fallback.
+
+## Troubleshooting
+
+### Cython build fails
+
+If you encounter errors building the Cython extension:
+
+```bash
+# Disable Cython and use Numba
+export MODULARCIRC_USE_CYTHON=0
+pip install -e .
+```
+
+### Want to rebuild Cython extension
+
+```bash
+# Clean and rebuild
+python setup.py clean --all
+python setup.py build_ext --inplace
+```
+
+### Verify installation
+
+```bash
+python verify_installation.py
+```
+
+This will show which implementation is active and available.

MANIFEST.in

@@ -0,0 +1,7 @@
+include README.md
+include LICENSE
+include CONTRIBUTING.md
+include pyproject.toml
+include setup.py
+recursive-include src/ModularCirc *.pyx *.pxd
+recursive-include src/ModularCirc *.py