NewGenRenter Scraping Project

This project uses Scrapy for web scraping rental listings.

Setup

1. Activate Virtual Environment

source venv/bin/activate

2. Install Dependencies (if setting up fresh)

pip install -r requirements.txt

3. Run Spider

python quotes_spider.py

Or if you create a proper Scrapy project:

scrapy crawl quotes

IDE Configuration

Make sure your IDE (VS Code/Cursor) is using the Python interpreter from the virtual environment:

1. Open Command Palette (Cmd+Shift+P)
2. Type "Python: Select Interpreter"
3. Choose: ./venv/bin/python

This ensures proper import resolution and code completion.

Deactivate Virtual Environment

When you're done working:

deactivate

Documentation

Comprehensive system documentation is available in the docs/ directory:

• System Architecture - Complete system design and technology stack
• Data Ingestion Pipeline - Data processing and validation systems
• Daily Automation Script - Scheduling and monitoring systems
• RAG Powered Search API - Intelligent search functionality

See the documentation README for navigation guidance and implementation roadmap.