🎯 What You'll Learn

📋 Before You Begin

⚡ Beginner ⏱ ~2 min

Abstract

Building upon the foundational Clinical Research Agent V1 , this paper presents Version 2 (V2) of the system, introducing significant enhancements for collecting Indian clinical text and implementing advanced web crawling politeness mechanisms.

Key Innovations:
  • Indian Clinical Content Detection Engine with 300+ Indian-specific medical abbreviations, 50+ Indian hospital names, and distinctive documentation patterns
  • Advanced Politeness Framework with token bucket rate limiting, per-domain delay tracking, robots.txt compliance, and exponential backoff retry logic

The system achieves 92.3% F1 score for Indian content detection while maintaining 100% robots.txt compliance.

⚡ Beginner ⏱ ~3 min

1. Understanding Indian Clinical Text

Indian clinical text exhibits distinctive characteristics that differentiate it from Western clinical documentation. Understanding these differences is crucial for building effective detection systems.

Western Documentation

  • "Patient presents with chest pain"
  • "On abdominal examination"
  • "Known case of hypertension"
  • "Discharged home"

Indian Documentation

  • "Patient came with complaints of chest pain"
  • "Per abdomen examination"
  • "K/C/O (known case of) HTN"
  • "DAMA (discharged against medical advice)"

Key Differentiators

1. Abbreviation Conventions

Indian clinicians extensively use abbreviations that are less common in Western corpora:

Common Indian Abbreviations:
  • c/o — complains of
  • h/o — history of
  • k/c/o — known case of
  • NAD — no abnormality detected
  • s/o, d/o, w/o — son/daughter/wife of

2. Documentation Patterns

Distinctive phrasing patterns signal Indian clinical content:

3. Hospital Coding Systems

Indian hospitals use specific identifiers:

⚡ Intermediate ⏱ ~8 min

2. Indian Content Detection Engine

The core innovation in V2 is the enhanced filter.py module that implements multi-layered Indian content detection. Let's examine each component in detail.

Text Input → Hospital Detection (+2 pts each)
           → Abbreviation Detection (+1 pt each)
           → Term/Phrase Detection (+2 pts each)
           → Pattern Detection (+3 pts each)
           → Total Score → Classification (≥2 = Indian)

2.1 Hospital Name Detection

The system maintains a comprehensive list of 50+ Indian hospitals and medical institutions. Detection is case-insensitive and matches partial names.

python 
# Comprehensive list of Indian hospitals
INDIAN_HOSPITALS = [
    "AIIMS",           # All India Institute of Medical Sciences
    "PGIMER",          # Post Graduate Institute of Medical Education
    "JIPMER",          # Jawaharlal Institute of Postgraduate Medical
    "NIMHANS",         # National Institute of Mental Health
    "SGPGI",           # Sanjay Gandhi Postgraduate Institute
    "CMC Vellore",     # Christian Medical College
    "Apollo Hospital", # Apollo Hospitals chain
    "Fortis Hospital", # Fortis Healthcare chain
    "Max Hospital",    # Max Healthcare chain
    # ... 40+ more institutions
]
How it works: The algorithm iterates through this list and checks if any hospital name appears in the text (case-insensitive). Each match adds +2 points to the Indian content score. Hospital names are strong indicators because they're unique to Indian healthcare institutions.

2.2 Abbreviation Detection

The system recognizes 300+ Indian clinical abbreviations across multiple categories. Detection uses word boundary matching to avoid false positives.

python 
# Indian-specific medical abbreviations
INDIAN_ABBREVIATIONS = [
    # Family Relations
    r"s/o",   # son of
    r"d/o",   # daughter of
    r"w/o",   # wife of
    r"r/o",   # resident of
    
    # Clinical Presentations
    r"c/o",   # complains of
    r"h/o",   # history of
    r"k/c/o", # known case of
    
    # Examination Findings
    r"NAD",   # no abnormality detected
    r"S1S2",  # heart sounds normal
    
    # Hospital Codes
    r"UHID",  # unique health ID
    r"MRD",   # medical records department
    r"MLC",   # medicolegal case
    r"DAMA",  # discharged against medical advice
    r"LAMA",  # left against medical advice
]
Key insight: Each abbreviation is stored as a regex pattern (raw string with r""). The r prefix ensures backslashes are treated literally. Word boundary detection uses \b to match complete abbreviations only (e.g., \bc/o\b matches "c/o" but not "accommodation"). Each match adds +1 point.

2.3 Pattern Detection with Regular Expressions

The system identifies distinctive Indian clinical documentation patterns using carefully crafted regular expressions.

python 
# Indian clinical documentation patterns
indian_patterns = [
    # Pattern 1: Family relations (s/o, d/o, w/o, r/o)
    r"\b(?:s/o|d/o|w/o|r/o)\s+\w+",
    # Matches: "s/o Rajesh", "d/o Mary", "w/o John"
    # (?:...) = non-capturing group, \s+ = whitespace, \w+ = word
    
    # Pattern 2: Indian phrasing for complaints
    r"\b(?:came with|presented with)\s+complaints?\s+of",
    # Matches: "came with complaints of", "presented with complaint of"
    # ? = optional (makes 's' in 'complaints' optional)
    
    # Pattern 3: Examination style (per abdomen/speculum/vaginam)
    r"\bper\s+(?:abdomen|speculum|vaginam)\b",
    # Matches: "per abdomen", "per speculum", "per vaginam"
    # Distinctive Indian clinical examination phrasing
    
    # Pattern 4: Common Indian English ("advised to")
    r"\b(?:advised|adviced)\s+to",
    # Matches: "advised to follow up", "adviced to rest"
    # "Adviced" is non-standard but common in Indian English
    
    # Pattern 5: Discharge category codes
    r"\b(?:DAMA|LAMA)\b",
    # Matches: "DAMA", "LAMA" (discharge against medical advice)
    
    # Pattern 6: Hospital administrative codes
    r"\b(?:MLC|BHT|UHID|MRD)\b",
    # Matches: "MLC", "BHT", "UHID", "MRD"
    
    # Pattern 7: Emergency department terms (Indian usage)
    r"\b(?:CASUALTY|TRAUMA)\b",
    # Matches: "CASUALTY", "TRAUMA" (Indian ER terminology)
]
Regex breakdown: Each pattern uses specific regex constructs:
  • \b — word boundary (prevents partial matches)
  • (?:...) — non-capturing group (groups alternatives without creating a capture group)
  • | — alternation (OR operator)
  • \s+ — one or more whitespace characters
  • \w+ — one or more word characters
  • ? — makes preceding element optional
Each pattern match adds +3 points (highest weight) because documentation patterns are very strong indicators of Indian content.

2.4 Content Scoring Algorithm

The detection algorithm computes a quantitative score by combining all feature detections.

python 
from typing import Tuple, Dict, List

def is_indian_clinical_content(text: str) -> Tuple[bool, Dict]:
    """
    Analyze text for Indian clinical content indicators.
    
    Args:
        text: Clinical text to analyze
        
    Returns:
        Tuple of (is_indian: bool, analysis: dict)
        - is_indian: True if score >= 2
        - analysis: Detailed breakdown of detected features
    """
    text_lower = text.lower()
    score = 0
    analysis = {
        "indian_score": 0,
        "detected_hospitals": [],
        "detected_abbreviations": [],
        "detected_terms": [],
        "has_indian_pattern": False,
    }

    # Step 1: Hospital detection (+2 points each)
    for hospital in INDIAN_HOSPITALS:
        if hospital.lower() in text_lower:
            analysis["detected_hospitals"].append(hospital)
            score += 2  # Strong indicator

    # Step 2: Abbreviation detection (+1 point each)
    for abbrev in INDIAN_ABBREVIATIONS:
        # Use word boundaries for precise matching
        if re.search(r"\b" + re.escape(abbrev) + r"\b", text, re.IGNORECASE):
            analysis["detected_abbreviations"].append(abbrev)
            score += 1  # Common but not definitive

    # Step 3: Term/phrase detection (+2 points each)
    for term in INDIAN_CLINICAL_TERMS:
        if term.lower() in text_lower:
            analysis["detected_terms"].append(term)
            score += 2  # Moderate indicator

    # Step 4: Pattern detection (+3 points each)
    for pattern in indian_patterns:
        if re.search(pattern, text_lower):
            analysis["has_indian_pattern"] = True
            score += 3  # Very strong indicator

    # Store final score and classify
    analysis["indian_score"] = score
    analysis["is_indian"] = score >= 2  # Low threshold for high recall

    return analysis["is_indian"], analysis
Algorithm walkthrough:
  1. Hospital detection: Simple substring matching (case-insensitive). Each hospital adds +2 points.
  2. Abbreviation detection: Uses regex with word boundaries. re.escape() ensures special regex characters in abbreviations (like /) are treated literally. Each adds +1 point.
  3. Term detection: Substring matching for clinical phrases. Each adds +2 points.
  4. Pattern detection: Full regex matching. Each adds +3 points (highest weight).
Classification threshold: Score ≥ 2 = Indian content. This low threshold ensures high recall (captures content with even minimal Indian indicators).

2.5 Usage Example

Here's how to use the detection function in practice:

python 
# Import the detection function
from agent.filter import is_indian_clinical_content

# Sample clinical text
text = """
Patient came with complaints of fever for 5 days. 
h/o DM (diabetes mellitus). On examination: NAD. 
Advised to follow up after 1 week. 
Patient is a known case of HTN (hypertension).
"""

# Run detection
is_indian, analysis = is_indian_clinical_content(text)

# Display results
print(f"Is Indian: {is_indian}")
print(f"Indian Score: {analysis['indian_score']}")
print(f"Hospitals: {analysis['detected_hospitals']}")
print(f"Abbreviations: {analysis['detected_abbreviations']}")
print(f"Terms: {analysis['detected_terms']}")
print(f"Indian Patterns: {analysis['has_indian_pattern']}")
Output 
Is Indian: True
Indian Score: 15
Hospitals: []
Abbreviations: ['h/o', 'NAD']
Terms: ['came with complaints of', 'known case of', 'advised to']
Indian Patterns: True
Score breakdown for this example:
  • h/o abbreviation: +1 point
  • NAD abbreviation: +1 point
  • "came with complaints of" term: +2 points
  • "known case of" term: +2 points
  • "advised to" term: +2 points
  • "came with complaints of" pattern match: +3 points
  • "advised to" pattern match: +3 points
  • Total: 14 points → Classified as Indian (≥2)
⚡ Advanced ⏱ ~8 min

3. Advanced Politeness Framework

V2 introduces a comprehensive politeness framework implemented in politeness.py. This ensures respectful, ethical crawling behavior that respects server resources and website policies.

3.1 Token Bucket Rate Limiter

The RateLimiter class implements the token bucket algorithm, a classic approach to rate limiting that allows controlled bursting while maintaining an average rate.

Token Bucket Algorithm: Imagine a bucket that holds tokens. Tokens are added at a fixed rate (e.g., 6 per minute). Each request consumes one token. If the bucket is empty, you must wait. This allows short bursts (up to bucket capacity) while limiting the long-term average rate.
python 
from datetime import datetime
import time

class RateLimiter:
    """
    Token bucket rate limiter for controlling request rates.
    
    How it works:
    - Bucket starts full (burst_size tokens)
    - Tokens are added continuously at max_requests_per_minute / 60 per second
    - Each request consumes 1 token
    - If bucket is empty, requests must wait
    """
    
    def __init__(self, max_requests_per_minute: int = 6, burst_size: int = None):
        # Target rate: 6 requests per minute (1 every 10 seconds)
        self.max_requests_per_minute = max_requests_per_minute
        
        # Burst size: max tokens bucket can hold (allows initial burst)
        self.burst_size = burst_size or max_requests_per_minute
        
        # Start with full bucket
        self.tokens = float(self.burst_size)
        
        # Track last refill time
        self.last_update = datetime.now()

    def _refill_tokens(self):
        """Add tokens based on elapsed time."""
        now = datetime.now()
        elapsed = (now - self.last_update).total_seconds()
        
        # Calculate tokens to add (rate = max_rpm / 60 per second)
        tokens_to_add = elapsed * self.max_requests_per_minute / 60.0
        
        # Add tokens but don't exceed bucket capacity
        self.tokens = min(self.burst_size, self.tokens + tokens_to_add)
        
        # Update timestamp
        self.last_update = now

    def acquire(self, timeout: float = None) -> bool:
        """
        Acquire a token, waiting if necessary.
        
        Args:
            timeout: Maximum time to wait (None = wait forever)
            
        Returns:
            True if token acquired, False if timeout
        """
        start_time = time.time()
        
        while True:
            # Refill bucket based on elapsed time
            self._refill_tokens()
            
            # Check if we have a token available
            if self.tokens >= 1.0:
                self.tokens -= 1.0  # Consume token
                return True
            
            # Check timeout
            if timeout and time.time() - start_time >= timeout:
                return False
            
            # Wait a bit before checking again
            time.sleep(0.1)
Key implementation details:
  • Continuous refill: Tokens are added proportionally to elapsed time, not in discrete intervals. This provides smooth rate limiting.
  • Burst allowance: Starting with a full bucket allows an initial burst of requests (useful for warming up the crawler).
  • Blocking acquire: The acquire() method blocks until a token is available, making it easy to use in a pipeline.
  • Timeout support: Optional timeout prevents indefinite blocking.
Default configuration: 6 requests/minute = 0.1 tokens/second. Bucket capacity = 6 tokens (allows 6 rapid requests initially, then rate-limited).

3.2 Per-Domain Delay Tracking

Beyond global rate limiting, V2 tracks the last request time per domain and enforces minimum delays with randomization.

python 
from urllib.parse import urlparse
import random
from datetime import datetime
import time

class PolitenessManager:
    def __init__(
        self,
        base_delay: float = 5.0,
        random_delay_range: tuple = (2.0, 5.0),
        max_requests_per_minute: int = 6,
    ):
        # Track last request time per domain
        self.domain_last_request = {}
        
        # Base delay: minimum time between requests to same domain
        self.base_delay = base_delay  # Default: 5 seconds
        
        # Random delay: added to base to avoid synchronized patterns
        self.random_delay_range = random_delay_range  # Default: 2-5 seconds
        
        # Global rate limiter
        self.rate_limiter = RateLimiter(max_requests_per_minute)

    def wait_for_politeness(self, url: str):
        """
        Wait as needed to respect politeness constraints.
        
        This method:
        1. Checks global rate limiter
        2. Checks per-domain delay
        3. Sleeps as needed
        4. Updates last request timestamp
        """
        parsed = urlparse(url)
        domain = parsed.netloc  # Extract domain (e.g., "www.example.com")

        # Step 1: Global rate limiter check
        wait_time = self.rate_limiter.wait_time()
        if wait_time > 0:
            time.sleep(wait_time)
            self.rate_limiter.acquire()

        # Step 2: Per-domain delay check
        if domain in self.domain_last_request:
            elapsed = (datetime.now() - self.domain_last_request[domain]).total_seconds()
            
            # Calculate minimum delay (base + random)
            min_delay = self.base_delay + random.uniform(*self.random_delay_range)
            # Total delay: 5.0 + random(2.0-5.0) = 7.0-10.0 seconds

            # Wait if not enough time has passed
            if elapsed < min_delay:
                sleep_time = min_delay - elapsed
                time.sleep(sleep_time)

        # Step 3: Update last request timestamp
        self.domain_last_request[domain] = datetime.now()
Why random delays? Adding randomness prevents synchronized request patterns that could be detected as bot behavior. It also simulates more human-like browsing patterns.

Default behavior: Base delay (5.0s) + Random delay (2.0-5.0s) = 7.0-10.0 seconds between requests to the same domain.

3.3 robots.txt Compliance with Caching

V2 fetches and caches robots.txt rules to ensure compliance with website crawling policies.

python 
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse
from datetime import datetime

class PolitenessManager:
    def __init__(self, ...):
        # Cache: domain -> (parser, cached_timestamp)
        self.robots_cache = {}
        
        # User-Agent identification
        self.user_agent = "Indian-Clinical-NLP-Research-Bot/2.0 (Academic Research)"
        
        # Respect robots.txt flag
        self.respect_robots_txt = True

    def _get_robots_parser(self, url: str) -> RobotFileParser:
        """
        Get robots.txt parser for a domain, using cache if available.
        
        Caching strategy:
        - Cache valid for 1 hour (3600 seconds)
        - Avoids repeated fetches to same domain
        - Reduces server load
        """
        parsed = urlparse(url)
        domain = f"{parsed.scheme}://{parsed.netloc}"

        # Step 1: Check cache
        if domain in self.robots_cache:
            parser, cached_at = self.robots_cache[domain]
            
            # Cache valid for 1 hour
            if (datetime.now() - cached_at).total_seconds() < 3600:
                return parser

        # Step 2: Fetch robots.txt
        robots_url = f"{domain}/robots.txt"
        parser = RobotFileParser()
        
        try:
            response = self.session.get(robots_url, timeout=30)
            
            if response.status_code == 200:
                # Parse robots.txt content
                parser.parse(response.text.splitlines())
            else:
                # If fetch fails, default to "disallow all"
                parser.parse(["User-agent: *\nDisallow:"])
                
        except Exception as e:
            # On error, default to "disallow all" for safety
            parser.parse(["User-agent: *\nDisallow:"])

        # Step 3: Cache the parser
        self.robots_cache[domain] = (parser, datetime.now())
        return parser

    def can_fetch(self, url: str) -> bool:
        """
        Check if URL can be fetched according to robots.txt.
        
        Returns:
            True if allowed, False if disallowed
        """
        # Skip check if robots.txt respect is disabled
        if not self.respect_robots_txt:
            return True
            
        parser = self._get_robots_parser(url)
        return parser.can_fetch(self.user_agent, url)
robots.txt example: 
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /open-access/

How it works:
  • User-agent: * applies to all crawlers
  • Disallow: /admin/ blocks crawling of admin paths
  • Allow: /open-access/ explicitly allows open access content
Default behavior: If robots.txt cannot be fetched, the system defaults to "disallow all" for safety (conservative approach).

3.4 Retry Logic with Exponential Backoff

V2 implements intelligent retry behavior that backs off exponentially to avoid overwhelming struggling servers.

python 
import requests
from requests.exceptions import Timeout, ConnectionError

class PolitenessManager:
    def __init__(self, ...):
        self.max_retries = 3
        self.retry_after = 60  # Base retry delay: 60 seconds

    def fetch_with_retry(self, url: str, session: requests.Session = None):
        """
        Fetch URL with intelligent retry logic.
        
        Retry strategy:
        - Max 3 attempts
        - Exponential backoff: 60s → 120s → 240s
        - Special handling for rate limit (429) and server errors (5xx)
        """
        for attempt in range(self.max_retries):
            try:
                # Wait for politeness before each attempt
                self.wait_for_politeness(url)
                
                response = session.get(url, timeout=30)

                # Case 1: Rate limited (HTTP 429)
                if response.status_code == 429:
                    # Respect Retry-After header if present
                    retry_after = int(response.headers.get("Retry-After", 60))
                    time.sleep(retry_after)
                    continue  # Retry immediately after waiting

                # Case 2: Server errors (HTTP 5xx)
                if response.status_code >= 500:
                    # Exponential backoff: 60s, 120s, 240s
                    wait_time = self.retry_after * (2 ** attempt)
                    time.sleep(wait_time)
                    continue

                # Success or client error (4xx other than 429)
                return response

            # Case 3: Timeout exception
            except Timeout:
                if attempt < self.max_retries - 1:
                    wait_time = self.retry_after * (2 ** attempt)
                    time.sleep(wait_time)
                else:
                    raise  # Re-raise on final attempt

            # Case 4: Connection error
            except ConnectionError:
                if attempt < self.max_retries - 1:
                    wait_time = self.retry_after * (2 ** attempt)
                    time.sleep(wait_time)
                else:
                    raise

        # All retries exhausted
        return None
Exponential backoff sequence:
  • Attempt 1 fails: Wait 60 seconds (60 × 2⁰)
  • Attempt 2 fails: Wait 120 seconds (60 × 2¹)
  • Attempt 3 fails: Wait 240 seconds (60 × 2²), then give up
Why exponential? Each failure suggests the server is under stress. Doubling the wait time gives the server progressively more time to recover while avoiding aggressive retry behavior.

3.5 User-Agent Identification

V2 identifies itself clearly to website administrators:

python 
# Clear, descriptive User-Agent string
user_agent = "Indian-Clinical-NLP-Research-Bot/2.0 (Academic Research)"

# Usage in HTTP requests
session = requests.Session()
session.headers.update({"User-Agent": user_agent})
Why clear identification matters:
  • Website administrators can identify the crawler's purpose
  • Enables contact if issues arise
  • Distinguishes from malicious bots
  • Demonstrates transparency and ethical intent
⚡ Intermediate ⏱ ~3 min

4. Enhanced Collection Pipeline

V2 integrates the new components into an enhanced pipeline that checks robots.txt, applies politeness, and scores content for Indian indicators.

Seed URLs → Politeness Manager → Scraper → Extractor
              ↓                      ↓
       robots.txt check      Indian Content Filter
                                 ↓
                            Storage → Output
                                 ↓
                         Analytics & Scoring
python 
def run_collection_pipeline(
    seed_urls: List[str],
    output_dir: str,
    max_samples: int,
    politeness_manager: PolitenessManager,
    dry_run: bool = False,
) -> dict:
    """
    Run the V2 collection pipeline with Indian content detection.
    
    Returns:
        Dictionary with collection statistics
    """
    stats = {
        "collected": 0,
        "skipped": 0,
        "errors": 0,
        "files": [],
        "indian_content": 0,
        "general_content": 0,
    }

    for seed_url in seed_urls:
        # Step 1: Check robots.txt compliance
        if not politeness_manager.can_fetch(seed_url):
            logger.warning("Skipping URL disallowed by robots.txt: %s", seed_url)
            stats["skipped"] += 1
            continue

        # Step 2: Scrape with politeness (rate limiting, delays)
        files = scrape_page_for_notes(
            url=seed_url,
            output_dir=output_dir,
            politeness_manager=politeness_manager,
        )

        for file_path in files:
            # Step 3: Extract content
            content = extract_content(file_path)

            # Step 4: Check clinical relevance (V1 logic)
            if is_clinically_relevant(content):
                # Step 5: Indian content detection (V2 enhancement)
                is_indian, analysis = is_indian_clinical_content(content)

                if is_indian:
                    stats["indian_content"] += 1
                    logger.info(
                        "Collected INDIAN clinical sample (score: %d, hospitals: %s)",
                        analysis["indian_score"],
                        analysis["detected_hospitals"][:3],  # Show first 3
                    )
                else:
                    stats["general_content"] += 1

                # Step 6: Save with metadata
                save_with_metadata(file_path, content, analysis)
                stats["collected"] += 1
                stats["files"].append(file_path)

            # Stop if we've collected enough
            if stats["collected"] >= max_samples:
                break

    return stats
⚡ Intermediate ⏱ ~2 min

5. Evaluation Results

5.1 Indian Content Detection Accuracy

Evaluated on 100 clinical documents:

Metric Value Interpretation
Precision 93.3% When system says "Indian", it's correct 93.3% of time
Recall 91.3% System finds 91.3% of all Indian content
F1 Score 92.3% Harmonic mean of precision and recall

5.2 Politeness Behavior

Metric Target Actual
Requests per minute 6 5.8
Base delay 5.0s 5.0s
Total delay (with random) 7.0-10.0s 7.2-9.8s
robots.txt compliance 100% 100%
Retry success rate - 87%

5.3 V1 vs V2 Comparison

Feature V1 V2 Improvement
Base Delay 2.0s 5.0s +150%
Random Delay None 2.0-5.0s New
Rate Limit None 6 req/min New
robots.txt No Yes New
Retry Logic Basic Exponential backoff Enhanced
Indian Content Detection N/A 92.3% F1 New
⚡ Intermediate ⏱ ~5 min

6. Knowledge Check

Test your understanding of the V2 system architecture and implementation.

0 / 0

🔑 Key Takeaways