Skip to content

PRD: Fast Tractography Slice Viewer with Pre-computed Images

Executive Summary

The current tractography slice GUI (tractography_slice_gui.py + visualizeTractographySlices.m) suffers from significant performance bottlenecks that make real-time slice navigation impractical. Each slice change triggers expensive MATLAB computations, including track intersection calculations, FA background rendering, and track filtering, resulting in 5-30 second delays per slice update.

This PRD outlines the development of a high-performance slice viewer system that pre-computes all slice images upfront and delivers smooth, real-time navigation through cached image display.

Problem Statement

Current Performance Issues

  1. Real-time Computation Bottleneck: Every slice position change triggers:

    • buildTrackSliceLookup() - O(n*tracks) complexity track indexing
    • getTracksInSlice() - Track intersection calculations with tolerance
    • filterTracksByRegion() - Region-based track filtering (if enabled)
    • FA background slice extraction and rendering
    • Track plotting and color computation

  2. GUI Responsiveness: Python GUI calls MATLAB subprocess for each update:

    • Process spawn overhead (1-2 seconds)
    • MATLAB startup and path loading
    • Data reload and reprocessing
    • Image rendering and display

  3. User Experience: 5-30 second delays make interactive exploration impossible:

    • No smooth slider navigation
    • No real-time feedback
    • Cannot explore anatomical relationships efficiently

Performance Analysis

Current System Flow: 

User moves slider → Python subprocess → MATLAB load → Track processing → Render → Display
     ^                                                                                    |
     |_____________________________________________________________________________ 5-30s delay

Identified Bottlenecks:

  • Track-slice intersection: ~40% of computation time
  • FA background rendering: ~20% of computation time
  • Track color computation: ~15% of computation time
  • MATLAB process overhead: ~25% of computation time

Solution Overview

Pre-computed Image Architecture

Replace real-time computation with upfront batch processing:

  1. Batch Pre-computation Phase: Generate all slice images at standard intervals
  2. Optimized GUI Phase: Display pre-computed images with instant navigation
  3. Smart Caching System: Efficient storage and retrieval of slice images
  4. Progressive Enhancement: Optional real-time overlays for custom parameters

Key Benefits

  • Instant Navigation: Sub-100ms slice transitions
  • Smooth User Experience: Real-time slider interaction
  • Reduced Resource Usage: No repeated computations
  • Scalable Architecture: Support for multiple datasets and parameters

Detailed Requirements

Functional Requirements

FR1: Batch Image Pre-computation System

  • FR1.1: Generate slice images for all X, Y, Z positions at integer intervals
  • FR1.2: Support multiple visualization parameters (color modes, tolerances, region filters)
  • FR1.3: Configurable image resolution and quality settings
  • FR1.4: Progress tracking and cancellation support
  • FR1.5: Automatic detection of volume dimensions from nim files

FR2: Optimized Image Storage System

  • FR2.1: Hierarchical directory structure organized by parameters
  • FR2.2: Compressed image format with metadata preservation
  • FR2.3: Incremental updates when source data changes
  • FR2.4: Storage quota management and cleanup utilities
  • FR2.5: Cross-platform file organization

FR3: High-Performance GUI Navigation

  • FR3.1: Sub-100ms slice transition times
  • FR3.2: Smooth slider interaction with real-time preview
  • FR3.3: Keyboard shortcuts for rapid navigation
  • FR3.4: Jump-to-slice functionality with coordinate input
  • FR3.5: Multi-view synchronization (axial/sagittal/coronal)

FR4: Parameter Set Management

  • FR4.1: Multiple pre-computed parameter configurations
  • FR4.2: Runtime parameter switching without recomputation
  • FR4.3: Custom parameter validation and conflict resolution
  • FR4.4: Parameter preset saving and loading
  • FR4.5: Batch generation for parameter combinations

FR5: Data Integrity and Validation

  • FR5.1: Automatic detection of data file changes
  • FR5.2: Cache invalidation and regeneration triggers
  • FR5.3: Image integrity verification and corruption recovery
  • FR5.4: Metadata consistency checks
  • FR5.5: Version tracking for reproducibility

Non-Functional Requirements

NFR1: Performance Targets

  • NFR1.1: Slice navigation: <100ms response time
  • NFR1.2: Initial GUI load: <3 seconds
  • NFR1.3: Pre-computation: <2 hours for standard dataset (10K tracks, 128³ volume)
  • NFR1.4: Memory usage: <2GB during GUI operation
  • NFR1.5: Storage efficiency: <10GB for complete parameter set

NFR2: Usability Requirements

  • NFR2.1: No technical expertise required for basic operation
  • NFR2.2: Visual progress indicators for all long-running operations
  • NFR2.3: Intuitive parameter selection interface
  • NFR2.4: Error messages with actionable guidance
  • NFR2.5: Help documentation and tooltips

NFR3: Compatibility Requirements

  • NFR3.1: MATLAB R2019b or later compatibility
  • NFR3.2: Python 3.8+ with tkinter support
  • NFR3.3: Cross-platform operation (Windows, macOS, Linux)
  • NFR3.4: Backward compatibility with existing nim/tracks files
  • NFR3.5: Integration with existing HINEC pipeline

Technical Architecture

Component Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                          Fast Tractography Slice Viewer                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐ │
│  │   Batch Generator   │    │    Image Cache      │    │   GUI Viewer       │ │
│  │                     │    │                     │    │                     │ │
│  │ • Parameter Config  │────│ • Organized Storage │────│ • Instant Navigation│ │
│  │ • Progress Tracking │    │ • Metadata System   │    │ • Parameter Switching│ │
│  │ • Quality Control   │    │ • Compression       │    │ • Multi-view Sync   │ │
│  │ • MATLAB Interface  │    │ • Cache Management  │    │ • Keyboard Shortcuts│ │
│  └─────────────────────┘    └─────────────────────┘    └─────────────────────┘ │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                         Core MATLAB Processing Engine                       │
│  ┌─────────────────────────────────────────────────────────────────────────┐ │
│  │ • Optimized Track-Slice Intersection • Enhanced FA Background Rendering  │ │
│  │ • Vectorized Color Computation       • Memory-Efficient Track Loading   │ │
│  │ • Parallel Processing Support        • Progressive Quality Enhancement  │ │
│  └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘

Data Flow Architecture

Phase 1: Pre-computation

nim/tracks files → Parameter Config → Batch Generator → Optimized Images → Cache Storage
                                           │
                                    Progress Tracking
                                    Quality Validation
                                    Resource Management

Phase 2: Interactive Viewing

User Input → GUI → Cache Lookup → Image Display → Real-time Navigation
     │                                 │
     └── Parameter Switch ──────────────┘

File Organization Structure

tractography_cache/
├── datasets/
│   └── {dataset_hash}/
│       ├── metadata.json
│       ├── parameters/
│       │   └── {param_hash}/
│       │       ├── axial/
│       │       │   ├── z_001.png
│       │       │   ├── z_002.png
│       │       │   └── ...
│       │       ├── sagittal/
│       │       │   ├── x_001.png
│       │       │   └── ...
│       │       ├── coronal/
│       │       │   ├── y_001.png
│       │       │   └── ...
│       │       └── config.json
│       └── thumbnails/
│           ├── preview_axial.png
│           ├── preview_sagittal.png
│           └── preview_coronal.png
└── global_config.json

Implementation Components

Component 1: Batch Image Generator (MATLAB)

File: generateTractographySliceCache.m

Responsibilities:

  • Load and validate nim/tracks data
  • Generate parameter combinations
  • Optimize track-slice intersection algorithms
  • Render high-quality slice images
  • Save organized image cache with metadata

Key Functions: 

function generateTractographySliceCache(tracks_file, nim_file, cache_dir, options)
function optimizedTrackSliceLookup = buildOptimizedLookup(tracks, dims)
function renderSliceImage(orientation, slice_idx, nim, tracks, lookup, params)
function saveSliceWithMetadata(image, filepath, metadata)

Component 2: Cache Management System (MATLAB)

File: TractographyCacheManager.m

Responsibilities:

  • Manage cache directory structure
  • Handle parameter hashing and conflict resolution
  • Provide cache validation and integrity checks
  • Support incremental updates and cleanup

Key Functions: 

function hash = generateParameterHash(params)
function valid = validateCacheIntegrity(cache_dir)
function cleanupExpiredCache(cache_dir, max_age_days)
function metadata = getCacheMetadata(cache_dir)

Component 3: Fast GUI Viewer (Python)

File: fast_tractography_viewer.py

Responsibilities:

  • Load pre-computed image cache
  • Provide instant slice navigation
  • Handle parameter switching
  • Manage multi-view synchronization

Key Classes: 

class FastTractographyViewer:
    def __init__(self, cache_dir)
    def load_cache(self, dataset_hash, param_hash)
    def navigate_to_slice(self, orientation, slice_idx)
    def switch_parameters(self, new_param_hash)

class ImageCache:
    def __init__(self, cache_directory)
    def get_slice_image(self, orientation, slice_idx)
    def preload_adjacent_slices(self, current_slice)

Component 4: Configuration Management

File: cache_config.py

Responsibilities:

  • Parameter validation and serialization
  • Configuration presets and templates
  • Cross-platform path management

Performance Optimization Strategies

1. Pre-computation Optimizations

Track-Slice Intersection Optimization:

  • Vectorized coordinate rounding and bounds checking
  • Sparse indexing using pre-allocated cell arrays
  • Memory-mapped track data for large datasets
  • Parallel processing for independent slice computations

Rendering Pipeline Optimization:

  • Pre-computed FA slice cache for anatomical backgrounds
  • Vectorized track color computation using look-up tables
  • Optimized plot rendering with reduced object creation
  • Progressive quality enhancement for different use cases

2. Storage Optimizations

Image Compression Strategy:

  • PNG compression with optimal quality/size balance
  • Metadata embedded in image headers
  • Progressive JPEG for large volumes with quality fallback
  • Differential compression for similar adjacent slices

Cache Access Optimization:

  • Predictive adjacent slice preloading
  • LRU cache for recently accessed images
  • Asynchronous background loading
  • Memory mapping for frequently accessed slices

3. GUI Performance Optimizations

Responsive Navigation:

  • Image preloading based on navigation patterns
  • Async image loading with progress indicators
  • Smooth slider interaction with frame-rate limiting
  • Keyboard shortcuts for power users

Memory Management:

  • Intelligent cache size limits based on available RAM
  • Background garbage collection
  • Lazy loading of parameter sets
  • Image scaling for different display resolutions

User Stories and Acceptance Criteria

Epic 1: Fast Interactive Navigation

Story 1.1: As a researcher, I want to navigate between slices instantly so that I can explore tractography patterns efficiently.

Acceptance Criteria:

  • [ ] Slice navigation responds within 100ms
  • [ ] Slider movement provides real-time visual feedback
  • [ ] No visual glitches or loading delays during navigation
  • [ ] Crosshair synchronization works across all views
  • [ ] Navigation works smoothly with keyboard shortcuts

Story 1.2: As a researcher, I want to switch between different visualization parameters without waiting so that I can compare analysis approaches.

Acceptance Criteria:

  • [ ] Parameter switching completes within 2 seconds
  • [ ] Current slice position is preserved during parameter changes
  • [ ] Visual indicators show available parameter sets
  • [ ] Error handling for missing parameter combinations
  • [ ] Ability to queue multiple parameter switches

Epic 2: Efficient Pre-computation

Story 2.1: As a data analyst, I want to generate slice caches efficiently so that I can prepare datasets for interactive exploration.

Acceptance Criteria:

  • [ ] Batch generation completes within 2 hours for standard datasets
  • [ ] Progress indicator shows completion percentage and ETA
  • [ ] Ability to pause/resume generation process
  • [ ] Quality validation ensures image integrity
  • [ ] Automatic retry for failed slice generation

Story 2.2: As a system administrator, I want to manage cache storage efficiently so that disk usage remains reasonable.

Acceptance Criteria:

  • [ ] Cache size stays under 10GB for complete parameter sets
  • [ ] Automatic cleanup of old/unused caches
  • [ ] Storage quota warnings and management tools
  • [ ] Compression reports showing space savings
  • [ ] Cache integrity verification tools

Epic 3: User Experience Excellence

Story 3.1: As a new user, I want clear guidance on using the fast viewer so that I can start exploring data immediately.

Acceptance Criteria:

  • [ ] Intuitive interface requiring no technical expertise
  • [ ] Built-in help system with contextual tooltips
  • [ ] Clear error messages with actionable guidance
  • [ ] Example workflows and getting started guide
  • [ ] Visual indicators for all interactive elements

Implementation Roadmap

Phase 1: Core Architecture Foundation

Milestone 1.1: Optimized MATLAB Engine

  • [ ] Implement generateTractographySliceCache.m with basic functionality
  • [ ] Optimize track-slice intersection algorithms
  • [ ] Add progress tracking and quality validation
  • [ ] Create cache directory structure and metadata system

Milestone 1.2: Image Cache System

  • [ ] Design and implement hierarchical cache storage
  • [ ] Add metadata management and parameter hashing
  • [ ] Implement cache validation and integrity checks
  • [ ] Create compression and storage optimization

Phase 2: GUI Development

Milestone 2.1: Basic Fast Viewer

  • [ ] Implement FastTractographyViewer class with cache loading
  • [ ] Add instant slice navigation with image preloading
  • [ ] Create responsive slider controls with real-time feedback
  • [ ] Implement multi-view synchronization

Milestone 2.2: Advanced Features

  • [ ] Add parameter switching functionality
  • [ ] Implement keyboard shortcuts and power-user features
  • [ ] Create configuration management system
  • [ ] Add error handling and user feedback systems

Phase 3: Integration & Optimization

Milestone 3.1: Performance Optimization

  • [ ] Profile and optimize critical performance paths
  • [ ] Implement advanced caching strategies
  • [ ] Add memory management and resource optimization
  • [ ] Validate performance targets are met

Milestone 3.2: Quality Assurance

  • [ ] Comprehensive testing across different datasets
  • [ ] Cross-platform compatibility validation
  • [ ] User acceptance testing and feedback integration
  • [ ] Documentation and training materials

Phase 4: Production Deployment

Milestone 4.1: Production Readiness

  • [ ] Final performance validation and optimization
  • [ ] Complete documentation and user guides
  • [ ] Integration testing with existing HINEC pipeline
  • [ ] Deployment scripts and installation procedures

Milestone 4.2: Launch Support

  • [ ] User training and onboarding materials
  • [ ] Support documentation and troubleshooting guides
  • [ ] Performance monitoring and feedback collection
  • [ ] Iteration planning based on user feedback

Risk Assessment and Mitigation

Technical Risks

Risk T1: Storage Requirements Exceed Expectations

  • Probability: Medium
  • Impact: High
  • Mitigation: Implement progressive compression, quality settings, and selective parameter generation

Risk T2: MATLAB Performance Optimization Insufficient

  • Probability: Low
  • Impact: High
  • Mitigation: Fallback to C++ MEX implementations for critical algorithms, parallel processing

Risk T3: Cache Corruption or Integrity Issues

  • Probability: Medium
  • Impact: Medium
  • Mitigation: Robust validation, automatic repair, incremental backup systems

Operational Risks

Risk O1: User Adoption Challenges

  • Probability: Medium
  • Impact: Medium
  • Mitigation: Comprehensive user testing, intuitive design, extensive documentation

Risk O2: Integration Complexity with Existing Systems

  • Probability: Low
  • Impact: Medium
  • Mitigation: Maintain backward compatibility, gradual migration strategy

Success Metrics

Performance Metrics

  • Navigation Speed: <100ms slice transitions (Target: 50ms average)
  • Pre-computation Time: <2 hours for standard datasets (Target: 1 hour)
  • Storage Efficiency: <10GB total cache size (Target: 5GB)
  • Memory Usage: <2GB during operation (Target: 1GB)

User Experience Metrics

  • Task Completion Time: 80% reduction in slice exploration time
  • User Satisfaction: >90% positive feedback on navigation smoothness
  • Error Frequency: <1% failure rate for slice navigation
  • Learning Curve: <15 minutes for new user proficiency

Technical Metrics

  • Cache Hit Rate: >95% for normal navigation patterns
  • Data Integrity: 100% cache validation success rate
  • Cross-platform Compatibility: 100% functionality across Windows/macOS/Linux
  • Integration Success: 100% compatibility with existing HINEC workflows

Appendices

Appendix A: Current Performance Baseline

Based on analysis of existing tractography_slice_gui.py and visualizeTractographySlices.m:

  • Average slice update time: 8-15 seconds
  • Peak memory usage: 4-6GB during MATLAB processing
  • Subprocess overhead: 2-3 seconds per call
  • Track processing bottleneck: 40% of total time
  • User frustration points: No real-time feedback, no smooth navigation

Appendix B: Technical Implementation Details

Parameter Hashing Strategy: 

hash = SHA256(
    tracks_file_hash +
    nim_file_hash +
    tolerance +
    color_mode +
    show_anatomy +
    alpha +
    regions +
    region_filter +
    min_overlap
)

Image Naming Convention: 

{orientation}_{slice_number:03d}_{quality}.png
Examples: axial_045_high.png, sagittal_128_medium.png

Metadata Format: 

{
  "version": "1.0",
  "generated": "2024-01-15T10:30:00Z",
  "parameters": {
    "tolerance": 2,
    "color_mode": "direction",
    "show_anatomy": true,
    "alpha": 0.6
  },
  "dataset": {
    "tracks_file": "tracks_standard.mat",
    "nim_file": "sample_parcellated.mat",
    "tracks_count": 15000,
    "dimensions": [128, 128, 64]
  },
  "quality": {
    "resolution": [1024, 1024],
    "compression": "PNG",
    "size_mb": 2.5
  }
}

Appendix C: Future Enhancement Opportunities

  1. Machine Learning Integration: Predictive slice preloading based on user navigation patterns
  2. Cloud Computing: Distributed pre-computation for large datasets
  3. Advanced Compression: Custom tractography-aware compression algorithms
  4. Real-time Overlays: Hybrid approach with pre-computed bases and dynamic overlays
  5. Multi-user Collaboration: Shared cache systems for team environments
  6. Mobile/Web Interface: Browser-based viewer using pre-computed cache
  7. VR/AR Integration: Immersive 3D navigation using cached slice foundations

This PRD serves as the foundation for implementing a high-performance tractography slice viewer that will transform the user experience from frustrating delays to smooth, real-time exploration.