mad-lawsuit/cline_docs/activeContext.md
TheMaddax 7fa831bff7 Update memory bank: Production server migration to 10.4.0.205 complete
- Updated activeContext.md with migration details and new deployment server
- Updated techContext.md with upgraded packages (React 19, Next.js 16, Tailwind 4, Express 5, Prisma 7)
- Added production server information (10.4.0.205) and deployment process
- Documented Caddy IP address configuration fix
- All data migrated: 158MB PDFs, 47MB database, Redis cache
- Website fully operational at https://mad-lawsuit.org
2025-12-14 18:37:51 -07:00

572 lines
25 KiB
Markdown

# Active Context - Current Work Status
## Current Task: PRODUCTION SERVER MIGRATION COMPLETED ✅
**MAD Lawsuit Website Successfully Migrated to New Infrastructure**:
- **Old Server**: chrishaulmark.com (DigitalOcean VPS - being decommissioned)
- **New Server**: 10.4.0.205 (Dedicated server behind NAT)
- **Status**: FULLY OPERATIONAL WITH ALL DATA MIGRATED ✅
### Migration Implementation Details
**Infrastructure Changes**:
1. **Package Upgrades** ✅ COMPLETED
- **Frontend**: React 18→19, Next.js 15→16, Tailwind 3→4
- **Backend**: Prisma 5→7, Express 4→5
- **All Dependencies**: Updated to latest stable versions
- **Docker Images**: Rebuilt with new package versions
2. **Caddy Reverse Proxy Fix** ✅ COMPLETED
- **Problem**: Docker DNS resolving container hostnames to wrong network IPs
- **Impact**: Caddy timing out trying to reach containers
- **Solution**: Updated Caddyfile to use direct IP addresses on caddy_network
- **Frontend IP**: 172.18.0.6:806 (instead of hostname)
- **Backend IP**: 172.18.0.5:901 (instead of hostname)
- **File**: `/home/chaulmark/docker/caddy/config/Caddyfile` on server
3. **Data Migration** ✅ COMPLETED
- **Database**: PostgreSQL data (47MB) migrated from old server
- **PDF Files**: 158MB of court documents (69 files) migrated
- **Redis Cache**: Cache data migrated
- **Method**: Created tar archive, transferred via rsync, extracted on new server
- **Location**: `/docker/websites/mad-lawsuit/` on 10.4.0.205
4. **SSL Certificates** ✅ COMPLETED
- **Provider**: Let's Encrypt via Caddy
- **Domains**: mad-lawsuit.org, files.mad-lawsuit.org
- **Status**: Active and auto-renewing
### Final Production Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Caddy Reverse Proxy (10.4.0.205) │
│ │
│ mad-lawsuit.org → 172.18.0.6:806 (Frontend) │
│ files.mad-lawsuit.org → 172.18.0.5:901 (Backend) │
└─────────────────────────────────────────────────────────┘
├─────────────────────────────┐
│ │
┌──────▼──────┐ ┌──────▼──────┐
│ Frontend │ │ Backend │
│ Port 806 │──────────────│ Port 901 │
│ Next.js 16 │ API Calls │ Express 5 │
│ React 19 │ │ Prisma 7 │
└─────────────┘ └─────────────┘
┌──────▼──────┐
│ PostgreSQL │
│ + Redis │
└─────────────┘
```
### Deployment Server Information
**IMPORTANT**: All deployments now go to **10.4.0.205** (NOT chrishaulmark.com)
**Server Details**:
- **Hostname**: public-websites (internal: 10.4.0.205)
- **External Access**: Via NAT through dedicated server
- **SSH Access**: `ssh chaulmark@10.4.0.205`
- **Project Location**: `~/websites/mad-lawsuit.org/`
- **Docker Volumes**: `/docker/websites/mad-lawsuit/`
**Deployment Commands**:
```bash
# SSH to production server
ssh chaulmark@10.4.0.205
# Navigate to project
cd ~/websites/mad-lawsuit.org
# Pull latest code
git pull
# Rebuild and restart containers
docker compose down
docker compose build
docker compose up -d
# Check container status
docker compose ps
docker compose logs -f
```
### Verification ✅
- **Website**: https://mad-lawsuit.org (HTTP/2 200 ✓)
- **Backend API**: https://files.mad-lawsuit.org (HTTP/2 200 ✓)
- **Database**: 63 docket entries with all historical data ✓
- **PDF Files**: 69 court documents accessible ✓
- **All Containers**: Running and healthy ✓
### Git Commits Made
- Migration completed with all package upgrades
- Caddyfile updated with direct IP addresses
- All changes committed to repository
## Previous Task: CACHE CONTROL HEADERS ADDED ✅
**MAD Lawsuit Website Cache Prevention Implemented**:
- **Problem**: HTTP response caching causing stale 404 responses for PDF downloads
- **Root Cause**: Caddy/CDN caching API responses, persisting even after fixes deployed
- **Solution**: Added Cache-Control headers to prevent future caching
- **Status**: CACHE HEADERS ACTIVE, LEGACY CACHE EXPIRING ⏳
### Cache Control Implementation Details
**Problem Identified**:
- After implementing dedicated backend subdomain, some PDF downloads still returned 404
- Testing revealed cached responses from before the fix was deployed
- Cache persisted through multiple Caddy restarts and data directory clearing
- Indicates upstream CDN or aggressive HTTP caching layer
**Solution Implemented** ✅:
1. **Added Cache-Control Headers to Caddyfile**
- **Location**: `mad-lawsuit.org` block in Caddyfile
- **Headers**: `Cache-Control: no-store, no-cache, must-revalidate`
- **Scope**: All `/api/*` routes
- **Purpose**: Prevent new caches from forming
2. **Caddyfile Update**:
```caddy
mad-lawsuit.org {
# Disable caching for API routes to prevent stale responses
header /api/* {
Cache-Control "no-store, no-cache, must-revalidate"
}
reverse_proxy 172.18.0.1:806
}
```
3. **Deployment Process**:
- Updated local Caddyfile
- SCP'd to server: `~/docker/caddy/config/Caddyfile`
- Restarted Caddy: `docker compose -f caddy-compose.yml down && up -d`
- Verified headers: `cache-control: no-store, no-cache, must-revalidate`
**Current Status**:
- ✅ Cache-Control headers active and being sent
- ✅ API endpoint working: `https://mad-lawsuit.org/api/docket-entries` returns 63 entries
- ✅ Direct backend working: All PDFs accessible via `https://files.mad-lawsuit.org`
- ⏳ Legacy cached 404s: Documents 4, 11, 13 still have cached responses (will expire naturally)
- ✅ New requests: Will NOT be cached due to Cache-Control headers
**Workaround for Users**:
- Use direct backend URL for immediate access: `https://files.mad-lawsuit.org/api/documents/{id}/download`
- Frontend proxy URLs will work correctly once legacy cache expires (typically 24 hours)
## Previous Task: DEDICATED BACKEND SUBDOMAIN IMPLEMENTED ✅
**MAD Lawsuit Website Backend Moved to files.mad-lawsuit.org**:
- **Problem**: PDF downloads unreliable through frontend proxy tunnel
- **Root Cause**: Frontend proxy (Next.js) creating instability for file downloads
- **Solution**: Created dedicated backend subdomain with direct Caddy routing
- **Status**: BACKEND SUBDOMAIN FULLY OPERATIONAL ✅
### Dedicated Backend Subdomain Implementation Details
**Architecture Changes**:
1. **Backend Port Change** ✅ COMPLETED
- **Changed**: Backend port from 809 → 901 (9xx range for backends)
- **Reason**: Follow port structure convention (8xx for frontends, 9xx for backends)
- **File**: `docker-compose.yml`
2. **New Subdomain Configuration** ✅ COMPLETED
- **Created**: `files.mad-lawsuit.org` subdomain for backend API
- **DNS**: Already resolving to same IP as mad-lawsuit.org
- **Caddy**: Direct reverse proxy to backend on port 901
- **File**: `Caddyfile`
3. **Simplified Frontend Routing** ✅ COMPLETED
- **Changed**: `mad-lawsuit.org` now only proxies to frontend (port 806)
- **Removed**: Complex path-based routing (/api/* handling)
- **Result**: Cleaner, more reliable frontend routing
4. **Frontend API Proxy Update** ✅ COMPLETED
- **Changed**: Frontend now calls `https://files.mad-lawsuit.org/api/*`
- **Previous**: Called internal Docker container `http://mad-lawsuit-backend-1:3001`
- **Benefit**: Direct backend access, no proxy tunnel issues
- **File**: `frontend/src/app/api/[...path]/route.ts`
5. **Backend CORS Update** ✅ COMPLETED
- **Added**: Both `mad-lawsuit.org` and `files.mad-lawsuit.org` to allowed origins
- **Reason**: Backend needs to accept requests from both domains
- **File**: `backend/src/index.ts`
### Final Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Caddy Reverse Proxy │
│ │
│ mad-lawsuit.org → 172.18.0.1:806 (Frontend) │
│ files.mad-lawsuit.org → 172.18.0.1:901 (Backend) │
└─────────────────────────────────────────────────────────┘
├─────────────────────────────┐
│ │
┌──────▼──────┐ ┌──────▼──────┐
│ Frontend │ │ Backend │
│ Port 806 │──────────────│ Port 901 │
│ Next.js │ API Calls │ Express.js │
└─────────────┘ └─────────────┘
```
### Benefits of New Architecture
1. **Reliability**: Direct backend access eliminates proxy tunnel issues
2. **Stability**: Backend isolated from frontend routing problems
3. **Performance**: Reduced latency without frontend proxy overhead
4. **Maintainability**: Cleaner separation of concerns
5. **Scalability**: Can scale frontend/backend independently
6. **Debugging**: Easier to troubleshoot backend-specific issues
### Deployment Status ✅
- **Code Changes**: Committed to Git (commit a2595246)
- **Caddyfile**: Uploaded and reloaded on server
- **Containers**: Rebuilt and restarted with new configuration
- **Verification**: Both domains responding correctly
- `https://mad-lawsuit.org` → Frontend (HTTP 200)
- `https://files.mad-lawsuit.org/api/health` → Backend (HTTP 200)
### Git Commit
```
commit a2595246
Implement dedicated backend subdomain files.mad-lawsuit.org
- Changed backend port from 809 to 901 (9xx range for backends)
- Updated docker-compose.yml to expose backend on port 901
- Created new Caddyfile with files.mad-lawsuit.org subdomain
- Simplified mad-lawsuit.org to only proxy to frontend (port 806)
- Updated frontend API proxy to use https://files.mad-lawsuit.org
- Updated backend CORS to allow both mad-lawsuit.org and files.mad-lawsuit.org
- This fixes PDF download reliability issues by providing direct backend access
```
## Previous Task: PDF DOWNLOAD ISSUE FULLY RESOLVED ✅
**MAD Lawsuit Website PDF Download Fixed via Direct Backend Routing**:
- **Problem**: PDF downloads failing with "Endpoint not found" error
- **Root Cause**: Caddy was routing ALL requests (including /api/*) to frontend, frontend's Next.js proxy couldn't properly handle the requests
- **Solution**: Exposed backend on port 809 and configured Caddy to route /api/* directly to backend
- **Status**: PDF DOWNLOADS WORKING ✅ (Now superseded by dedicated subdomain)
### PDF Download Fix Implementation Details
**Issues Resolved**:
1. **Caddy Routing Issue** ✅ FIXED
- **Problem**: Caddy configuration only proxied to frontend (port 806), no separate API routing
- **Impact**: All /api/* requests went through frontend's Next.js proxy, causing routing issues
- **Solution**: Exposed backend on port 809 and configured Caddy to route /api/* directly to backend
- **Files**: `docker-compose.yml`, Caddy configuration
2. **Backend Port Exposure** ✅ FIXED
- **Problem**: Backend not exposed to host, only accessible within Docker network
- **Solution**: Added `ports: - "809:3001"` to backend service in docker-compose.yml
- **File**: `docker-compose.yml`
3. **Caddy Configuration Update** ✅ FIXED
- **Problem**: Simple reverse proxy to frontend didn't handle API routes separately
- **Solution**: Added handle blocks to route /api/* to backend (809) and everything else to frontend (806)
- **File**: `/home/chaulmark/docker/caddy/config/Caddyfile`
### Final Caddy Configuration
```
mad-lawsuit.org {
# Route API requests directly to backend
handle /api/* {
reverse_proxy 172.18.0.1:809
}
# Route everything else to frontend
handle {
reverse_proxy 172.18.0.1:806
}
}
```
### Verification ✅
```bash
# Backend accessible on port 809
curl http://localhost:809/api/documents/9/download
# HTTP 200 - PDF content returned
# Public domain working
curl https://mad-lawsuit.org/api/documents/9/download
# HTTP 200 - PDF content returned
```
### Git Commits Made
1. `13d219db` - Expose backend on port 809 for direct Caddy routing to fix PDF download
## Previous Task: DOCKER BUILD ISSUES FULLY RESOLVED ✅
**MAD Lawsuit Website Docker Deployment Successfully Fixed**:
- **Problem**: Docker containers failing to build and run due to pnpm and Prisma issues
- **Root Cause**: Prisma CLI in devDependencies but needed in production for client generation
- **Solution**: Dependency restructuring + Docker build process optimization
- **Status**: ALL CONTAINERS RUNNING AND HEALTHY ✅
### Docker Fix Implementation Details
**Issues Resolved**:
1. **pnpm TTY Error** ✅ FIXED
- **Problem**: `pnpm prune --prod` failing with TTY error in Docker build
- **Solution**: Added `ENV CI=true` to Dockerfile before prune command
- **File**: `backend/Dockerfile`
2. **Prisma Client Missing After Prune** ✅ FIXED
- **Problem**: `prisma` CLI in devDependencies, removed during production prune
- **Impact**: Prisma client generation failing, backend crashing with MODULE_NOT_FOUND
- **Solution**: Moved `prisma` from devDependencies to dependencies in package.json
- **Files**: `backend/package.json`, `backend/pnpm-lock.yaml`
3. **Outdated Lockfile** ✅ FIXED
- **Problem**: pnpm-lock.yaml outdated after dependency changes
- **Impact**: `pnpm install --frozen-lockfile` failing in Docker build
- **Solution**: Regenerated lockfile locally and committed to repository
### Final Docker Build Process
```dockerfile
# Install all dependencies
RUN pnpm install --frozen-lockfile
# Generate Prisma client (first time)
RUN pnpm prisma generate
# Build application
RUN pnpm build
# Remove dev dependencies (but keep prisma CLI)
ENV CI=true
RUN pnpm prune --prod
# Regenerate Prisma client (after prune, using production deps)
RUN pnpm prisma generate
```
### Current Container Status - ALL HEALTHY ✅
- **mad-lawsuit-frontend-1**: Running and healthy on port 806
- **mad-lawsuit-backend-1**: Running and healthy with working database connections
- **mad-lawsuit-postgres-1**: Running on port 5432
- **mad-lawsuit-redis-1**: Running on port 6379
### Backend Verification ✅
```
> node dist/index.js
prisma:info Starting a postgresql pool with 3 connections.
prisma:query SELECT 1
prisma:query SELECT "public"."docket_entries"...
```
- Prisma client working correctly
- Database connections established
- Queries executing successfully
### Git Commits Made
1. `b291ee9b` - Fix Docker build: Add CI=true env var for pnpm prune command
2. `d087ca9c` - Fix Prisma client: Regenerate after pnpm prune to restore missing binaries
3. `0035b7cc` - Fix Prisma generate: Use npx instead of pnpm after prune removes CLI
4. `cf071cc4` - Fix Prisma prune: Use --config.ignore-scripts=false to preserve Prisma client
5. `b59ae90c` - Fix Prisma dependencies: Move prisma CLI to production deps and regenerate after prune
6. `f962327a` - Update pnpm-lock.yaml after moving prisma to production dependencies
## Previous Task: GMAIL SMUGGLER FULLY OPERATIONAL ✅
**SMTP Relay Solution Successfully Implemented**:
- **DigitalOcean Server (chrishaulmark.com)**: SMTP ports 25, 465, 587 BLOCKED by DigitalOcean
- **Dedicated Server (74.80.182.50)**: All SMTP ports WORKING ✅
- **Gmail Smuggler Container**: DEPLOYED and RUNNING ✅ on port 2525 (unblocked)
**Solution**: DNS override + port 2525 SMTP relay bypasses DigitalOcean SMTP blocking completely.
### Gmail Smuggler Final Configuration
- **Container Name**: `gmail-smuggler`
- **Location**: `10.4.0.206:~/gmail-smuggler/` (internal VM)
- **Status**: Running with Docker Compose auto-restart
- **Port**: 2525 (external) → 587 (internal, forwarding to smtp.gmail.com:587)
- **Technology**: Alpine Linux + socat TCP relay
- **OPNsense**: Port forwarding 2525 → 10.4.0.206:2525
### Implementation Details
1. **DNS Override**: `74.80.182.50 smtp.gmail.com` in `/etc/hosts`
2. **Backend Configuration**: `SMTP_PORT=2525` in `backend/.env`
3. **Port Forwarding**: OPNsense forwards port 2525 to internal VM ✅
4. **Connection Verified**: DigitalOcean VPS → 74.80.182.50:2525 → Gmail ✅
### Email Flow
```
Court Docket Website → smtp.gmail.com:2525 → 74.80.182.50:2525 → 10.4.0.206:2525 → Gmail SMTP
```
## Previous Task: COMPLETED ✅
**DeafGain LLC Footer Addition** - Added "Designed by DeafGain LLC" footer to the court docket website.
## Previous Task: COMPLETED ✅
**Email System URL Fix** - Updated email notifications to use production domain and added unsubscribe instructions.
## Previous Task: COMPLETED ✅
**PDF Upload Issue Resolution** - Successfully identified and fixed all issues preventing PDF document uploads in the admin dashboard.
## Recent Work Completed
### DeafGain LLC Footer Addition - COMPLETED ✅
**Problem**: Website was missing professional branding footer like other DeafGain websites.
**Solution Implemented**:
1. **Footer Design** ✅ ADDED
- **Implementation**: Added footer section at bottom of home page
- **Styling**: Matches website theme with dark background and yellow accent text
- **Content**: "Designed by DeafGain LLC" with link to http://deafgain.org
- **File**: `frontend/src/app/page.tsx`
2. **Visual Consistency** ✅ ACHIEVED
- **Reference**: Based on Chris Haulmark website footer implementation
- **Colors**: Gray text with yellow (#fbbf24) DeafGain LLC link
- **Hover Effect**: Color changes to darker yellow (#f59e0b) on hover
- **Layout**: Centered text in footer section
### Footer Implementation:
```jsx
<footer style={{
width: '100%',
padding: '2rem 0',
backgroundColor: 'rgba(57, 64, 83, 0.95)',
borderTop: '1px solid #4E4A59',
textAlign: 'center',
display: 'block'
}}>
<div style={{ textAlign: 'center', display: 'block', width: '100%' }}>
<span style={{ color: '#9ca3af' }}>Designed by </span>
<a href="http://deafgain.org" target="_blank" rel="noopener noreferrer"
style={{ color: '#fbbf24', textDecoration: 'none' }}>
DeafGain LLC
</a>
</div>
</footer>
```
### Email System URL Fix - COMPLETED ✅
**Problem**: Email notifications were linking to localhost instead of production domain, and missing unsubscribe instructions.
**Issues Fixed**:
1. **Production URL Issue** ✅ FIXED
- **Problem**: Email links pointed to `localhost:3000` instead of production domain
- **Impact**: Subscribers couldn't access website from email notifications
- **Solution**: Updated all email links to use `https://mad-lawsuit.org/`
- **File**: `backend/src/services/emailService.ts`
2. **Email Text Improvement** ✅ FIXED
- **Problem**: Generic "court docket website" text in emails
- **Impact**: Less professional email presentation
- **Solution**: Changed to "You can view the complete docket and any associated documents by visiting the website:" with website as clickable link
- **File**: `backend/src/services/emailService.ts`
3. **Missing Unsubscribe Instructions** ✅ FIXED
- **Problem**: No clear unsubscribe instructions for email recipients
- **Impact**: Users couldn't easily unsubscribe from notifications
- **Solution**: Added "To unsubscribe, please send an email to eliza.kragh@gmail.com" in footer
- **File**: `backend/src/services/emailService.ts`
### Email Changes Made:
```html
<!-- Before -->
<p>You can view the complete docket and any associated documents by visiting the court docket website:</p>
<a href="${process.env['FRONTEND_URL'] || 'http://localhost:3000'}" class="button">View Court Docket</a>
<!-- After -->
<p>You can view the complete docket and any associated documents by visiting the <a href="https://mad-lawsuit.org/">website</a>:</p>
<a href="https://mad-lawsuit.org/" class="button">View Court Docket</a>
<p>To unsubscribe, please send an email to eliza.kragh@gmail.com</p>
```
### PDF Upload Issue - FULLY RESOLVED ✅
**Problem**: Users could not upload PDF documents through the admin dashboard, receiving various errors.
**Root Causes Identified and Fixed**:
1. **Network Connectivity Issue** ✅ FIXED
- **Problem**: Frontend container only on `caddy_network`, backend on `app-network`
- **Impact**: Frontend couldn't reach backend for API calls
- **Solution**: Added frontend to both networks in `docker-compose.yml`
- **Commit**: 910bd525
2. **MIME Type Validation Issue** ✅ FIXED
- **Problem**: Backend fileFilter only accepted `application/pdf` MIME type
- **Impact**: Valid PDFs rejected due to browser MIME detection variations
- **Solution**: Enhanced fileFilter to accept multiple PDF MIME types + file extension fallback
- **File**: `backend/src/routes/documents.ts`
- **Commit**: 9ab09196
3. **Missing Form Data Issue** ✅ FIXED
- **Problem**: Frontend only sending `file` and `docketEntryId`, missing required `title` field
- **Impact**: Backend validation failing with "title is required" error
- **Solution**: Added all required fields to FormData in dashboard upload
- **File**: `frontend/src/app/admin/dashboard/page.tsx`
- **Commit**: 8027ef0a
### Technical Details
**Network Architecture Fixed**:
```yaml
frontend:
networks:
- app-network # Added for backend communication
- caddy_network # Existing for reverse proxy
```
**Enhanced PDF Detection**:
```javascript
const allowedMimeTypes = [
'application/pdf',
'application/x-pdf',
'application/acrobat',
'applications/vnd.pdf',
'text/pdf',
'text/x-pdf'
];
```
**Complete Form Data**:
```javascript
uploadFormData.append('file', selectedFile);
uploadFormData.append('docketEntryId', data.entry.id.toString());
uploadFormData.append('title', selectedFile.name.replace('.pdf', ''));
uploadFormData.append('summary', '');
uploadFormData.append('notes', '');
```
## Current Status
### What's Working ✅
- **Docker containers all running and healthy**
- **Backend with working Prisma database connections**
- **Frontend accessible on port 806**
- Network connectivity between frontend and backend
- PDF MIME type detection (multiple formats)
- Form validation with all required fields
- Error logging and debugging
- **Email system with production URLs** (https://mad-lawsuit.org/)
- **Professional unsubscribe instructions** (eliza.kragh@gmail.com)
- **DeafGain LLC footer branding** with professional styling
- All code committed and pushed to Git
### Next Steps for User
1. **Website is fully operational** - no further action needed
2. **All containers healthy and running** on remote server
3. **Database connections working** - Prisma queries executing successfully
4. **Email notifications functional** with proper URLs
5. **Professional branding** with DeafGain LLC footer
### Files Modified (Docker Fix)
- `backend/Dockerfile` - Added CI=true, optimized Prisma generation
- `backend/package.json` - Moved prisma from devDependencies to dependencies
- `backend/pnpm-lock.yaml` - Updated lockfile for new dependency structure
### Files Modified (Previous Tasks)
- `docker-compose.yml` - Network configuration
- `backend/src/routes/documents.ts` - Enhanced PDF validation
- `frontend/src/app/admin/dashboard/page.tsx` - Fixed form data
- **`backend/src/services/emailService.ts` - Updated URLs and unsubscribe info**
- **`frontend/src/app/page.tsx` - Added DeafGain LLC footer**
### Deployment Notes
- **Docker deployment successful** - all containers running on remote server
- **No further deployment needed** - website fully operational
- All fixes are committed to Git repository
- Backend and frontend both successfully deployed with latest fixes
## Investigation Process
1. **Docker Build Analysis**: Identified pnpm TTY and Prisma dependency issues
2. **Dependency Management**: Restructured package.json for proper production builds
3. **Build Process Optimization**: Enhanced Dockerfile for reliable Prisma client generation
4. **Systematic Testing**: Verified all containers healthy and database connections working
The MAD lawsuit website Docker deployment is now fully operational with all containers running successfully.