hackathon/docs/quality/THREAD_SAFETY_GUIDELINES.md

# Thread Safety Guidelines
**Version**: 1.0
**Project**: HTTP Sender Plugin (HSP)
**Last Updated**: 2025-11-20

## Purpose

This document defines thread safety requirements, patterns, and best practices for the HSP project. The HSP system uses **Java 25 Virtual Threads** for concurrent HTTP polling and must ensure thread-safe operations for shared state.

---

## 🎯 Thread Safety Requirements

### 1. Critical Thread-Safe Components

The following components **MUST be thread-safe** per requirements:

| Component | Requirement | Concurrency Pattern | Justification |
|-----------|-------------|---------------------|---------------|
| **BufferManager** | Req-FR-26, Arch-7 | Thread-safe queue | Multiple producers (HTTP pollers), single consumer (gRPC transmitter) |
| **CollectionStatistics** | Req-NFR-8, Arch-8 | Atomic counters | Multiple threads updating statistics concurrently |
| **DataCollectionService** | Req-FR-14, Arch-6 | Virtual threads | 1000 concurrent HTTP polling tasks |
| **RateLimiter** | Enhancement | Thread-safe rate limiting | Multiple threads requesting rate limit permits |
| **BackpressureController** | Req-FR-27 | Atomic monitoring | Buffer usage checked by multiple threads |

### 2. Immutable Components (Thread-Safe by Design)

The following components **MUST be immutable**:

| Component | Type | Thread Safety |
|-----------|------|---------------|
| **DiagnosticData** | Value Object | Immutable (final fields, no setters) |
| **Configuration** | Value Object | Immutable (loaded once at startup) |
| **HealthCheckResponse** | Value Object | Immutable (snapshot of current state) |
| **BufferStatistics** | Value Object | Immutable (snapshot of buffer state) |

### 3. Single-Threaded Components (No Thread Safety Needed)

The following components run on **dedicated threads** (no concurrent access):

| Component | Thread Model | Justification |
|-----------|-------------|---------------|
| **DataTransmissionService** | Single consumer thread | Req-FR-25: One thread consumes from buffer |
| **ConfigurationManager** | Startup only | Loaded once before concurrent operations start |
| **Adapters** | Per-request isolation | Each request creates new adapter instance or uses thread-local state |

---

## 🧵 Concurrency Model Overview

### System Threading Architecture

```
┌─────────────────────────────────────────────────────────┐
│ HTTP Sender Plugin (HSP) Threading Model                │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  Main Thread                                             │
│  └─> Startup & Configuration                            │
│                                                          │
│  Virtual Thread Pool (HTTP Polling)                     │
│  ├─> Virtual Thread 1 → HttpPollingAdapter              │
│  ├─> Virtual Thread 2 → HttpPollingAdapter              │
│  ├─> Virtual Thread 3 → HttpPollingAdapter              │
│  └─> ... (up to 1000 concurrent virtual threads)        │
│       ↓                                                  │
│  [Thread-Safe BufferManager] (ArrayBlockingQueue)       │
│       ↓                                                  │
│  Single Consumer Thread (gRPC Transmission)             │
│  └─> DataTransmissionService → GrpcStreamAdapter        │
│                                                          │
│  Health Check HTTP Server Thread                        │
│  └─> HealthCheckController (embedded Jetty)             │
│                                                          │
└─────────────────────────────────────────────────────────┘
```

### Virtual Threads (Java 25)

**Why Virtual Threads?**
- **Requirement**: Req-NFR-1: Support 1000 concurrent endpoints
- **Benefit**: Lightweight threads (millions possible vs. thousands of platform threads)
- **Use Case**: I/O-bound HTTP polling (mostly waiting for network responses)

**Virtual Thread Best Practices**:
- ✅ **DO**: Use for I/O-bound tasks (HTTP requests, file I/O)
- ✅ **DO**: Create one virtual thread per endpoint poll
- ✅ **DO**: Let virtual threads block (don't use async APIs unnecessarily)
- ❌ **DON'T**: Use for CPU-bound tasks (use platform threads instead)
- ❌ **DON'T**: Use with `synchronized` on long-running operations (use `ReentrantLock`)

**Creating Virtual Threads**:
```java
// CORRECT: Virtual thread executor for HTTP polling
ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();

// Schedule polling tasks
for (String url : endpoints) {
    executor.submit(() -> pollEndpoint(url));
}
```

---

## 🔒 Thread Safety Patterns

### Pattern 1: Immutability (Preferred)

**When to Use**: Value objects, configuration, data transfer objects

**Benefits**:
- Thread-safe by design (no synchronization needed)
- No defensive copying required
- Easier to reason about

**Implementation**:
```java
/**
 * Immutable value object representing diagnostic data.
 * Thread-safe by design (immutable).
 *
 * @requirement Req-FR-22 Immutable data representation
 */
public final class DiagnosticData {
    private final String endpointUrl;
    private final byte[] data;
    private final Instant timestamp;

    public DiagnosticData(String endpointUrl, byte[] data) {
        this.endpointUrl = Objects.requireNonNull(endpointUrl);
        // Defensive copy of mutable array
        this.data = Arrays.copyOf(data, data.length);
        this.timestamp = Instant.now();
    }

    // Only getters, no setters
    public String getEndpointUrl() {
        return endpointUrl;
    }

    public byte[] getData() {
        // Return defensive copy
        return Arrays.copyOf(data, data.length);
    }

    public Instant getTimestamp() {
        return timestamp; // Instant is immutable
    }

    // equals, hashCode, toString...
}
```

**Checklist**:
- [ ] Class declared `final` (cannot be subclassed)
- [ ] All fields declared `final` (assigned once in constructor)
- [ ] No setter methods (only getters)
- [ ] Defensive copies for mutable fields (arrays, collections)
- [ ] Getters return defensive copies of mutable fields

### Pattern 2: Concurrent Collections

**When to Use**: Shared data structures accessed by multiple threads

**Preferred Collections**:
- `ArrayBlockingQueue<T>`: Fixed-size blocking queue (buffer)
- `ConcurrentHashMap<K,V>`: Thread-safe map (if needed)
- `CopyOnWriteArrayList<T>`: Thread-safe list for read-heavy workloads

**Implementation (BufferManager)**:
```java
/**
 * Thread-safe buffer manager using ArrayBlockingQueue.
 *
 * <p>Supports multiple producers (HTTP polling threads) and single
 * consumer (gRPC transmission thread).
 *
 * @requirement Req-FR-26 Thread-safe buffer
 * @requirement Req-Arch-7 Concurrent collection usage
 */
public class BufferManager {
    private final BlockingQueue<DiagnosticData> buffer;
    private final int capacity;

    // Statistics (atomic counters)
    private final AtomicLong totalOffered = new AtomicLong(0);
    private final AtomicLong totalDiscarded = new AtomicLong(0);

    public BufferManager(int capacity) {
        this.capacity = capacity;
        // ArrayBlockingQueue is thread-safe
        this.buffer = new ArrayBlockingQueue<>(capacity);
    }

    /**
     * Offers data to buffer. Thread-safe.
     * Discards oldest if buffer is full (FIFO).
     *
     * @requirement Req-FR-27 FIFO overflow handling
     */
    public void offer(DiagnosticData data) {
        Objects.requireNonNull(data, "data cannot be null");
        totalOffered.incrementAndGet();

        if (!buffer.offer(data)) {
            // Buffer full, discard oldest (FIFO)
            buffer.poll(); // Remove oldest
            buffer.offer(data); // Add new
            totalDiscarded.incrementAndGet();
        }
    }

    /**
     * Polls data from buffer. Thread-safe.
     * Blocks if buffer is empty (up to timeout).
     */
    public DiagnosticData poll(long timeout, TimeUnit unit)
            throws InterruptedException {
        return buffer.poll(timeout, unit);
    }

    /**
     * Returns current buffer size. Thread-safe.
     */
    public int size() {
        return buffer.size();
    }

    /**
     * Returns buffer statistics snapshot. Thread-safe.
     */
    public BufferStatistics getStatistics() {
        return new BufferStatistics(
            size(),
            capacity,
            totalOffered.get(),
            totalDiscarded.get()
        );
    }
}
```

**Checklist**:
- [ ] Use `BlockingQueue` for producer-consumer patterns
- [ ] Use `ArrayBlockingQueue` for bounded buffers
- [ ] Use `ConcurrentHashMap` for thread-safe maps
- [ ] Avoid `synchronized` on collection itself (use concurrent collection)

### Pattern 3: Atomic Variables

**When to Use**: Counters, flags, simple shared state

**Atomic Classes**:
- `AtomicInteger`: Thread-safe integer counter
- `AtomicLong`: Thread-safe long counter
- `AtomicBoolean`: Thread-safe boolean flag
- `AtomicReference<T>`: Thread-safe object reference

**Implementation (CollectionStatistics)**:
```java
/**
 * Thread-safe collection statistics using atomic variables.
 *
 * @requirement Req-NFR-8 Statistics tracking
 * @requirement Req-Arch-8 Atomic operations
 */
public class CollectionStatistics {
    // Atomic counters for thread safety
    private final AtomicLong totalPolls = new AtomicLong(0);
    private final AtomicLong successfulPolls = new AtomicLong(0);
    private final AtomicLong failedPolls = new AtomicLong(0);

    // Time-windowed metrics (last 30 seconds)
    private final Queue<Long> recentPolls = new ConcurrentLinkedQueue<>();

    /**
     * Increments total poll count. Thread-safe.
     */
    public void incrementTotalPolls() {
        long count = totalPolls.incrementAndGet();
        recentPolls.offer(System.currentTimeMillis());
        cleanupOldMetrics();
    }

    /**
     * Increments successful poll count. Thread-safe.
     */
    public void incrementSuccessfulPolls() {
        successfulPolls.incrementAndGet();
    }

    /**
     * Increments failed poll count. Thread-safe.
     */
    public void incrementFailedPolls() {
        failedPolls.incrementAndGet();
    }

    /**
     * Returns snapshot of current statistics. Thread-safe.
     */
    public StatisticsSnapshot getSnapshot() {
        return new StatisticsSnapshot(
            totalPolls.get(),
            successfulPolls.get(),
            failedPolls.get(),
            calculateRecentRate()
        );
    }

    /**
     * Removes metrics older than 30 seconds.
     */
    private void cleanupOldMetrics() {
        long cutoff = System.currentTimeMillis() - 30_000;
        recentPolls.removeIf(timestamp -> timestamp < cutoff);
    }

    private double calculateRecentRate() {
        return recentPolls.size() / 30.0; // polls per second
    }
}
```

**Checklist**:
- [ ] Use `AtomicLong` for counters (not `long` with `synchronized`)
- [ ] Use `incrementAndGet()` for atomic increment-and-read
- [ ] Use `get()` for atomic read
- [ ] Use `compareAndSet()` for atomic compare-and-swap (if needed)

### Pattern 4: Locks (When Needed)

**When to Use**: Complex synchronized operations, multiple state updates

**Lock Types**:
- `ReentrantLock`: Exclusive lock (mutual exclusion)
- `ReentrantReadWriteLock`: Read-write lock (multiple readers, one writer)
- `StampedLock`: Optimistic locking (Java 8+)

**Implementation (if needed)**:
```java
public class RateLimitedAdapter {
    private final ReentrantLock lock = new ReentrantLock();
    private long lastRequestTime = 0;
    private final long minIntervalMs;

    /**
     * Thread-safe rate limiting with explicit lock.
     *
     * Prefer ReentrantLock over synchronized for virtual threads.
     */
    public void acquirePermit() throws InterruptedException {
        lock.lock();
        try {
            long now = System.currentTimeMillis();
            long elapsed = now - lastRequestTime;
            if (elapsed < minIntervalMs) {
                Thread.sleep(minIntervalMs - elapsed);
            }
            lastRequestTime = System.currentTimeMillis();
        } finally {
            lock.unlock(); // ALWAYS unlock in finally
        }
    }
}
```

**Checklist**:
- [ ] Use `ReentrantLock` instead of `synchronized` for virtual threads
- [ ] Always unlock in `finally` block
- [ ] Avoid holding locks during I/O operations (risk of pinning virtual threads)
- [ ] Document lock ordering if multiple locks used (prevent deadlock)

### Pattern 5: Thread Confinement

**When to Use**: State that doesn't need to be shared

**Strategies**:
- **Stack Confinement**: Use local variables (method parameters, local vars)
- **Thread-Local**: Use `ThreadLocal<T>` for thread-specific state
- **Instance-Per-Thread**: Create new instances per thread

**Implementation**:
```java
/**
 * HttpPollingAdapter is thread-confined by design.
 * Each virtual thread creates its own adapter instance.
 * No shared mutable state → no thread safety needed.
 */
public class HttpPollingAdapter implements IHttpPollingPort {
    // Immutable configuration (thread-safe)
    private final Configuration config;

    // Thread-confined HttpClient (one per instance)
    private final HttpClient httpClient;

    public HttpPollingAdapter(Configuration config) {
        this.config = config; // Immutable
        this.httpClient = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(30))
            .build();
    }

    @Override
    public CompletableFuture<byte[]> pollEndpoint(String url) {
        // This method is called by a single virtual thread
        // No shared mutable state → thread-safe by design
        return httpClient.sendAsync(buildRequest(url), BodyHandlers.ofByteArray())
            .thenApply(HttpResponse::body);
    }
}
```

**Checklist**:
- [ ] Prefer immutability and thread confinement over synchronization
- [ ] Document thread ownership in Javadoc
- [ ] Avoid sharing mutable state when possible

---

## 🧪 Testing Thread Safety

### Test Strategy

**1. Unit Tests with Concurrent Access**:
```java
@Test
void shouldBeThreadSafe_whenMultipleThreadsOfferConcurrently() {
    // Given
    BufferManager buffer = new BufferManager(100);
    int numThreads = 50;
    int offersPerThread = 100;

    // When: Multiple threads offer concurrently
    ExecutorService executor = Executors.newFixedThreadPool(numThreads);
    List<Future<?>> futures = new ArrayList<>();

    for (int i = 0; i < numThreads; i++) {
        futures.add(executor.submit(() -> {
            for (int j = 0; j < offersPerThread; j++) {
                buffer.offer(new DiagnosticData("url", new byte[]{1,2,3}));
            }
        }));
    }

    // Wait for completion
    for (Future<?> future : futures) {
        future.get();
    }
    executor.shutdown();

    // Then: All offers processed (no data loss except overflow)
    BufferStatistics stats = buffer.getStatistics();
    assertThat(stats.totalOffered()).isEqualTo(numThreads * offersPerThread);
}
```

**2. Stress Tests**:
```java
@Test
void shouldHandleHighConcurrency_with1000ProducersAndConsumers() {
    BufferManager buffer = new BufferManager(300);

    // 1000 producers
    ExecutorService producers = Executors.newVirtualThreadPerTaskExecutor();
    for (int i = 0; i < 1000; i++) {
        producers.submit(() -> {
            for (int j = 0; j < 1000; j++) {
                buffer.offer(new DiagnosticData("url", new byte[]{1,2,3}));
            }
        });
    }

    // 1 consumer
    ExecutorService consumer = Executors.newSingleThreadExecutor();
    AtomicLong consumed = new AtomicLong(0);
    consumer.submit(() -> {
        while (consumed.get() < 1_000_000) {
            try {
                DiagnosticData data = buffer.poll(1, TimeUnit.SECONDS);
                if (data != null) {
                    consumed.incrementAndGet();
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                break;
            }
        }
    });

    // Wait for completion and verify
    producers.shutdown();
    producers.awaitTermination(10, TimeUnit.MINUTES);
    consumer.shutdown();
    consumer.awaitTermination(1, TimeUnit.MINUTES);

    // Verify: No deadlock, no data corruption
    assertThat(consumed.get()).isGreaterThan(0);
}
```

**3. Race Condition Detection**:
```java
@Test
void shouldNotHaveRaceCondition_inAtomicIncrement() {
    CollectionStatistics stats = new CollectionStatistics();
    int numThreads = 100;
    int incrementsPerThread = 10000;

    ExecutorService executor = Executors.newFixedThreadPool(numThreads);
    for (int i = 0; i < numThreads; i++) {
        executor.submit(() -> {
            for (int j = 0; j < incrementsPerThread; j++) {
                stats.incrementTotalPolls();
            }
        });
    }

    executor.shutdown();
    executor.awaitTermination(1, TimeUnit.MINUTES);

    // Verify: Exact count (no lost updates)
    assertThat(stats.getSnapshot().totalPolls())
        .isEqualTo((long) numThreads * incrementsPerThread);
}
```

### Thread Safety Test Checklist

- [ ] **Concurrent access tests**: Multiple threads accessing shared state
- [ ] **Stress tests**: 100+ threads, 1000+ operations per thread
- [ ] **Race condition tests**: Verify atomic operations (no lost updates)
- [ ] **Deadlock tests**: Complex locking scenarios (if applicable)
- [ ] **Immutability tests**: Verify no setters, defensive copies

---

## 🚨 Common Thread Safety Mistakes

### Mistake 1: Non-Atomic Check-Then-Act

```java
❌ WRONG: Race condition
public void offer(DiagnosticData data) {
    if (buffer.size() < capacity) { // Check
        buffer.add(data);            // Act (another thread may have added meanwhile)
    }
}

✅ CORRECT: Atomic operation
public void offer(DiagnosticData data) {
    buffer.offer(data); // ArrayBlockingQueue handles atomicity
}
```

### Mistake 2: Mutable Shared State

```java
❌ WRONG: Mutable shared field
public class Statistics {
    private long totalPolls = 0; // Not thread-safe!

    public void increment() {
        totalPolls++; // Race condition: read-modify-write
    }
}

✅ CORRECT: Atomic variable
public class Statistics {
    private final AtomicLong totalPolls = new AtomicLong(0);

    public void increment() {
        totalPolls.incrementAndGet(); // Atomic operation
    }
}
```

### Mistake 3: Exposing Mutable Internal State

```java
❌ WRONG: Exposing internal array
public class DiagnosticData {
    private final byte[] data;

    public byte[] getData() {
        return data; // Caller can modify internal state!
    }
}

✅ CORRECT: Defensive copy
public class DiagnosticData {
    private final byte[] data;

    public byte[] getData() {
        return Arrays.copyOf(data, data.length); // Safe copy
    }
}
```

### Mistake 4: Synchronized on Long-Running Operation

```java
❌ WRONG: Holding lock during I/O (blocks virtual threads)
public synchronized byte[] pollEndpoint(String url) {
    return httpClient.send(request).body(); // I/O while holding lock!
}

✅ CORRECT: No synchronization for thread-confined code
public byte[] pollEndpoint(String url) {
    // Each thread has its own adapter instance
    return httpClient.send(request).body(); // No shared state
}
```

### Mistake 5: Inconsistent Locking

```java
❌ WRONG: Inconsistent synchronization
public void increment() {
    synchronized (this) {
        count++;
    }
}

public int getCount() {
    return count; // Not synchronized! Can see stale value
}

✅ CORRECT: Consistent synchronization or atomic variable
private final AtomicInteger count = new AtomicInteger(0);

public void increment() {
    count.incrementAndGet();
}

public int getCount() {
    return count.get();
}
```

---

## 📊 Thread Safety Review Checklist

Use this checklist during code reviews:

### Immutability
- [ ] Value objects are `final` classes
- [ ] All fields are `final`
- [ ] No setter methods
- [ ] Defensive copies for mutable fields (arrays, collections)
- [ ] Getters return defensive copies of mutable fields

### Concurrent Collections
- [ ] Use `BlockingQueue` for producer-consumer patterns
- [ ] Use `ArrayBlockingQueue` for bounded buffers
- [ ] Use `ConcurrentHashMap` for thread-safe maps
- [ ] Avoid manual synchronization on collections

### Atomic Variables
- [ ] Use `AtomicLong`/`AtomicInteger` for counters
- [ ] Use atomic operations (`incrementAndGet`, `get`, `compareAndSet`)
- [ ] No read-modify-write with plain `long`/`int`

### Locks
- [ ] Use `ReentrantLock` instead of `synchronized` (for virtual threads)
- [ ] Always unlock in `finally` block
- [ ] No I/O operations while holding lock
- [ ] Lock ordering documented (if multiple locks)

### Virtual Threads
- [ ] Virtual threads used for I/O-bound tasks
- [ ] No `synchronized` on long-running operations
- [ ] No CPU-bound work in virtual threads

### Testing
- [ ] Concurrent access tests exist (multiple threads)
- [ ] Stress tests exist (100+ threads, 1000+ operations)
- [ ] Race condition tests verify atomic operations
- [ ] No flaky tests (deterministic results)

---

## 📚 Resources

### Java Concurrency References
- "Java Concurrency in Practice" by Brian Goetz (Chapter 2-5)
- JDK 25 Documentation: Virtual Threads (JEP 444)
- Java Memory Model (JLS §17.4)

### Internal Documentation
- [Project Implementation Plan](../PROJECT_IMPLEMENTATION_PLAN.md)
- [Architecture Decisions](../ARCHITECTURE_DECISIONS.md)
- [Code Review Guidelines](CODE_REVIEW_GUIDELINES.md)

---

## 🎯 Summary: Thread Safety Mindset

> **"Thread safety is not optional—it's correctness."**

### The Thread Safety Hierarchy (Prefer in Order)

1. **Immutability** (best): No shared mutable state
2. **Thread Confinement**: State not shared between threads
3. **Concurrent Collections**: Use built-in thread-safe collections
4. **Atomic Variables**: For simple shared state (counters, flags)
5. **Locks**: For complex synchronized operations (last resort)

### Key Principles

- **Design for immutability first**: Mutable shared state is the enemy
- **Prefer composition over manual synchronization**: Use `BlockingQueue`, `AtomicLong`, etc.
- **Test concurrency explicitly**: Don't rely on "it works in single-threaded tests"
- **Document thread safety**: Javadoc must state thread safety guarantees

**When in doubt, ask: "What happens if two threads call this at the same time?"**

---

**Document Control**:
- Version: 1.0
- Created: 2025-11-20
- Status: Active
- Review Cycle: After each sprint