hackathon/docs/architecture/system-architecture.md

# HTTP Sender Plugin (HSP) - System Architecture
## Hexagonal Architecture with Complete Requirement Traceability

**Document Version**: 1.1
**Date**: 2025-11-19
**Updated**: 2025-11-19 (Critical Issues Resolved)
**Architect**: System Architect Agent (Hive Mind)
**Status**: Design Complete ✅

---

## Executive Summary

This document defines the complete system architecture for the HTTP Sender Plugin (HSP) using hexagonal architecture (ports and adapters pattern). Every architectural component is traceable to one or more requirements from the requirements catalog.

**Architecture Pattern**: Hexagonal Architecture (Ports and Adapters)
**Technology Stack**: OpenJDK 25, Java 25, gRPC 1.60+, Protocol Buffers 3.25+
**Threading Model**: Virtual threads for HTTP polling, separate threads for gRPC transmission
**Design Pattern**: Producer-Consumer with circular buffer

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [Core Domain Layer](#2-core-domain-layer)
3. [Primary Adapters (Inbound)](#3-primary-adapters-inbound)
4. [Secondary Adapters (Outbound)](#4-secondary-adapters-outbound)
5. [Application Layer](#5-application-layer)
6. [Threading Architecture](#6-threading-architecture)
7. [Data Flow Architecture](#7-data-flow-architecture)
8. [Configuration Architecture](#8-configuration-architecture)
9. [Error Handling Architecture](#9-error-handling-architecture)
10. [Health Monitoring Architecture](#10-health-monitoring-architecture)
11. [Deployment Architecture](#11-deployment-architecture)

---

## 1. Architecture Overview

### 1.1 Hexagonal Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                      PRIMARY ADAPTERS (Inbound)                     │
│   ┌──────────────────┐  ┌──────────────────┐  ┌────────────────┐  │
│   │ Configuration    │  │ Health Check     │  │ Main           │  │
│   │ File Adapter     │  │ HTTP Adapter     │  │ Application    │  │
│   │ (Req-FR-9,10)    │  │ (Req-NFR-7,8)    │  │ (Req-FR-1-8)   │  │
│   └────────┬─────────┘  └────────┬─────────┘  └────────┬───────┘  │
│            │                     │                       │           │
│            ▼                     ▼                       ▼           │
│   ┌─────────────────────────────────────────────────────────────┐  │
│   │                  PRIMARY PORTS (Inbound)                     │  │
│   │  • IConfigurationPort     (Req-FR-9-13)                     │  │
│   │  • IHealthCheckPort       (Req-Test-1-2)                    │  │
│   │  • ILifecyclePort         (Req-FR-1-8)                      │  │
│   └───────────────────────────┬─────────────────────────────────┘  │
│                               │                                     │
│   ┌───────────────────────────▼─────────────────────────────────┐  │
│   │                                                               │  │
│   │                  CORE DOMAIN (Hexagon Center)               │  │
│   │                    Business Logic Layer                      │  │
│   │                                                               │  │
│   │  ┌─────────────────────────────────────────────────────┐    │  │
│   │  │ DataCollectionService                                │    │  │
│   │  │ • Orchestrates HTTP polling (Req-FR-14-21)          │    │  │
│   │  │ • Data validation (Req-FR-21)                       │    │  │
│   │  │ • JSON serialization (Req-FR-22-24)                 │    │  │
│   │  └─────────────────────────────────────────────────────┘    │  │
│   │                                                               │  │
│   │  ┌─────────────────────────────────────────────────────┐    │  │
│   │  │ DataTransmissionService                              │    │  │
│   │  │ • Send data to Collector Sender Core (Req-FR-25)   │    │  │
│   │  │ • Manages gRPC streaming (Req-FR-28-33)             │    │  │
│   │  │ • Message batching (4MB max, 1s timeout)            │    │  │
│   │  │ • Connection management (Req-FR-6, Req-FR-30)       │    │  │
│   │  └─────────────────────────────────────────────────────┘    │  │
│   │                                                               │  │
│   │  ┌─────────────────────────────────────────────────────┐    │  │
│   │  │ ConfigurationManager                                 │    │  │
│   │  │ • Configuration validation (Req-FR-11)              │    │  │
│   │  │ • Configuration domain model (Req-FR-9-13)          │    │  │
│   │  └─────────────────────────────────────────────────────┘    │  │
│   │                                                               │  │
│   │  ┌─────────────────────────────────────────────────────┐    │  │
│   │  │ BufferManager                                        │    │  │
│   │  │ • Circular buffer (Req-FR-26, Req-FR-27)            │    │  │
│   │  │ • Thread-safe collections (Req-Arch-8)              │    │  │
│   │  │ • FIFO overflow (Req-FR-27)                         │    │  │
│   │  │ • Producer-Consumer coordination (Req-Arch-7)       │    │  │
│   │  └─────────────────────────────────────────────────────┘    │  │
│   │                                                               │  │
│   └───────────────────────────┬─────────────────────────────────┘  │
│                               │                                     │
│   ┌───────────────────────────▼─────────────────────────────────┐  │
│   │                 SECONDARY PORTS (Outbound)                   │  │
│   │  • IHttpPollingPort       (Req-FR-14-21)                    │  │
│   │  • IGrpcStreamPort        (Req-FR-25, Req-FR-28-33)        │  │
│   │  • ILoggingPort           (Req-Arch-3,4)                    │  │
│   │  • IBufferPort            (Req-FR-26,27)                    │  │
│   └───────────────────────────┬─────────────────────────────────┘  │
│                               │                                     │
│            ┌──────────────────┼──────────────────┐                 │
│            ▼                  ▼                  ▼                 │
│   ┌────────────────┐  ┌────────────────┐  ┌────────────────┐     │
│   │ HTTP Polling   │  │ gRPC Stream    │  │ File Logging   │     │
│   │ Adapter        │  │ Adapter        │  │ Adapter        │     │
│   │ (IF1)          │  │ (IF2)          │  │ (Req-Arch-3,4) │     │
│   │ Virtual Threads│  │ Platform Thread│  │ Rotation       │     │
│   │ (Req-Arch-6)   │  │ (Req-Arch-6)   │  │ 100MB × 5      │     │
│   └────────────────┘  └────────────────┘  └────────────────┘     │
│                    SECONDARY ADAPTERS (Outbound)                   │
└─────────────────────────────────────────────────────────────────────┘
```

### 1.2 Architecture Principles

**Requirement Traceability**: Req-Norm-6 (maintainable design)

1. **Dependency Rule**: Dependencies point inward toward the domain
2. **Port Isolation**: External systems accessed only through ports
3. **Technology Independence**: Core domain has zero external dependencies
4. **Testability**: All components mockable through port interfaces
5. **Thread Safety**: Explicit thread safety at boundaries

---

## 2. Core Domain Layer

**Package**: `com.siemens.coreshield.hsp.domain`
**Purpose**: Pure business logic with no external dependencies

### 2.1 DataCollectionService

**Requirements**: Req-FR-14, Req-FR-16, Req-FR-21, Req-FR-22, Req-FR-23, Req-FR-24

**Responsibilities**:
- Orchestrate HTTP endpoint polling
- Validate collected data (1MB size limit)
- Serialize data to JSON with Base64 encoding
- Coordinate with BufferManager for data storage

**Component Interface**:
```java
public interface IDataCollectionService {
    /**
     * Start periodic data collection from configured endpoints
     * Req-FR-14: HTTP connection establishment
     * Req-FR-16: Polling at configured intervals
     */
    void startCollection();

    /**
     * Stop data collection gracefully
     * Req-Arch-5: Graceful shutdown
     */
    void stopCollection();

    /**
     * Get current collection statistics
     * Req-NFR-8: Collection metrics for health check
     */
    CollectionStatistics getStatistics();
}
```

**Key Methods**:
```java
/**
 * Collect data from a single endpoint
 * Req-FR-21: Validate data size (max 1MB)
 * Req-FR-22: JSON serialization
 * Req-FR-23: Base64 encoding
 * Req-FR-24: JSON structure (plugin_name, timestamp, source_endpoint, data_size, payload)
 */
private DiagnosticData collectFromEndpoint(String endpointUrl);

/**
 * Validate diagnostic data
 * Req-FR-21: Reject files > 1MB, log warning
 */
private ValidationResult validateData(byte[] data, String sourceUrl);
```

**Thread Safety**:
- Uses IHttpPollingPort which manages virtual thread pools (Req-Arch-6)
- All state synchronized through BufferManager (Req-Arch-8)

**Testing Requirements**:
- Mock IHttpPollingPort for unit tests (Req-NFR-9)
- Integration test with mock HTTP server (Req-NFR-7 testing)

---

### 2.2 DataTransmissionService

**Requirements**: Req-FR-25, Req-FR-27, Req-FR-28, Req-FR-29, Req-FR-30, Req-FR-31, Req-FR-32

**Responsibilities**:
- Send collected and aggregated data to Collector Sender Core (Req-FR-25)
- Manage single bidirectional gRPC stream (Req-FR-28, Req-FR-29)
- Batch messages up to 4MB (Req-FR-31)
- Send batches within 1 second if not full (Req-FR-32)
- Handle connection failures with retry (Req-FR-30)
- Buffer data when transmission fails (Req-FR-26, Req-FR-27)

**Component Interface**:
```java
public interface IDataTransmissionService {
    /**
     * Establish gRPC connection to Collector Sender Core
     * Req-FR-25: Send data to Collector Sender Core
     * Req-FR-28: Single bidirectional stream
     * Req-FR-29: Maintain for lifetime of application
     */
    void connect() throws ConnectionException;

    /**
     * Transmit diagnostic data to Collector Sender Core
     * Req-FR-25: Send collected and aggregated data
     * Req-FR-31: Batch up to 4MB
     * Req-FR-32: Max 1s latency, receiver_id = 99
     */
    void transmit(DiagnosticData data);

    /**
     * Get connection status
     * Req-NFR-8: gRPC connection status for health check
     */
    ConnectionStatus getConnectionStatus();
}
```

**Key Methods**:
```java
/**
 * Batch messages into TransferRequest
 * Req-FR-30: Max 4MB per request (4,194,304 bytes)
 * Req-FR-32: Set receiver_id to 99
 */
private TransferRequest batchMessages(List<DiagnosticData> messages);

/**
 * Handle stream failure and reconnection
 * Req-FR-29: Close stream, wait 5s, re-establish
 * Req-FR-6: Retry every 5s, log warnings every 1 min
 */
private void handleStreamFailure();
```

**Thread Safety**:
- Single consumer thread from BufferManager (Req-Arch-7)
- Synchronized access to gRPC stream (not thread-safe)

**Testing Requirements**:
- Mock IGrpcStreamPort for unit tests
- Integration test with mock gRPC server (Req-NFR-8 testing)

---

### 2.3 ConfigurationManager

**Requirements**: Req-FR-9, Req-FR-10, Req-FR-11, Req-FR-12, Req-FR-13

**Responsibilities**:
- Load configuration from file
- Validate all configuration parameters
- Provide configuration to other components
- Terminate application on validation failure

**Component Interface**:
```java
public interface IConfigurationManager {
    /**
     * Load configuration from file
     * Req-FR-10: Read from application directory
     */
    Configuration loadConfiguration() throws ConfigurationException;

    /**
     * Validate configuration
     * Req-FR-11: Validate all parameters within limits
     */
    ValidationResult validateConfiguration(Configuration config);
}
```

**Configuration Model**:
```java
public final class Configuration {
    // gRPC Configuration (Req-FR-28-33)
    private final String grpcServerAddress;
    private final int grpcServerPort;
    private final int grpcTimeoutSeconds;

    // HTTP Configuration (Req-FR-14-21)
    private final List<String> httpEndpoints;  // Max 1000 (Req-NFR-1)
    private final int pollingIntervalSeconds;  // 1-3600
    private final int requestTimeoutSeconds;   // Default 30 (Req-FR-15)
    private final int maxRetries;              // Default 3 (Req-FR-17)
    private final int retryIntervalSeconds;    // Default 5 (Req-FR-17)

    // Buffer Configuration (Req-FR-26, Req-FR-27)
    private final int bufferMaxMessages;       // Default 300

    // Backoff Configuration (Req-FR-18, Req-FR-6)
    private final int httpBackoffStartSeconds;     // Default 5
    private final int httpBackoffMaxSeconds;       // Default 300
    private final int httpBackoffIncrementSeconds; // Default 5
    private final int grpcRetryIntervalSeconds;    // Default 5
}
```

**Validation Rules** (Req-FR-11):
```java
private ValidationResult validateConfiguration(Configuration config) {
    // gRPC validation
    validatePort(config.grpcServerPort, 1, 65535);
    validateTimeout(config.grpcTimeoutSeconds, 1, 3600);

    // HTTP validation
    validateEndpointCount(config.httpEndpoints, 1, 1000); // Req-NFR-1
    validatePollingInterval(config.pollingIntervalSeconds, 1, 3600);
    validateTimeout(config.requestTimeoutSeconds, 1, 300);
    validateRetries(config.maxRetries, 0, 10);

    // Buffer validation
    validateBufferSize(config.bufferMaxMessages, 1, 1000000);

    // Backoff validation
    validateBackoff(
        config.httpBackoffStartSeconds,
        config.httpBackoffMaxSeconds,
        config.httpBackoffIncrementSeconds
    );
}
```

**Error Handling**:
- Req-FR-12: Terminate with exit code 1 on validation failure
- Req-FR-13: Log validation failure reason

**Thread Safety**: Immutable configuration object, thread-safe

---

### 2.4 BufferManager

**Requirements**: Req-FR-26, Req-FR-27, Req-Arch-7, Req-Arch-8

**Responsibilities**:
- Implement circular buffer with configurable capacity
- Thread-safe producer-consumer coordination
- FIFO overflow handling (discard oldest)
- Buffer statistics for health monitoring

**Component Interface**:
```java
public interface IBufferManager {
    /**
     * Producer: Add data to buffer
     * Req-FR-26: Buffer collected data
     * Req-FR-27: Discard oldest if full
     */
    boolean offer(DiagnosticData data);

    /**
     * Consumer: Take data from buffer
     * Req-FR-26: Consumer reads from buffer
     */
    Optional<DiagnosticData> poll();

    /**
     * Get buffer statistics
     * Req-NFR-8: Buffer metrics for health check
     */
    BufferStatistics getStatistics();

    /**
     * Get current buffer capacity
     */
    int remainingCapacity();
}
```

**Implementation Details**:
```java
public class BufferManager implements IBufferManager {
    // Req-Arch-8: Thread-safe collection
    private final BlockingQueue<DiagnosticData> buffer;
    private final int capacity;
    private final AtomicLong offeredCount = new AtomicLong(0);
    private final AtomicLong droppedCount = new AtomicLong(0);

    public BufferManager(int capacity) {
        // Use ArrayBlockingQueue for bounded FIFO behavior
        this.buffer = new ArrayBlockingQueue<>(capacity);
        this.capacity = capacity;
    }

    /**
     * Req-FR-27: Drop oldest when full
     */
    @Override
    public boolean offer(DiagnosticData data) {
        offeredCount.incrementAndGet();

        if (!buffer.offer(data)) {
            // Buffer full, remove oldest and retry
            buffer.poll(); // Remove oldest
            droppedCount.incrementAndGet();
            return buffer.offer(data); // Add new
        }
        return true;
    }

    @Override
    public Optional<DiagnosticData> poll() {
        return Optional.ofNullable(buffer.poll());
    }
}
```

**Thread Safety** (Req-Arch-8):
- `ArrayBlockingQueue` provides thread-safe operations
- `AtomicLong` for thread-safe statistics
- No explicit synchronization needed

**Producer-Consumer Pattern** (Req-Arch-7):
- Multiple producer threads (HTTP polling via virtual threads)
- Single consumer thread (gRPC transmission)
- Blocking queue provides coordination

---

## 3. Primary Adapters (Inbound)

**Package**: `com.siemens.coreshield.hsp.adapter.inbound`

### 3.1 ConfigurationFileAdapter

**Requirements**: Req-FR-9, Req-FR-10, Req-FR-11, Req-FR-12, Req-FR-13

**Purpose**: Load configuration from JSON file

**Port Interface**: `IConfigurationPort`

**Implementation**:
```java
public class ConfigurationFileAdapter implements IConfigurationPort {
    private static final String CONFIG_FILE = "./hsp-config.json";
    private final ObjectMapper jsonMapper;

    /**
     * Req-FR-10: Load from application directory
     */
    @Override
    public Configuration loadConfiguration() throws ConfigurationException {
        try {
            File configFile = new File(CONFIG_FILE);
            return jsonMapper.readValue(configFile, Configuration.class);
        } catch (IOException e) {
            // Req-FR-13: Log validation failure
            logger.error("Failed to load configuration", e);
            throw new ConfigurationException("Configuration file not found or invalid", e);
        }
    }
}
```

**Thread Safety**: Stateless adapter, thread-safe

**Testing**: Unit test with sample configuration files

---

### 3.2 HealthCheckController

**Requirements**: Req-Test-1, Req-Test-2

**Purpose**: Expose HTTP health check endpoint

**Port Interface**: `IHealthCheckPort`

**Implementation**:
```java
public class HealthCheckController implements IHealthCheckPort {
    private final IDataCollectionService collectionService;
    private final IDataTransmissionService transmissionService;
    private final IBufferManager bufferManager;

    /**
     * Req-Test-1: GET localhost:8080/health
     * Req-Test-2: Return JSON with component status
     */
    @Override
    public HealthCheckResponse getHealthStatus() {
        return new HealthCheckResponse(
            determineServiceStatus(),
            collectionService.getStatistics().getLastSuccessfulCollection(),
            transmissionService.getConnectionStatus().isConnected(),
            collectionService.getStatistics().getErrorCount(),
            collectionService.getStatistics().getSuccessCount30s(),
            collectionService.getStatistics().getFailedCount30s()
        );
    }

    private ServiceStatus determineServiceStatus() {
        boolean grpcConnected = transmissionService.getConnectionStatus().isConnected();
        boolean collectionRunning = collectionService.getStatistics().isRunning();

        if (grpcConnected && collectionRunning) {
            return ServiceStatus.RUNNING;
        } else if (collectionRunning) {
            return ServiceStatus.DEGRADED;
        } else {
            return ServiceStatus.DOWN;
        }
    }
}
```

**JSON Response Schema** (Req-Test-2):
```json
{
  "service_status": "RUNNING | DEGRADED | DOWN",
  "last_successful_collection_ts": "2025-11-19T10:52:10Z",
  "grpc_connection_status": "CONNECTED | DISCONNECTED",
  "http_collection_error_count": 15,
  "endpoints_success_last_30s": 998,
  "endpoints_failed_last_30s": 2
}
```

**Thread Safety**: Read-only access to statistics, thread-safe

**Testing**: Integration test with HTTP client, verify JSON schema

---

## 4. Secondary Adapters (Outbound)

**Package**: `com.siemens.coreshield.hsp.adapter.outbound`

### 4.1 HttpPollingAdapter

**Requirements**: Req-FR-14, Req-FR-15, Req-FR-16, Req-FR-17, Req-FR-18, Req-FR-19, Req-FR-20, Req-FR-21

**Purpose**: Poll HTTP endpoints and retrieve diagnostic data

**Port Interface**: `IHttpPollingPort`

**Implementation**:
```java
public class HttpPollingAdapter implements IHttpPollingPort {
    private final HttpClient httpClient;
    private final Map<String, Semaphore> endpointLocks; // Req-FR-19: No concurrent connections

    /**
     * Req-FR-15: HTTP GET with 30s timeout
     * Req-FR-17: Retry 3 times with 5s intervals
     * Req-FR-18: Linear backoff 5s → 300s
     */
    @Override
    public CompletableFuture<byte[]> pollEndpoint(String endpointUrl) {
        // Req-FR-19: Acquire endpoint lock
        Semaphore lock = endpointLocks.computeIfAbsent(endpointUrl, k -> new Semaphore(1));
        lock.acquire();

        try {
            return pollWithRetry(endpointUrl, 0);
        } finally {
            lock.release();
        }
    }

    private CompletableFuture<byte[]> pollWithRetry(String url, int attempt) {
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create(url))
            .timeout(Duration.ofSeconds(30)) // Req-FR-15
            .GET()
            .build();

        return httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofByteArray())
            .thenApply(response -> {
                if (response.statusCode() == 200) {
                    byte[] data = response.body();
                    // Req-FR-21: Validate size (max 1MB)
                    if (data.length > 1_048_576) {
                        logger.warn("Endpoint {} returned data > 1MB: {} bytes", url, data.length);
                        throw new OversizedDataException(url, data.length);
                    }
                    return data;
                } else {
                    throw new HttpException("HTTP " + response.statusCode());
                }
            })
            .exceptionally(ex -> {
                if (attempt < 3) { // Req-FR-17: Max 3 retries
                    // Req-FR-17: Wait 5s between retries
                    Thread.sleep(5000);
                    return pollWithRetry(url, attempt + 1).join();
                } else {
                    // Req-FR-18: Linear backoff for subsequent polls
                    scheduleBackoff(url, attempt);
                    throw new PollingFailedException(url, ex);
                }
            });
    }

    /**
     * Req-FR-18: Linear backoff: start=5s, max=300s, increment=5s
     */
    private void scheduleBackoff(String url, int failureCount) {
        int delaySec = Math.min(5 + (failureCount * 5), 300);
        // Schedule next poll after delay
    }
}
```

**Thread Safety** (Req-Arch-6):
- `HttpClient` is thread-safe (Java 11+)
- Semaphore per endpoint ensures no concurrent connections (Req-FR-19)
- Virtual threads used for polling (managed by executor)

**Fault Isolation** (Req-FR-20):
- Failures isolated per endpoint
- Exception handling prevents cascade failures

**Testing**:
- Mock HTTP server for integration tests (Req-NFR-7 testing)
- Unit tests for retry logic and backoff calculation

---

### 4.2 GrpcStreamAdapter

**Requirements**: Req-FR-28, Req-FR-29, Req-FR-30, Req-FR-31, Req-FR-32, Req-FR-33

**Purpose**: Manage gRPC streaming to Collector Sender Core

**Port Interface**: `IGrpcStreamPort`

**Implementation**:
```java
public class GrpcStreamAdapter implements IGrpcStreamPort {
    private ManagedChannel channel;
    private TransferServiceGrpc.TransferServiceStub asyncStub;
    private StreamObserver<TransferRequest> requestStream;
    private final Object streamLock = new Object(); // gRPC streams not thread-safe

    /**
     * Req-FR-29: Establish single bidirectional stream
     */
    @Override
    public void connect(String host, int port) {
        channel = ManagedChannelBuilder.forAddress(host, port)
            .usePlaintext() // Req-NFR-4: TCP mode only
            .build();

        asyncStub = TransferServiceGrpc.newStub(channel);

        StreamObserver<TransferResponse> responseObserver = new StreamObserver<>() {
            @Override
            public void onNext(TransferResponse response) {
                logger.info("Received response code: {}", response.getResponseCode());
            }

            @Override
            public void onError(Throwable t) {
                // Req-FR-30: Handle stream failure
                handleStreamFailure(t);
            }

            @Override
            public void onCompleted() {
                logger.info("Stream completed");
            }
        };

        synchronized (streamLock) {
            requestStream = asyncStub.transferStream(responseObserver);
        }
    }

    /**
     * Req-FR-32: Send batch (max 4MB)
     * Req-FR-33: receiver_id = 99
     */
    @Override
    public void sendBatch(List<DiagnosticData> batch) {
        // Serialize batch to JSON
        ByteString data = serializeBatch(batch);

        // Req-FR-31: Validate size (max 4MB)
        if (data.size() > 4_194_304) {
            throw new OversizedBatchException(data.size());
        }

        TransferRequest request = TransferRequest.newBuilder()
            .setReceiverId(99) // Req-FR-33
            .setData(data)
            .build();

        synchronized (streamLock) {
            requestStream.onNext(request);
        }
    }

    /**
     * Req-FR-30: Close stream, wait 5s, re-establish
     * Req-FR-6: Log warnings every 1 minute
     */
    private void handleStreamFailure(Throwable error) {
        logger.error("gRPC stream failed", error);

        synchronized (streamLock) {
            if (requestStream != null) {
                requestStream.onCompleted();
            }
            if (channel != null) {
                channel.shutdown();
            }
        }

        // Wait 5s before reconnection
        ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
        scheduler.schedule(() -> {
            try {
                connect(host, port);
                logger.info("gRPC stream reconnected successfully");
            } catch (Exception e) {
                logger.warn("gRPC reconnection failed, will retry");
                handleStreamFailure(e);
            }
        }, 5, TimeUnit.SECONDS);
    }
}
```

**Thread Safety**:
- `synchronized` block around stream operations (gRPC streams not thread-safe)
- Single consumer thread from BufferManager

**Connection Management** (Req-FR-6):
- Retry every 5 seconds indefinitely
- Log warnings every 1 minute (implemented in caller)

**Testing**:
- Mock gRPC server for integration tests (Req-NFR-8 testing)
- Unit tests for batch serialization and size validation

---

### 4.3 FileLoggingAdapter

**Requirements**: Req-Arch-3, Req-Arch-4

**Purpose**: Log to file with rotation

**Port Interface**: `ILoggingPort`

**Implementation**:
```java
public class FileLoggingAdapter implements ILoggingPort {
    private static final Logger logger = Logger.getLogger(FileLoggingAdapter.class.getName());

    /**
     * Req-Arch-3: Log to hsp.log in temp directory
     * Req-Arch-4: Java Logging API with rotation (100MB, 5 files)
     */
    public FileLoggingAdapter() throws IOException {
        String logDir = System.getProperty("java.io.tmpdir");
        String logFile = logDir + File.separator + "hsp.log";

        FileHandler fileHandler = new FileHandler(
            logFile,
            100 * 1024 * 1024,  // Req-Arch-4: 100MB per file
            5,                   // Req-Arch-4: 5 files
            true                 // Append mode
        );

        fileHandler.setFormatter(new SimpleFormatter());
        logger.addHandler(fileHandler);
        logger.setLevel(Level.ALL);
    }

    @Override
    public void info(String message) {
        logger.info(message);
    }

    @Override
    public void warn(String message) {
        logger.warning(message);
    }

    @Override
    public void error(String message, Throwable error) {
        logger.log(Level.SEVERE, message, error);
    }
}
```

**Thread Safety**: Java Logger is thread-safe

**Testing**: Integration test with file creation and rotation validation

---

## 5. Application Layer

**Package**: `com.siemens.coreshield.hsp.application`

### 5.1 Startup Orchestration

**Requirements**: Req-FR-1, Req-FR-2, Req-FR-3, Req-FR-4, Req-FR-5, Req-FR-6, Req-FR-7, Req-FR-8

**Main Application**:
```java
public class HspApplication {
    public static void main(String[] args) {
        try {
            // Req-FR-2: Load and validate configuration
            IConfigurationPort configPort = new ConfigurationFileAdapter();
            Configuration config = configPort.loadConfiguration();

            IConfigurationManager configManager = new ConfigurationManager();
            ValidationResult validation = configManager.validateConfiguration(config);

            if (!validation.isValid()) {
                // Req-FR-13: Log validation failure
                logger.error("Configuration validation failed: {}", validation.getErrors());
                // Req-FR-12: Terminate with exit code 1
                System.exit(1);
            }

            // Req-FR-3: Initialize logging
            ILoggingPort loggingPort = new FileLoggingAdapter();

            // Req-FR-4: Establish gRPC connection
            IGrpcStreamPort grpcPort = new GrpcStreamAdapter();
            IDataTransmissionService transmissionService = new DataTransmissionService(grpcPort, config);

            // Req-FR-6: Retry gRPC connection every 5s until successful
            while (!transmissionService.isConnected()) {
                try {
                    transmissionService.connect();
                } catch (ConnectionException e) {
                    logger.warn("gRPC connection failed, retrying in 5s");
                    Thread.sleep(5000);
                }
            }

            // Req-FR-7: Wait for gRPC before starting HTTP polling
            if (transmissionService.isConnected()) {
                // Req-FR-5: Begin HTTP polling
                IHttpPollingPort httpPort = new HttpPollingAdapter(config);
                IBufferManager bufferManager = new BufferManager(config.getBufferMaxMessages());
                IDataCollectionService collectionService = new DataCollectionService(httpPort, bufferManager, config);

                collectionService.startCollection();

                // Req-FR-8: Log successful startup
                logger.info("HSP started successfully");

                // Start transmission service consumer thread
                transmissionService.startConsuming(bufferManager);

                // Start health check endpoint
                HealthCheckController healthCheck = new HealthCheckController(collectionService, transmissionService, bufferManager);
                startHealthCheckServer(healthCheck); // Req-NFR-7
            }

            // Req-Arch-5: Run indefinitely unless unrecoverable error
            awaitTermination();

        } catch (Exception e) {
            logger.error("Fatal error during startup", e);
            System.exit(1);
        }
    }

    /**
     * Req-Arch-5: Always run unless unrecoverable error
     */
    private static void awaitTermination() {
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            logger.info("Shutdown signal received, stopping HSP");
            // Graceful shutdown logic
        }));

        // Block main thread
        Thread.currentThread().join();
    }
}
```

---

## 6. Threading Architecture

**Requirements**: Req-Arch-6, Req-Arch-7, Req-NFR-1

### 6.1 Thread Model

```
┌─────────────────────────────────────────────────────────────┐
│                      MAIN THREAD                             │
│  • Application startup (Req-FR-1-8)                         │
│  • Configuration loading (Req-FR-2)                         │
│  • Component initialization (Req-FR-3-5)                    │
└───────────────────────┬─────────────────────────────────────┘
                        │
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────────┐
│ VIRTUAL       │ │ PLATFORM      │ │ HEALTH CHECK      │
│ THREAD POOL   │ │ THREAD        │ │ HTTP SERVER       │
│ (HTTP Polling)│ │ (gRPC         │ │ (localhost:8080)  │
│               │ │  Consumer)    │ │                   │
│ Req-Arch-6    │ │ Req-Arch-6    │ │ Req-NFR-7         │
│ Req-NFR-1     │ │               │ │                   │
└───────┬───────┘ └───────┬───────┘ └─────────────────┘
        │               │
        │               │
┌───────▼───────────────▼───────┐
│   CIRCULAR BUFFER             │
│   (Thread-Safe Queue)         │
│   Req-Arch-7, Req-Arch-8      │
│   Req-FR-26, Req-FR-27        │
│   • Producer-Consumer Pattern │
│   • Max 300 messages          │
│   • FIFO overflow handling    │
└───────────────────────────────┘
```

### 6.2 Virtual Thread Pool (HTTP Polling)

**Requirements**: Req-Arch-6, Req-NFR-1

**Configuration**:
```java
ExecutorService virtualThreadPool = Executors.newVirtualThreadPerTaskExecutor();

// Submit polling tasks for each endpoint
for (String endpoint : config.getHttpEndpoints()) {
    virtualThreadPool.submit(() -> {
        while (running) {
            pollEndpoint(endpoint);
            Thread.sleep(config.getPollingIntervalSeconds() * 1000);
        }
    });
}
```

**Benefits**:
- Supports 1000+ concurrent connections (Req-NFR-1)
- Low memory footprint compared to platform threads
- Automatic thread management by JVM

**Thread Safety**:
- Each virtual thread handles one endpoint
- No shared mutable state between threads
- Buffer access synchronized via thread-safe queue

---

### 6.3 Platform Thread (gRPC Consumer)

**Requirements**: Req-Arch-6, Req-Arch-7

**Configuration**:
```java
Thread consumerThread = new Thread(() -> {
    while (running) {
        Optional<DiagnosticData> data = bufferManager.poll();

        if (data.isPresent()) {
            batch.add(data.get());

            // Req-FR-31: Send when batch reaches 4MB
            if (getBatchSize(batch) >= 4_194_304) {
                grpcPort.sendBatch(batch);
                batch.clear();
            }

            // Req-FR-32: Send within 1s if not full
            if (Duration.between(batchStartTime, Instant.now()).toMillis() >= 1000) {
                grpcPort.sendBatch(batch);
                batch.clear();
            }
        } else {
            Thread.sleep(10); // Buffer empty, brief wait
        }
    }
});
consumerThread.start();
```

**Thread Safety**:
- Single consumer thread (no contention)
- Reads from thread-safe buffer
- Synchronized access to gRPC stream

---

## 7. Data Flow Architecture

**Requirements**: Req-Arch-7, Req-FR-14-32

### 7.1 Data Flow Diagram

```
┌──────────────────────────────────────────────────────────────────┐
│                    HTTP ENDPOINTS (IF1)                          │
│  • Endpoint 1 → http://device1.local:8080/diagnostics           │
│  • Endpoint 2 → http://device2.local:8080/diagnostics           │
│  • ...                                                           │
│  • Endpoint N → http://deviceN.local:8080/diagnostics (max 1000)│
└────────────────────────┬─────────────────────────────────────────┘
                         │
                         │ HTTP GET (Req-FR-15)
                         │ Timeout: 30s
                         │ Retry: 3x, 5s interval (Req-FR-17)
                         │ Backoff: 5s → 300s (Req-FR-18)
                         ▼
        ┌────────────────────────────────────┐
        │  VIRTUAL THREAD POOL               │
        │  (HttpPollingAdapter)              │
        │  • Poll each endpoint              │
        │  • Validate size ≤ 1MB (Req-FR-21) │
        └────────────────┬───────────────────┘
                         │
                         │ Binary data (max 1MB)
                         ▼
        ┌────────────────────────────────────┐
        │  DATA COLLECTION SERVICE           │
        │  • JSON serialization (Req-FR-22)  │
        │  • Base64 encoding (Req-FR-23)     │
        │  • Add metadata (Req-FR-24)        │
        └────────────────┬───────────────────┘
                         │
                         │ DiagnosticData
                         │ {plugin_name, timestamp, source_endpoint,
                         │  data_size, payload}
                         ▼
        ┌────────────────────────────────────┐
        │  CIRCULAR BUFFER (Req-FR-26)       │
        │  • Capacity: 300 messages          │
        │  • FIFO overflow (Req-FR-26)       │
        │  • Thread-safe queue               │
        └────────────────┬───────────────────┘
                         │
                         │ Poll() - Non-blocking
                         ▼
        ┌────────────────────────────────────┐
        │  DATA TRANSMISSION SERVICE         │
        │  • Batch messages                  │
        │  • Max 4MB per batch (Req-FR-30)   │
        │  • Max 1s latency (Req-FR-31)      │
        └────────────────┬───────────────────┘
                         │
                         │ TransferRequest
                         │ {receiver_id: 99, data: bytes}
                         ▼
        ┌────────────────────────────────────┐
        │  GRPC STREAM ADAPTER (IF2)         │
        │  • Bidirectional stream            │
        │  • Auto-reconnect (Req-FR-29)      │
        │  • TCP mode (Req-NFR-4)            │
        └────────────────┬───────────────────┘
                         │
                         │ gRPC over TCP
                         ▼
┌──────────────────────────────────────────────────────────────────┐
│              COLLECTOR SENDER CORE (IF2)                         │
│  • Receives TransferRequest messages                            │
│  • Returns TransferResponse                                     │
└──────────────────────────────────────────────────────────────────┘
```

### 7.2 Data Transformation Pipeline

**Stage 1: HTTP Collection** (Req-FR-14-21)
- Input: HTTP endpoint URL
- Output: `byte[] diagnosticData`
- Validation: Size ≤ 1MB
- Error Handling: Retry 3x, then backoff

**Stage 2: JSON Serialization** (Req-FR-22-24)
- Input: `byte[] diagnosticData`
- Processing:
  1. Base64 encode binary data
  2. Create JSON structure with metadata
  3. Calculate data_size field
- Output: `DiagnosticData` object

**DiagnosticData Structure**:
```json
{
  "plugin_name": "HTTP sender plugin",
  "timestamp": "2025-11-19T10:52:10.123Z",  // ISO 8601
  "source_endpoint": "http://device1.local:8080/diagnostics",
  "data_size": 524288,  // bytes
  "payload": "SGVsbG8gV29ybGQh..."  // Base64 encoded
}
```

**Stage 3: Buffering** (Req-FR-26-27)
- Input: `DiagnosticData` object
- Storage: Circular buffer (FIFO)
- Capacity: 300 messages
- Overflow: Discard oldest message

**Stage 4: Batching** (Req-FR-31-32)
- Input: `List<DiagnosticData>`
- Constraints:
  - Max size: 4MB (4,194,304 bytes)
  - Max latency: 1 second
- Output: `TransferRequest` protobuf

**Stage 5: gRPC Transmission** (Req-FR-28-33)
- Input: `TransferRequest`
- Protocol: gRPC bidirectional stream
- receiver_id: 99 (constant)
- Output: `TransferResponse`

---

## 8. Configuration Architecture

**Requirements**: Req-FR-9-13

### 8.1 Configuration File Schema

**File**: `./hsp-config.json`
**Format**: JSON

```json
{
  "grpc": {
    "server_address": "localhost",
    "server_port": 50051,
    "timeout_seconds": 30
  },
  "http": {
    "endpoints": [
      "http://device1.local:8080/diagnostics",
      "http://device2.local:8080/diagnostics"
    ],
    "polling_interval_seconds": 1,
    "request_timeout_seconds": 30,
    "max_retries": 3,
    "retry_interval_seconds": 5
  },
  "buffer": {
    "max_messages": 300
  },
  "backoff": {
    "http_start_seconds": 5,
    "http_max_seconds": 300,
    "http_increment_seconds": 5,
    "grpc_interval_seconds": 5
  }
}
```

### 8.2 Configuration Validation Rules

**gRPC Configuration** (Req-FR-11):
- `server_address`: Non-empty string, valid hostname or IP
- `server_port`: Integer, range 1-65535
- `timeout_seconds`: Integer, range 1-3600

**HTTP Configuration** (Req-FR-11, Req-NFR-1):
- `endpoints`: Array, min 1, max 1000 URLs
- `polling_interval_seconds`: Integer, range 1-3600
- `request_timeout_seconds`: Integer, range 1-300
- `max_retries`: Integer, range 0-10
- `retry_interval_seconds`: Integer, range 1-60

**Buffer Configuration** (Req-FR-11):
- `max_messages`: Integer, range 1-1000000

**Backoff Configuration** (Req-FR-11):
- `http_start_seconds`: Integer, range 1-60
- `http_max_seconds`: Integer, range 60-3600
- `http_increment_seconds`: Integer, range 1-60
- `grpc_interval_seconds`: Integer, range 1-60
- Constraint: `http_start_seconds` ≤ `http_max_seconds`

### 8.3 Validation Failure Handling

**Req-FR-12**: Terminate with exit code 1
**Req-FR-13**: Log validation failure reason

```java
ValidationResult result = configManager.validateConfiguration(config);

if (!result.isValid()) {
    for (ValidationError error : result.getErrors()) {
        logger.error("Configuration validation error: {} - {}",
            error.getField(), error.getMessage());
    }
    logger.error("HSP startup failed due to invalid configuration");
    System.exit(1);
}
```

---

## 9. Error Handling Architecture

**Requirements**: Req-Norm-3, Req-FR-6, Req-FR-17, Req-FR-18, Req-FR-29

### 9.1 Error Categories

**1. HTTP Polling Errors** (Req-FR-17-18, Req-FR-20)
- Network timeout → Retry 3x with 5s interval
- HTTP error (4xx, 5xx) → Retry 3x with 5s interval
- After 3 failures → Linear backoff 5s → 300s
- Continue polling other endpoints (fault isolation)

**2. Data Validation Errors** (Req-FR-21)
- Oversized data (> 1MB) → Log warning, discard data, continue
- Invalid data format → Log error, discard data, continue

**3. gRPC Connection Errors** (Req-FR-6, Req-FR-29)
- Connection failure → Retry every 5s indefinitely
- Stream failure → Close stream, wait 5s, re-establish
- Log warnings every 1 minute during connection failures

**4. Buffer Overflow** (Req-FR-26)
- Buffer full → Discard oldest message, accept new message
- Log dropped message count in statistics

**5. Configuration Errors** (Req-FR-12-13)
- Validation failure → Log errors, terminate with exit code 1
- File not found → Log error, terminate with exit code 1
- Parse error → Log error, terminate with exit code 1

### 9.2 Error Recovery Strategies

**Retry Strategy** (Req-FR-17):
```java
public <T> T executeWithRetry(Supplier<T> operation, int maxRetries) {
    int attempt = 0;
    while (attempt < maxRetries) {
        try {
            return operation.get();
        } catch (Exception e) {
            attempt++;
            if (attempt >= maxRetries) {
                throw e;
            }
            Thread.sleep(5000); // 5s between retries
        }
    }
}
```

**Backoff Strategy** (Req-FR-18):
```java
public int calculateBackoff(int failureCount) {
    int delay = 5 + (failureCount * 5); // Start 5s, increment 5s
    return Math.min(delay, 300); // Max 300s
}
```

**Connection Recovery** (Req-FR-29):
```java
private void recoverConnection() {
    closeStream();
    Thread.sleep(5000); // Wait 5s
    establishStream();
}
```

---

## 10. Health Monitoring Architecture

**Requirements**: Req-NFR-7, Req-NFR-8

### 10.1 Health Check Endpoint

**Endpoint**: `GET localhost:8080/health`
**Response Format**: JSON
**Response Time**: < 100ms

### 10.2 Health Metrics

**Service Status** (Req-NFR-8):
- `RUNNING`: All components operational
- `DEGRADED`: Some components operational (e.g., gRPC disconnected)
- `DOWN`: Critical components failed

**gRPC Connection Status** (Req-NFR-8):
- `CONNECTED`: Stream active and healthy
- `DISCONNECTED`: Stream failed or not established

**Collection Statistics** (Req-NFR-8):
- `last_successful_collection_ts`: ISO 8601 timestamp
- `http_collection_error_count`: Total errors since startup
- `endpoints_success_last_30s`: Successful polls in last 30 seconds
- `endpoints_failed_last_30s`: Failed polls in last 30 seconds

### 10.3 Metrics Collection

**CollectionStatistics**:
```java
public class CollectionStatistics {
    private final AtomicLong totalPolls = new AtomicLong(0);
    private final AtomicLong totalErrors = new AtomicLong(0);
    private volatile Instant lastSuccessfulCollection;
    private final ConcurrentLinkedQueue<PollResult> recentPolls; // Last 30s

    public void recordPoll(String endpoint, boolean success) {
        totalPolls.incrementAndGet();
        if (success) {
            lastSuccessfulCollection = Instant.now();
        } else {
            totalErrors.incrementAndGet();
        }

        // Add to time-windowed queue
        recentPolls.offer(new PollResult(endpoint, success, Instant.now()));

        // Remove polls older than 30s
        cleanupOldPolls();
    }

    public int getSuccessCount30s() {
        return (int) recentPolls.stream()
            .filter(PollResult::isSuccess)
            .count();
    }

    public int getFailedCount30s() {
        return (int) recentPolls.stream()
            .filter(p -> !p.isSuccess())
            .count();
    }
}
```

---

## 11. Deployment Architecture

**Requirements**: Req-Arch-1, Req-Arch-2, Req-NFR-5, Req-NFR-6

### 11.1 Build Configuration

**Build Tool**: Maven 3.9+ (Req-NFR-5)
**Java Version**: OpenJDK 25 / Java 25 (Req-Arch-1)
**Packaging**: Executable Fat JAR (Req-NFR-6)

**pom.xml Configuration**:
```xml
<project>
    <properties>
        <maven.compiler.source>25</maven.compiler.source>
        <maven.compiler.target>25</maven.compiler.target>
        <grpc.version>1.60.0</grpc.version>
        <protobuf.version>3.25.0</protobuf.version>
    </properties>

    <dependencies>
        <!-- Req-Arch-2: gRPC Java 1.60+ -->
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty-shaded</artifactId>
            <version>${grpc.version}</version>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-protobuf</artifactId>
            <version>${grpc.version}</version>
        </dependency>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-stub</artifactId>
            <version>${grpc.version}</version>
        </dependency>

        <!-- Req-Arch-2: Protocol Buffers 3.25+ -->
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>${protobuf.version}</version>
        </dependency>

        <!-- Testing (Req-NFR-9) -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>5.5.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- Req-NFR-6: Fat JAR -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.5.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.siemens.coreshield.hsp.HspApplication</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <!-- Protocol Buffers Compiler -->
            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.6.1</version>
                <configuration>
                    <protocArtifact>com.google.protobuf:protoc:${protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
                    <pluginId>grpc-java</pluginId>
                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>compile-custom</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>
```

### 11.2 Deployment Package Structure

```
hsp-1.0.0.jar (Fat JAR - Req-NFR-6)
├── com/siemens/coreshield/hsp/  (Application classes)
├── com/siemens/coreshield/owg/shared/grpc/  (Generated protobuf classes)
├── io/grpc/  (gRPC dependencies)
├── com/google/protobuf/  (Protobuf dependencies)
└── META-INF/
    └── MANIFEST.MF
        Main-Class: com.siemens.coreshield.hsp.HspApplication

hsp-config.json (Configuration file - Req-FR-10)
```

### 11.3 Runtime Requirements

**JVM**: OpenJDK 25 or compatible (Req-Arch-1)
**Memory**: Max 4096MB (Req-NFR-2)
**Network**:
- Outbound HTTP access to endpoint devices (IF1)
- Outbound TCP access to Collector Sender Core (IF2)
- Inbound HTTP access on localhost:8080 (health check)

**Startup Command**:
```bash
java -Xmx4096m -jar hsp-1.0.0.jar
```

---

## Summary

This system architecture provides complete traceability from requirements to implementation components:

**Total Requirements Traced: 62 ✅
- **Hexagonal architecture** ensures maintainability and testability
- **Thread-safe design** with virtual threads for scalability
- **Producer-consumer pattern** with circular buffer (300 messages)
- **Complete error handling** with retry and backoff strategies
- **Health monitoring** for operational visibility
- **All critical issues resolved** (2025-11-19) ✅

**Next Steps**:
1. Review and approve architecture
2. Implement detailed component design (see component-mapping.md)
3. Begin test-driven development
4. Integration testing with mock servers
5. Performance validation (1000 endpoints, 4GB RAM limit)

---

**Document Metadata**:
- Total Requirements Traced: 62 ✅
- Total Components: 15 major components
- Thread Safety: 8 critical thread-safe components
- Test Classes: 35+ estimated
- Lines of Code: ~5000 estimated
- **Critical Issues**: All resolved (2025-11-19) ✅