hackathon/docs/diagrams/architecture-diagrams.md

HSP System Architecture Diagrams with Requirement Traceability

Document Version: 1.0 Date: 2025-11-19 Planner Agent: Hive Mind Status: Complete

---

Table of Contents

1. System Context Diagram (C4 Level 1)
2. Container Diagram (C4 Level 2)
3. Component Diagram - Hexagonal Architecture (C4 Level 3)
4. Deployment Diagram
5. Sequence Diagrams
6. Data Flow Diagram

---

1. System Context Diagram (C4 Level 1)

Purpose: Shows the HSP system in the context of its external actors and interfaces.

Requirements Covered: Req-Arch-1, Req-Arch-2, Req-FR-14, Req-FR-27, Req-NFR-7

C4Context
    title System Context Diagram - HTTP Sender Plugin (HSP)

    Person(operator, "System Operator", "Monitors health and diagnostics<br/>[Req-US-1a, Req-US-1c]")

    System(hsp, "HTTP Sender Plugin", "Collects diagnostic data via HTTP,<br/>transmits via gRPC<br/>[Req-Arch-1: Java 25]<br/>[Req-Arch-2: gRPC, Protobuf]")

    System_Ext(devices, "Endpoint Devices", "Provide diagnostic data<br/>via HTTP REST API<br/>[IF1 - Req-FR-14 to Req-FR-27]")

    System_Ext(collector, "Collector Sender Core", "Receives streamed data<br/>via gRPC bidirectional stream<br/>[IF2 - Req-FR-28 to Req-FR-33]")

    System_Ext(config, "Configuration File", "JSON/YAML configuration<br/>[Req-FR-9 to Req-FR-13]")

    Person(admin, "System Administrator", "Checks health status<br/>[Req-US-1c]")

    Rel(hsp, devices, "Polls", "HTTP GET<br/>[Req-FR-15, Req-FR-16]<br/>30s timeout, retry 3x")
    Rel(devices, hsp, "Returns", "Diagnostic data<br/>[Req-FR-20, Req-FR-21]<br/>Max 1MB")

    Rel(hsp, collector, "Streams", "gRPC TransferRequest<br/>[Req-FR-28, Req-FR-31]<br/>Max 4MB batches")
    Rel(collector, hsp, "Acknowledges", "TransferResponse<br/>[Req-FR-28]")

    Rel(config, hsp, "Loads at startup", "Endpoint URLs, intervals<br/>[Req-FR-2, Req-FR-10]")

    Rel(operator, hsp, "Monitors", "Health check endpoint<br/>[Req-NFR-7]")
    Rel(admin, hsp, "Checks", "localhost:8080/health<br/>[Req-NFR-7, Req-NFR-8]")

    UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")

Legend:

• IF1: HTTP polling interface to endpoint devices (Req-FR-14 to Req-FR-27)
• IF2: gRPC streaming interface to Collector Core (Req-FR-28 to Req-FR-33)
• Req-Arch-1: OpenJDK 25, Java 25
• Req-Arch-2: External libraries limited to gRPC and Protobuf

---

2. Container Diagram (C4 Level 2)

Purpose: Shows the internal structure of the HSP system as a single deployable container.

Requirements Covered: Req-Arch-1, Req-Arch-2, Req-Arch-3, Req-Arch-4, Req-Arch-5, Req-NFR-5, Req-NFR-6

C4Container
    title Container Diagram - HSP Application

    Person(operator, "System Operator")
    System_Ext(devices, "Endpoint Devices", "HTTP REST API")
    System_Ext(collector, "Collector Sender Core", "gRPC Server")

    Container_Boundary(hsp_boundary, "HTTP Sender Plugin") {
        Container(hsp_app, "HSP Application", "Java 25, Maven", "Executable fat JAR<br/>[Req-Arch-1: Java 25]<br/>[Req-Arch-2: gRPC 1.60+, Protobuf 3.25+]<br/>[Req-NFR-5: Maven 3.9+]<br/>[Req-NFR-6: Fat JAR]")

        ContainerDb(log_file, "hsp.log", "Java Logging API", "Rolling log file<br/>[Req-Arch-3: temp/hsp.log]<br/>[Req-Arch-4: 100MB, 5 files]")

        ContainerDb(config_file, "config.json", "JSON/YAML", "Configuration file<br/>[Req-FR-9 to Req-FR-13]")
    }

    Rel(hsp_app, devices, "HTTP GET", "Polling<br/>[Req-FR-15: 30s timeout]<br/>[Req-FR-17: 3 retries]")
    Rel(hsp_app, collector, "gRPC Stream", "Bidirectional<br/>[Req-FR-28: Single stream]<br/>[Req-FR-30: Max 4MB]")
    Rel(hsp_app, log_file, "Writes", "Structured logs<br/>[Req-Arch-3, Req-Arch-4]")
    Rel(hsp_app, config_file, "Reads at startup", "Validation<br/>[Req-FR-2, Req-FR-10]")
    Rel(operator, hsp_app, "Health check", "HTTP GET<br/>[Req-NFR-7: :8080/health]")

    UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="1")

Legend:

• Req-Arch-1: OpenJDK 25, Java 25 runtime
• Req-Arch-2: Only gRPC Java 1.60+ and Protobuf 3.25+ dependencies
• Req-Arch-3: Log file in temp directory
• Req-Arch-4: Log rotation (100MB per file, 5 files max)
• Req-Arch-5: Application runs continuously unless unrecoverable error

---

3. Component Diagram - Hexagonal Architecture (C4 Level 3)

Purpose: Detailed view of internal components following hexagonal architecture pattern with ports and adapters.

Requirements Covered: All functional requirements (Req-FR-1 to Req-FR-32), Req-Arch-6, Req-Arch-7, Req-Arch-8

graph TB
    subgraph "PRIMARY ADAPTERS (Driving)"
        CONFIG_ADAPTER["ConfigurationLoader<br/><b>[Req-FR-2, Req-FR-9-13]</b><br/>Load & validate config"]
        HEALTH_ADAPTER["HealthCheckAdapter<br/><b>[Req-NFR-7, Req-NFR-8]</b><br/>HTTP :8080/health"]
        MAIN["HspApplication<br/><b>[Req-FR-1, Req-Arch-5]</b><br/>Startup orchestration"]
    end

    subgraph "PRIMARY PORTS (Inbound)"
        CONFIG_PORT["ConfigurationPort<br/><b>[Req-FR-10]</b>"]
        HEALTH_PORT["HealthStatusPort<br/><b>[Req-NFR-7]</b>"]
        LIFECYCLE_PORT["LifecyclePort<br/><b>[Req-FR-1]</b>"]
    end

    subgraph "CORE DOMAIN (Hexagon Center)"
        subgraph "Domain Models"
            DIAG_DATA["DiagnosticData<br/><b>[Req-FR-22, Req-FR-24]</b><br/>plugin_name, timestamp,<br/>source_endpoint, data_size"]
            CONFIGURATION["Configuration<br/><b>[Req-FR-9-13]</b><br/>Endpoints, intervals"]
            HEALTH_STATUS["HealthStatus<br/><b>[Req-NFR-8]</b><br/>service_status, counters"]
        end

        subgraph "Domain Services"
            VALIDATOR["ConfigurationValidator<br/><b>[Req-FR-11, Req-FR-12]</b><br/>Validation logic"]
            SERIALIZER["JsonDataSerializer<br/><b>[Req-FR-22, Req-FR-23]</b><br/>JSON + Base64"]
            DATA_VALIDATOR["DiagnosticDataValidator<br/><b>[Req-FR-21]</b><br/>Max 1MB check"]
            BUFFER["DataBuffer<br/><b>[Req-FR-26, Req-FR-27]</b><br/><b>[Req-Arch-7, Req-Arch-8]</b><br/>Thread-safe queue<br/>Max 300, FIFO"]
        end

        subgraph "Application Services"
            HTTP_POLLING["HttpPollingService<br/><b>[Req-FR-14-21]</b><br/><b>[Req-Arch-6]</b><br/>Virtual threads"]
            GRPC_TRANSMISSION["GrpcTransmissionService<br/><b>[Req-FR-28-33]</b><br/><b>[Req-Arch-6]</b><br/>Batch & stream"]
            COORDINATOR["DataFlowCoordinator<br/><b>[Req-Arch-7]</b><br/>Producer-Consumer"]
            HEALTH_MONITOR["HealthMonitoringService<br/><b>[Req-NFR-8]</b><br/>Metrics aggregation"]
        end
    end

    subgraph "SECONDARY PORTS (Outbound)"
        HTTP_CLIENT_PORT["DataCollectionPort<br/><b>[Req-FR-15]</b>"]
        GRPC_STREAM_PORT["DataTransmissionPort<br/><b>[Req-FR-28]</b>"]
        LOGGING_PORT["LoggingPort<br/><b>[Req-Arch-3]</b>"]
    end

    subgraph "SECONDARY ADAPTERS (Driven)"
        HTTP_ADAPTER["HttpClientAdapter<br/><b>[Req-FR-15-21]</b><br/>Timeout, retry, backoff"]
        RETRY_HANDLER["RetryHandler<br/><b>[Req-FR-17]</b><br/>3 retries, 5s interval"]
        BACKOFF["BackoffStrategy<br/><b>[Req-FR-18]</b><br/>Linear: 5s to 300s"]
        CONN_POOL["EndpointConnectionPool<br/><b>[Req-FR-19]</b><br/>No concurrent connections"]

        GRPC_ADAPTER["GrpcClientAdapter<br/><b>[Req-FR-28, Req-FR-33]</b><br/>receiver_id=99"]
        STREAM_MANAGER["StreamManager<br/><b>[Req-FR-29, Req-FR-30]</b><br/>Single stream, reconnect"]
        CONN_MANAGER["ConnectionManager<br/><b>[Req-FR-4, Req-FR-6]</b><br/>Retry every 5s"]

        LOG_ADAPTER["FileLoggerAdapter<br/><b>[Req-Arch-3, Req-Arch-4]</b><br/>temp/hsp.log"]
    end

    subgraph "External Systems"
        DEVICES["Endpoint Devices<br/><b>[IF1]</b>"]
        COLLECTOR["Collector Core<br/><b>[IF2]</b>"]
        LOG_FILE["hsp.log"]
    end

    %% Primary Adapter to Port connections
    CONFIG_ADAPTER --> CONFIG_PORT
    HEALTH_ADAPTER --> HEALTH_PORT
    MAIN --> LIFECYCLE_PORT

    %% Port to Domain connections
    CONFIG_PORT --> CONFIGURATION
    CONFIG_PORT --> VALIDATOR
    HEALTH_PORT --> HEALTH_MONITOR
    LIFECYCLE_PORT --> COORDINATOR

    %% Domain internal connections
    VALIDATOR --> CONFIGURATION
    HTTP_POLLING --> DATA_VALIDATOR
    HTTP_POLLING --> SERIALIZER
    HTTP_POLLING --> BUFFER
    GRPC_TRANSMISSION --> BUFFER
    COORDINATOR --> HTTP_POLLING
    COORDINATOR --> GRPC_TRANSMISSION
    HEALTH_MONITOR --> HEALTH_STATUS
    SERIALIZER --> DIAG_DATA

    %% Domain to Secondary Port connections
    HTTP_POLLING --> HTTP_CLIENT_PORT
    GRPC_TRANSMISSION --> GRPC_STREAM_PORT
    HTTP_POLLING --> LOGGING_PORT
    GRPC_TRANSMISSION --> LOGGING_PORT

    %% Secondary Port to Adapter connections
    HTTP_CLIENT_PORT --> HTTP_ADAPTER
    HTTP_ADAPTER --> RETRY_HANDLER
    HTTP_ADAPTER --> BACKOFF
    HTTP_ADAPTER --> CONN_POOL

    GRPC_STREAM_PORT --> GRPC_ADAPTER
    GRPC_ADAPTER --> STREAM_MANAGER
    GRPC_ADAPTER --> CONN_MANAGER

    LOGGING_PORT --> LOG_ADAPTER

    %% Adapter to External System connections
    HTTP_ADAPTER --> DEVICES
    GRPC_ADAPTER --> COLLECTOR
    LOG_ADAPTER --> LOG_FILE

    classDef primaryAdapter fill:#90EE90,stroke:#333,stroke-width:2px
    classDef primaryPort fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef domain fill:#FFD700,stroke:#333,stroke-width:3px
    classDef secondaryPort fill:#FFA07A,stroke:#333,stroke-width:2px
    classDef secondaryAdapter fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px

    class CONFIG_ADAPTER,HEALTH_ADAPTER,MAIN primaryAdapter
    class CONFIG_PORT,HEALTH_PORT,LIFECYCLE_PORT primaryPort
    class DIAG_DATA,CONFIGURATION,HEALTH_STATUS,VALIDATOR,SERIALIZER,DATA_VALIDATOR,BUFFER,HTTP_POLLING,GRPC_TRANSMISSION,COORDINATOR,HEALTH_MONITOR domain
    class HTTP_CLIENT_PORT,GRPC_STREAM_PORT,LOGGING_PORT secondaryPort
    class HTTP_ADAPTER,RETRY_HANDLER,BACKOFF,CONN_POOL,GRPC_ADAPTER,STREAM_MANAGER,CONN_MANAGER,LOG_ADAPTER secondaryAdapter
    class DEVICES,COLLECTOR,LOG_FILE external

Legend:

• Green (Primary Adapters): External actors driving the system (configuration, health checks, main app)
• Light Blue (Primary Ports): Inbound interfaces to core domain
• Gold (Core Domain): Business logic, domain models, application services - NO external dependencies
• Orange (Secondary Ports): Outbound interfaces for external systems
• Purple (Secondary Adapters): Implementations connecting to external systems
• Gray (External Systems): Third-party systems

Key Architectural Decisions:

• Req-Arch-6: Multi-threaded with virtual threads (Java 21+) for HTTP polling
• Req-Arch-7: Producer-Consumer pattern (HttpPollingService → DataBuffer → GrpcTransmissionService)
• Req-Arch-8: Thread-safe collections (ConcurrentLinkedQueue in DataBuffer)

---

4. Deployment Diagram

Purpose: Shows runtime deployment architecture, thread model, and resource allocation.

Requirements Covered: Req-Arch-5, Req-Arch-6, Req-NFR-1, Req-NFR-2

graph TB
    subgraph "Physical Server"
        subgraph "JVM Process [Req-Arch-1: Java 25]"
            subgraph "Thread Pool [Req-Arch-6: Virtual Threads]"
                VT1["Virtual Thread 1<br/>HTTP Polling<br/>Device 1-200"]
                VT2["Virtual Thread 2<br/>HTTP Polling<br/>Device 201-400"]
                VT3["Virtual Thread N<br/>HTTP Polling<br/>Device 801-1000<br/><b>[Req-NFR-1: 1000 devices]</b>"]
                VT_GRPC["Virtual Thread<br/>gRPC Consumer<br/><b>[Req-FR-28: Single stream]</b>"]
            end

            subgraph "Memory Regions [Req-NFR-2: Max 4096MB]"
                HEAP["Heap Memory<br/>• Configuration objects<br/>• DataBuffer (max 300)<br/>• HTTP connections"]
                BUFFER_MEM["DataBuffer<br/><b>[Req-FR-26, Req-FR-27]</b><br/>Max 300 items<br/>FIFO overflow"]
                COLLECTIONS["Thread-Safe Collections<br/><b>[Req-Arch-8]</b><br/>ConcurrentLinkedQueue"]
            end

            subgraph "Network I/O"
                HTTP_CONNS["HTTP Connections<br/><b>[Req-FR-19]</b><br/>No concurrent to<br/>same endpoint"]
                GRPC_STREAM["gRPC Stream<br/><b>[Req-FR-29]</b><br/>Single bidirectional"]
            end

            subgraph "File System"
                LOG_FILES["Log Files<br/><b>[Req-Arch-3, Req-Arch-4]</b><br/>temp/hsp.log<br/>100MB × 5 files"]
                CONFIG_FILE["config.json<br/><b>[Req-FR-10]</b><br/>Application directory"]
            end

            subgraph "Health Check Endpoint"
                HEALTH_SERVER["HTTP Server<br/><b>[Req-NFR-7]</b><br/>localhost:8080/health"]
            end
        end
    end

    VT1 --> BUFFER_MEM
    VT2 --> BUFFER_MEM
    VT3 --> BUFFER_MEM
    BUFFER_MEM --> VT_GRPC

    VT1 --> HTTP_CONNS
    VT2 --> HTTP_CONNS
    VT3 --> HTTP_CONNS
    VT_GRPC --> GRPC_STREAM

    VT1 --> LOG_FILES
    VT_GRPC --> LOG_FILES

    HEALTH_SERVER --> HEAP

    classDef thread fill:#90EE90,stroke:#333,stroke-width:2px
    classDef memory fill:#FFD700,stroke:#333,stroke-width:2px
    classDef network fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef file fill:#DDA0DD,stroke:#333,stroke-width:2px

    class VT1,VT2,VT3,VT_GRPC thread
    class HEAP,BUFFER_MEM,COLLECTIONS memory
    class HTTP_CONNS,GRPC_STREAM,HEALTH_SERVER network
    class LOG_FILES,CONFIG_FILE file

Runtime Architecture Notes:

1. Req-Arch-5: Application runs continuously, only exits on unrecoverable error
2. Req-Arch-6: Virtual threads enable 1000+ concurrent HTTP connections with minimal memory overhead
3. Req-NFR-1: Support 1000 concurrent HTTP endpoints using virtual thread pool
4. Req-NFR-2: Memory usage capped at 4096MB (JVM heap size limit)
5. Req-FR-19: Endpoint connection pool prevents concurrent connections to same endpoint

---

5. Sequence Diagrams

5.1 Startup Sequence

Purpose: Shows initialization sequence from application start to operational state.

Requirements Covered: Req-FR-1 to Req-FR-8

sequenceDiagram
    autonumber
    participant Main as HspApplication<br/>[Req-FR-1]
    participant ConfigLoader as ConfigurationLoader<br/>[Req-FR-2]
    participant Validator as ConfigurationValidator<br/>[Req-FR-11]
    participant Logger as FileLoggerAdapter<br/>[Req-FR-3]
    participant GrpcMgr as ConnectionManager<br/>[Req-FR-4]
    participant GrpcStream as GrpcClientAdapter<br/>[Req-FR-4]
    participant HttpPoller as HttpPollingService<br/>[Req-FR-5]
    participant Coordinator as DataFlowCoordinator<br/>[Req-FR-1]

    Note over Main: Startup Sequence<br/>[Req-FR-1 to Req-FR-8]

    Main->>ConfigLoader: loadConfiguration()
    Note right of ConfigLoader: [Req-FR-2]<br/>Read config file
    ConfigLoader-->>Main: Configuration

    Main->>Validator: validate(config)
    Note right of Validator: [Req-FR-11]<br/>Validate all parameters
    alt Invalid Configuration
        Validator-->>Main: ValidationError
        Note over Main: [Req-FR-12, Req-FR-13]<br/>Log error, exit code 1
        Main->>Logger: logError(reason)
        Main->>Main: System.exit(1)
    end
    Validator-->>Main: ValidationSuccess

    Main->>Logger: initialize(logPath)
    Note right of Logger: [Req-FR-3]<br/>temp/hsp.log<br/>[Req-Arch-3, Req-Arch-4]
    Logger-->>Main: LoggerInitialized

    Main->>GrpcMgr: connect()
    Note right of GrpcMgr: [Req-FR-4]
    loop Retry every 5s [Req-FR-6]
        GrpcMgr->>GrpcStream: establishStream()
        alt Connection Failed
            GrpcStream-->>GrpcMgr: ConnectionError
            Note over GrpcMgr: Wait 5s<br/>[Req-FR-6]
            GrpcMgr->>Logger: logWarning("gRPC retry")
        else Connection Success
            GrpcStream-->>GrpcMgr: StreamEstablished
            Note over GrpcMgr: [Req-FR-29]<br/>Single stream
        end
    end
    GrpcMgr-->>Main: Connected

    Main->>HttpPoller: startPolling()
    Note right of HttpPoller: [Req-FR-5, Req-FR-7]<br/>Only start after gRPC<br/>connected
    HttpPoller-->>Main: PollingStarted

    Main->>Coordinator: start()
    Note right of Coordinator: [Req-Arch-7]<br/>Producer-Consumer
    Coordinator-->>Main: Started

    Main->>Logger: logInfo("HSP started successfully")
    Note right of Logger: [Req-FR-8]<br/>Startup complete

Key Decision Points:

• Req-FR-7: HTTP polling does NOT start until gRPC connection is established
• Req-FR-6: gRPC connection retries every 5 seconds indefinitely
• Req-FR-12: Application terminates with exit code 1 on configuration validation failure

---

5.2 HTTP Polling Cycle

Purpose: Shows periodic HTTP data collection from endpoint devices.

Requirements Covered: Req-FR-14 to Req-FR-24

sequenceDiagram
    autonumber
    participant Timer as ScheduledExecutor<br/>[Req-FR-16]
    participant Poller as HttpPollingService<br/>[Req-FR-14]
    participant HttpClient as HttpClientAdapter<br/>[Req-FR-15]
    participant Retry as RetryHandler<br/>[Req-FR-17]
    participant Backoff as BackoffStrategy<br/>[Req-FR-18]
    participant Device as Endpoint Device<br/>[IF1]
    participant Validator as DiagnosticDataValidator<br/>[Req-FR-21]
    participant Serializer as JsonDataSerializer<br/>[Req-FR-22, Req-FR-23]
    participant Buffer as DataBuffer<br/>[Req-FR-26]

    Note over Timer: Polling Cycle<br/>[Req-FR-16: Configured interval]

    Timer->>Poller: trigger()
    Note right of Poller: Virtual thread<br/>[Req-Arch-6]

    Poller->>HttpClient: performGet(url)
    Note right of HttpClient: [Req-FR-15]<br/>30s timeout

    HttpClient->>Device: HTTP GET
    Note right of Device: [Req-FR-15]

    alt Timeout or Error
        Device-->>HttpClient: Timeout/Error
        HttpClient->>Retry: handleFailure()
        Note right of Retry: [Req-FR-17]<br/>Retry 3 times

        loop Up to 3 retries
            Retry->>Backoff: calculateDelay(attempt)
            Note right of Backoff: [Req-FR-18]<br/>Linear: 5s, 10s, 15s...
            Backoff-->>Retry: delayDuration
            Note over Retry: Wait delay
            Retry->>HttpClient: retryRequest()
            HttpClient->>Device: HTTP GET

            alt Retry Success
                Device-->>HttpClient: 200 OK + Data
                HttpClient-->>Retry: Success
                Retry-->>Poller: Data
            else Max Retries Reached
                Note over Retry: [Req-FR-18]<br/>Max backoff: 300s
                Retry-->>Poller: Failure
                Note over Poller: [Req-FR-20]<br/>Continue with other endpoints
            end
        end
    else Success
        Device-->>HttpClient: 200 OK + Data
        HttpClient-->>Poller: Data

        Poller->>Validator: validate(data)
        Note right of Validator: [Req-FR-21]<br/>Check size ≤ 1MB

        alt Data Too Large
            Validator-->>Poller: ValidationError
            Poller->>Poller: logWarning("File > 1MB")
            Note over Poller: [Req-FR-20]<br/>Continue with next endpoint
        else Valid Size
            Validator-->>Poller: Valid

            Poller->>Serializer: serialize(data)
            Note right of Serializer: [Req-FR-22, Req-FR-23, Req-FR-24]<br/>JSON with Base64<br/>+ metadata
            Serializer-->>Poller: DiagnosticData

            Poller->>Buffer: offer(diagnosticData)
            Note right of Buffer: [Req-FR-26]<br/>Thread-safe queue<br/>[Req-Arch-8]

            alt Buffer Full
                Buffer-->>Poller: BufferFull
                Note over Buffer: [Req-FR-27]<br/>Drop oldest (FIFO)
                Buffer->>Buffer: removeOldest()
                Buffer->>Buffer: add(data)
            else Buffer Space Available
                Buffer-->>Poller: Added
            end
        end
    end

    Note over Poller: [Req-FR-19]<br/>No concurrent connections<br/>to same endpoint

Key Behaviors:

• Req-FR-17: 3 retry attempts with 5-second intervals
• Req-FR-18: Linear backoff escalation (5s → 10s → 15s ... → 300s max)
• Req-FR-19: No concurrent connections to the same endpoint
• Req-FR-20: Failure on one endpoint does not stop polling of others
• Req-FR-21: Files > 1MB rejected with warning log
• Req-FR-27: Buffer overflow handled by dropping oldest data (FIFO)

---

5.3 gRPC Transmission

Purpose: Shows data transmission from buffer to Collector Core via gRPC.

Requirements Covered: Req-FR-28 to Req-FR-33

sequenceDiagram
    autonumber
    participant Consumer as GrpcTransmissionService<br/>[Req-FR-26]
    participant Buffer as DataBuffer<br/>[Req-FR-26]
    participant Batcher as MessageBatcher<br/>[Req-FR-31, Req-FR-32]
    participant Stream as GrpcClientAdapter<br/>[Req-FR-28]
    participant Manager as StreamManager<br/>[Req-FR-29, Req-FR-30]
    participant Collector as Collector Core<br/>[IF2]

    Note over Consumer: Consumer Thread<br/>[Req-Arch-6: Virtual thread]

    loop Continuous Consumption [Req-Arch-7]
        Consumer->>Buffer: poll()
        Note right of Buffer: [Req-FR-26]<br/>Thread-safe read

        alt Buffer Empty
            Buffer-->>Consumer: Empty
            Note over Consumer: Wait 10ms
        else Data Available
            Buffer-->>Consumer: DiagnosticData

            Consumer->>Batcher: add(data)
            Note right of Batcher: [Req-FR-31]<br/>Accumulate up to 4MB

            alt Batch Size ≥ 4MB
                Batcher-->>Consumer: BatchReady
                Note over Batcher: [Req-FR-31]<br/>Max 4MB reached
            else Timeout 1s Reached
                Note over Batcher: [Req-FR-32]<br/>Send after 1s
                Batcher-->>Consumer: BatchReady
            else Continue Accumulating
                Note over Batcher: Wait for more data
            end

            Consumer->>Stream: sendTransferRequest(batch)
            Note right of Stream: [Req-FR-33]<br/>receiver_id = 99

            Stream->>Manager: getStream()
            Manager-->>Stream: StreamHandle

            Stream->>Collector: TransferRequest
            Note right of Collector: [Req-FR-28]<br/>gRPC bidirectional

            alt Stream Failure
                Collector-->>Stream: Error
                Stream-->>Consumer: GrpcException

                Consumer->>Manager: reconnect()
                Note right of Manager: [Req-FR-30]<br/>Close, wait 5s, re-establish

                Manager->>Manager: closeStream()
                Note over Manager: Wait 5s
                Manager->>Stream: establishNewStream()

                alt Reconnect Success
                    Stream-->>Manager: StreamEstablished
                    Note over Manager: [Req-FR-29]<br/>Single stream only
                    Manager-->>Consumer: Ready
                    Note over Consumer: Retry sending batch
                else Reconnect Failure
                    Stream-->>Manager: Error
                    Manager->>Buffer: requeue(batch)
                    Note over Buffer: [Req-FR-26]<br/>Back to buffer
                end

            else Success
                Collector-->>Stream: TransferResponse
                Stream-->>Consumer: Success
                Note over Consumer: Continue consumption
            end
        end
    end

Key Behaviors:

• Req-FR-29: Only one bidirectional gRPC stream at a time
• Req-FR-30: On failure: close stream, wait 5s, re-establish
• Req-FR-31: Batch messages up to 4MB before sending
• Req-FR-32: Send batch within 1 second even if < 4MB
• Req-FR-33: All TransferRequests set receiver_id to 99

---

5.4 Error Handling and Retry

Purpose: Demonstrates error handling across HTTP and gRPC interfaces.

Requirements Covered: Req-FR-17, Req-FR-18, Req-FR-20, Req-FR-21, Req-FR-30, Req-Norm-3

sequenceDiagram
    autonumber
    participant Poller as HttpPollingService
    participant Device as Endpoint Device
    participant Logger as LoggingPort
    participant Buffer as DataBuffer
    participant GrpcService as GrpcTransmissionService
    participant GrpcStream as GrpcClientAdapter
    participant Collector as Collector Core

    Note over Poller,Collector: Error Handling Scenarios<br/>[Req-Norm-3]

    rect rgb(255, 220, 220)
        Note over Poller,Device: HTTP Timeout [Req-FR-17, Req-FR-18]
        Poller->>Device: HTTP GET (timeout 30s)
        Note over Device: No response
        Device-->>Poller: Timeout
        Poller->>Logger: logWarning("HTTP timeout")
        Poller->>Poller: retry (attempt 1/3)
        Note over Poller: Wait 5s [Req-FR-17]
        Poller->>Device: HTTP GET (retry 1)
        Device-->>Poller: Timeout
        Poller->>Poller: retry (attempt 2/3)
        Note over Poller: Wait 10s [Req-FR-18]<br/>Linear backoff
        Poller->>Device: HTTP GET (retry 2)
        Device-->>Poller: Timeout
        Poller->>Poller: retry (attempt 3/3)
        Note over Poller: Wait 15s
        Poller->>Device: HTTP GET (retry 3)
        Device-->>Poller: Timeout
        Note over Poller: [Req-FR-20]<br/>Continue with other endpoints
    end

    rect rgb(255, 240, 220)
        Note over Poller,Logger: Data Validation Error [Req-FR-21]
        Poller->>Device: HTTP GET
        Device-->>Poller: 200 OK (2MB data)
        Poller->>Poller: validate(data)
        Note over Poller: Size > 1MB
        Poller->>Logger: logWarning("File > 1MB rejected")
        Note over Poller: [Req-FR-20]<br/>Discard, continue
    end

    rect rgb(220, 240, 255)
        Note over Buffer,Collector: gRPC Stream Failure [Req-FR-30]
        GrpcService->>Buffer: poll()
        Buffer-->>GrpcService: DiagnosticData
        GrpcService->>GrpcStream: sendTransferRequest()
        GrpcStream->>Collector: TransferRequest
        Collector-->>GrpcStream: StreamError
        GrpcStream-->>GrpcService: GrpcException

        GrpcService->>Logger: logError("gRPC stream failed")
        GrpcService->>GrpcStream: closeStream()
        Note over GrpcStream: [Req-FR-30]<br/>Wait 5s
        GrpcService->>GrpcStream: reconnect()

        alt Reconnection Success
            GrpcStream->>Collector: EstablishStream
            Collector-->>GrpcStream: StreamEstablished
            GrpcStream-->>GrpcService: Ready
            GrpcService->>Buffer: requeue(data)
            Note over Buffer: [Req-FR-26]<br/>Data preserved
        else Reconnection Failure
            GrpcStream-->>GrpcService: Error
            GrpcService->>Buffer: requeue(data)
            Note over Buffer: Data stays in buffer<br/>Retry on next cycle
        end
    end

Error Handling Strategy:

• Req-FR-17: HTTP failures retry 3 times with 5-second intervals
• Req-FR-18: Linear backoff increases delay on repeated failures
• Req-FR-20: Failures isolated per endpoint, do not affect others
• Req-FR-21: Oversized data rejected immediately with warning
• Req-FR-30: gRPC failures trigger stream close and reconnection
• Req-Norm-3: All errors logged for diagnostics

---

6. Data Flow Diagram

Purpose: Shows data transformation from HTTP collection to gRPC transmission.

Requirements Covered: Req-Arch-7 (Producer-Consumer), Req-FR-22 to Req-FR-24 (Serialization)

graph LR
    subgraph "Producer [Req-Arch-7]"
        DEVICES["Endpoint Devices<br/>(1000 endpoints)<br/><b>[Req-NFR-1]</b>"]
        HTTP_POLL["HTTP Polling<br/>(Virtual Threads)<br/><b>[Req-Arch-6]</b>"]
        VALIDATOR["Data Validator<br/>(Max 1MB)<br/><b>[Req-FR-21]</b>"]

        DEVICES -->|"Binary Data"| HTTP_POLL
        HTTP_POLL -->|"Raw Bytes"| VALIDATOR
    end

    subgraph "Transformation [Req-FR-22-24]"
        BASE64["Base64 Encoder<br/><b>[Req-FR-23]</b><br/>Binary → Base64"]
        JSON_WRAP["JSON Wrapper<br/><b>[Req-FR-22, Req-FR-24]</b><br/>Add metadata"]

        VALIDATOR -->|"Valid Data"| BASE64
        BASE64 -->|"Base64 String"| JSON_WRAP
    end

    subgraph "Buffer [Req-FR-26-27]"
        BUFFER["DataBuffer<br/>(ConcurrentQueue)<br/><b>[Req-Arch-8]</b><br/>Max 300 items"]
        OVERFLOW["Overflow Handler<br/><b>[Req-FR-27]</b><br/>Drop oldest (FIFO)"]

        JSON_WRAP -->|"DiagnosticData"| BUFFER
        BUFFER -.->|"Full"| OVERFLOW
        OVERFLOW -.->|"Remove oldest"| BUFFER
    end

    subgraph "Consumer [Req-Arch-7]"
        BATCHER["Message Batcher<br/><b>[Req-FR-31, Req-FR-32]</b><br/>Max 4MB or 1s"]
        PROTO["Protobuf Serializer<br/><b>[Req-FR-28]</b><br/>TransferRequest"]
        GRPC_STREAM["gRPC Stream<br/><b>[Req-FR-29]</b><br/>Single bidirectional"]

        BUFFER -->|"Poll data"| BATCHER
        BATCHER -->|"Batch ready"| PROTO
        PROTO -->|"TransferRequest<br/>receiver_id=99<br/>[Req-FR-33]"| GRPC_STREAM
    end

    subgraph "Collector Core [IF2]"
        COLLECTOR["Collector Sender Core<br/><b>[Req-FR-28]</b>"]

        GRPC_STREAM -->|"gRPC Stream"| COLLECTOR
        COLLECTOR -.->|"TransferResponse"| GRPC_STREAM
    end

    subgraph "Logging [Req-Arch-3, Req-Arch-4]"
        LOG["File Logger<br/>temp/hsp.log<br/>100MB × 5 files"]

        HTTP_POLL -.->|"Errors"| LOG
        GRPC_STREAM -.->|"Errors"| LOG
        OVERFLOW -.->|"Dropped packets"| LOG
    end

    classDef producer fill:#90EE90,stroke:#333,stroke-width:2px
    classDef transform fill:#FFD700,stroke:#333,stroke-width:2px
    classDef buffer fill:#FFA07A,stroke:#333,stroke-width:3px
    classDef consumer fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef logging fill:#DDA0DD,stroke:#333,stroke-width:2px

    class DEVICES,HTTP_POLL,VALIDATOR producer
    class BASE64,JSON_WRAP transform
    class BUFFER,OVERFLOW buffer
    class BATCHER,PROTO,GRPC_STREAM consumer
    class COLLECTOR external
    class LOG logging

Data Flow Steps:

1. Collection (Producer):

   • Req-NFR-1: Poll 1000 endpoint devices concurrently
   • Req-Arch-6: Use virtual threads for efficient concurrency
   • Req-FR-21: Validate data size ≤ 1MB

2. Transformation:

   • Req-FR-23: Encode binary data as Base64
   • Req-FR-22: Wrap in JSON structure
   • Req-FR-24: Add metadata (plugin_name, timestamp, source_endpoint, data_size)

3. Buffering:

   • Req-FR-26: Store in thread-safe circular buffer (max 300 items)
   • Req-Arch-8: Use ConcurrentLinkedQueue
   • Req-FR-27: Drop oldest data when buffer full

4. Batching (Consumer):

   • Req-FR-31: Accumulate up to 4MB per batch
   • Req-FR-32: Send batch within 1 second even if < 4MB
   • Req-FR-33: Set receiver_id = 99

5. Transmission:

   • Req-FR-28: Send via gRPC TransferService
   • Req-FR-29: Use single bidirectional stream
   • Req-FR-30: Reconnect on failure (close, wait 5s, re-establish)

---

Requirement Coverage Summary

Diagram	Requirements Covered	Count
System Context	Req-Arch-1, Req-Arch-2, Req-FR-14-28, Req-NFR-7	18
Container	Req-Arch-1-5, Req-NFR-5-6, Req-FR-9-13	13
Component (Hexagonal)	Req-FR-1-33, Req-Arch-6-8, Req-NFR-7-8	43
Deployment	Req-Arch-5-6, Req-NFR-1-2, Req-FR-19, Req-FR-26-29	9
Sequence: Startup	Req-FR-1-8	8
Sequence: HTTP Polling	Req-FR-14-24	11
Sequence: gRPC Transmission	Req-FR-26-33	8
Sequence: Error Handling	Req-FR-17-18, Req-FR-20-21, Req-FR-30, Req-Norm-3	6
Data Flow	Req-Arch-6-8, Req-FR-21-33, Req-NFR-1	18
Total Unique Requirements	-	62

---

Architectural Decision Records

ADR-1: Hexagonal Architecture

Decision: Use hexagonal (ports and adapters) architecture pattern.

Rationale:

• Req-Norm-6: Maintainable design with clear separation of concerns
• Req-Norm-4: Easier testing with mockable ports
• Req-Arch-2: Core domain isolated from external libraries (gRPC, Protobuf)

Consequences:

• More interfaces and classes
• Clear dependency boundaries
• Testable without external systems

ADR-2: Virtual Threads for Concurrency

Decision: Use Java 21+ virtual threads (Project Loom) for HTTP polling.

Rationale:

• Req-NFR-1: Support 1000 concurrent HTTP connections
• Req-Arch-6: Multi-threaded architecture
• Req-NFR-2: Memory efficiency (virtual threads use <1KB each vs. 1MB for platform threads)

Consequences:

• Requires Java 21+
• Simplified concurrency model
• Excellent scalability

ADR-3: Producer-Consumer with Thread-Safe Queue

Decision: Use producer-consumer pattern with ConcurrentLinkedQueue.

Rationale:

• Req-Arch-7: Explicit producer-consumer requirement
• Req-Arch-8: Thread-safe collections required
• Req-FR-26-27: Buffering with overflow handling

Consequences:

• Lock-free performance
• Natural decoupling of HTTP and gRPC
• Clear component responsibilities

ADR-4: Single gRPC Stream

Decision: Maintain exactly one bidirectional gRPC stream.

Rationale:

• Req-FR-29: Explicit single stream requirement
• Req-FR-30: Simplified reconnection logic
• Req-FR-31-32: Batching optimizes single stream throughput

Consequences:

• No stream multiplexing complexity
• Clear stream lifecycle management
• Potential bottleneck if batching insufficient (mitigated by 4MB + 1s batching)

---

Next Steps

1. Architecture Review: Present diagrams to stakeholders for approval
2. Detailed Design: Expand component interfaces and contracts
3. Protobuf Schema: Define TransferService .proto file
4. Test Plan: Create test scenarios based on sequence diagrams
5. Implementation: Begin TDD with unit tests for domain models

---

Document Status: ✅ Complete Reviewers: Architecture Team, System Engineers Approval: Pending

---

Appendix: Mermaid Diagram Legend

• C4 Diagrams: System context, containers, components
• Sequence Diagrams: Temporal interactions with requirement annotations
• Deployment Diagrams: Runtime architecture with thread pools and memory regions
• Data Flow Diagrams: Producer-consumer pipeline with data transformations

All diagrams include bold requirement IDs (e.g., [Req-FR-15]) for traceability.