hackathon/requirements/DataCollector SRS.md

Software Requirement Specification Collector HTTP Plugin

Introduction

Purpose

This document shall describes the Collector sender plugin responsible for collecting endpoint device diagnostic data via HTTP. This document contains a description and all relevant requirements towards functionality and architecture.

System Description

The "Collector HTTP Plugin" (CHP) is based on the Collector main product. The main task of the Collector is to connect multiple network zones in a safe and secure way so that data can only be transferred in a unidirectional way. The Collector itself has been fully developed and is ready for use.

This project is about the development of the "Collector HTTP Plugin" (CHP) extension for the Collector main application. The CHP is desinged to be an additional software component to the Collector application running as a separate service.

The CHP is composed of two parts. The "HTTP Sender Plugin" (HSP) and the "HTTP Receiver Plugin" (HRP). As a first stage only that HSP shall be developed. The HRP will be part of a separate development.

The main task of the HSP is to connect the Collector to a large amount of endpoint devices and cyclically collect diagnostic data via HTTP. In a later stage the HRP will be responsible to forward the collected data to the receiver destination.

Requirements

HSP Architecture requirements

Unique ID	Requirement Description
Prose	The requirements specify the architectural framework and assumptions
Req-Arch-1	The HSP shall be developed in OpenJDK 25, featuring Java 25.
Req-Arch-2	HSP shall use only the following external libraries: gRPC Java 1.60+, Protocol Buffers 3.25+, and their transitive dependencies. All other functionality shall use only OpenJDK 25 standard library classes.
Req-Arch-3	HSP shall log all log messages and errors to the file "hsp.log" in a temp directory.
Req-Arch-4	For logging the Java Logging API with log rotation (max 100MB per file, 5 files) shall be used.
Req-Arch-5	HSP shall always run and not terminate unless an unrecoverable error occurs.
Req-Arch-6	HSP shall use a multi-threaded architecture with separate threads for HTTP polling and gRPC transmission. For the HTTP polling virtual threads shall be used to lower the resource demand.
Req-Arch-7	HSP shall implement the Producer-Consumer pattern for data flow between IF1 and IF2.
Req-Arch-8	HSP shall use thread-safe collections for data buffering.

HSP Functional requirements

Unique ID	Requirement Description
Prose	This section describes the initialization of the HSP
Req-FR-1	HSP shall execute the following startup sequence:
Req-FR-2	Startup step 1: Load and validate configuration as described below
Req-FR-3	Startup step 2: Initialize logging
Req-FR-4	Startup step 3: Establish gRPC connection to the Collector Sender Core as described below
Req-FR-5	Startup step 4: Begin HTTP polling of diagnostic data from the endpoint devices
Req-FR-6	If gRPC connection fails, HSP shall retry every 5 seconds indefinitely and log warnings every 1 minute.
Req-FR-7	HSP shall not begin HTTP polling until the gRPC connection is successfully established.
Req-FR-8	HSP shall log "HSP started successfully" at INFO level when all initialization steps complete.
Prose	This section describes the configuration of the HSP
Req-FR-9	The HSP shall be configurable via a configuration file as described in the HSP Configuration File Specfication.
Req-FR-10	At startup HSP shall read in the configuration file located in the application directory and apply the configuration.
Prose	The configuration file contains: HTTP endpoint URLs, polling interval, gRPC server address/port, timeout values, and retry policies.
Req-FR-11	HSP shall validate all configuration parameters to be within given limits
Req-FR-12	If validation fails HSP shall terminate with error code 1.
Req-FR-13	If validation fails HSP shall log the reason for the failed validation
Prose	This section describes the connection and mechanism used for the HSP to connect to the endpoint devices to poll diagnostic data
Req-FR-14	The HSP shall establish a connection to all configured devices according to interface IF1
Req-FR-15	HSP shall set a timeout of 30 seconds for each HTTP GET request.
Req-FR-16	HSP shall poll each configured HTTP endpoint device at intervals specified in the configuration
Req-FR-17	If the HTTP GET request fails HSP shall retry up to 3 times with 5-second intervals before marking the connection to the endpoint device as failed.
Req-FR-18	HSP shall implement linear backoff for failed endpoint connections. Starting at 5s to a maximum of 300s, adding 5s in every attempt.
Req-FR-19	HSP shall not have concurrent connections to the same endpoint device.
Req-FR-20	HSP shall continue polling other endpoint devices if one endpoint device fails.
Prose	The endpoint devices will answer the polling by transmitting a binary file.
Req-FR-21	HSP shall reject binary files larger than 1MB and shall log a warning.
Req-FR-22	HSP shall wrap the collected data in JSON as data serialization format.
Req-FR-23	HSP shall encode binary file as Base64 strings within the JSON payload.
Req-FR-24	Each JSON message shall include: "HTTP sender plugin" as the plugin name, timestamp (ISO 8601), source_endpoint (URL), data_size (bytes), and payload (Base64 encoded binary).
Req-FR-25	HSP shall then send the collected and aggregated data to the CollectorSender Core as decribed below.
Req-FR-26	If gRPC transmission fails, HSP shall buffer collected data in memory (max 300 messages).
Req-FR-27	If the buffer is full and new data is collected, HSP shall discard the oldest data.
Prose	This section describes the connection and mechanism used for the HSP to connect to the Collector Sender Core to transmit aggregated collected data
Req-FR-28	The HSP shall communicate with the Collector Sender Core according to Interface IF2
Req-FR-29	HSP shall automatically establish a single bidirectional gRPC stream to the Collector Sender Core at startup and maintain it for the lifetime of the application.
Req-FR-30	If the gRPC stream fails, HSP shall close the stream, wait 5 seconds, and try to establish a new stream.
Req-FR-31	HSP shall send one TransferRequest message containing as many messages as fit into 4MB (transfer maximum)
Req-FR-32	HSP shall send one TransferRequest message containing less then 4MB (transfer maximum) latest 1s after the last message.
Req-FR-33	The receiver_id field shall be set to 99 for all requests.

Non-Functional Requirements

Performance

Unique ID	Requirement Description
Req-NFR-1	HSP shall support concurrent polling of up to 1000 HTTP endpoint devices.
Req-NFR-2	HSP shall not exceed 4096MB of RAM usage under normal operation.

Security

Unique ID	Requirement Description
Req-NFR-3	HSP shall not use HTTP authentication.
Req-NFR-4	HSP shall use TCP mode only for the gRPC interface.

Usability

Unique ID	Requirement Description
Req-NFR-5	HSP shall be built using Maven 3.9+ with a provided pom.xml.
Req-NFR-6	HSP shall be packaged as an executable JAR with all dependencies included (fat JAR).

Reliability

Unique ID	Requirement Description
Req-NFR-7	HSP shall expose a health check HTTP endpoint on localhost:8080/health returning JSON status.
Req-NFR-8	Health check shall include: service status, last successful collection timestamp, gRPC connection status, error count of HTTP collection attempts, number of successfully collected HTTP endpoints last 30s, number of failed HTTP endpoints last 30s.

Normative Requirements

Unique ID	Requirement Description
Req-Norm-1	The software shall be developed in accordance with ISO-9001
Req-Norm-2	The software shall be developed in accordance with Cenelec EN 50716 Basic Entegrity
Req-Norm-3	The software shall implement measures for error detection and handling, including but not limited to, invalid sensor data, communication timeouts, and internal faults.
Req-Norm-4	The software shall be subjected to rigorous testing, including unit testing, integration testing, and validation testing, to ensure that it meets the specified safety requirements.
Req-Norm-5	The software development process shall be documented, including requirements, design, implementation, and testing, to provide a clear audit trail.
Req-Norm-6	The software shall be designed to be maintainable, with clear and concise code, and a modular architecture.

Testing Requirements

Unique ID	Requirement Description
Req-Test-1	Integration tests shall verify HTTP collection with a mock HTTP server.
Req-Test-2	Integration tests shall verify gRPC transmission with a mock gRPC server.
Req-Test-3	Tests shall use JUnit 5 and Mockito frameworks.
Req-Test-4	All tests shall be executable via 'mvn test' command.

User Stories

Unique ID	Requirement Description
Req-US-1	As a system operator, I want HSP to automatically collect diagnostic data from configured HTTP endpoints every second, so that real-time device health can be monitored without manual intervention.
Req-US-2	As a data analyst, I want all collected diagnostic data to be reliably transmitted to the Collector Sender Core via gRPC, so that I can analyze device behavior even if temporary network issues occur.
Req-US-3	As a system administrator, I want to check HSP health status via HTTP endpoint, so that I can monitor the service without accessing logs.

Assumptions and Dependencies

The Collector Core Sender gRPC server is always available.

Normative Requirements

TBD

List any other requirements not covered above

TBD

Glossary

• HSP: HTTP Sender Plugin
• HRP: HTTP Receiver Plugin
• Diagnostic Data: Binary files containing specific application dependent content
• IF1: HTTP-based interface for data collection
• IF2: gRPC-based interface for data transmission
• Endpoint device: A endpoint device is a device in a network providing diagnostic data via HTTP