Configurable Reporting System

Customer

An industrial machinery business that engineers energy-efficient heating and cooling products and solutions for buildings, industry, infrastructure, and the food cold chain, with about 11,300 employees worldwide and €3.1bn in net sales.

Problem

The old reporting system was not configurable and not scalable. It was a single service built for one specific tenant, and it supported only about three reports. Any new report or change had to be done by developers.

The system also had to be reliable. Customers receive many important reports every day, for example HACCP reports for stores, which matter for industry food safety standards and must be kept for 3 years. Failures could not go unnoticed.

The goal was a flexible and reliable system where users from any tenant can create their own reports from the UI and manage them, instead of developers doing it.

Architecture

The system is event-driven and split into two flows. In the generation flow, users manage report schedules in the platform UI through a configuration service. A scheduler service reads the schedules and publishes events to Kafka. The report engine consumes them and creates report tasks in the database. A Debezium CDC connector captures changes on the tasks and publishes them to Kafka, so the engine picks up each task from the topic. For each task it gets report data from a data API, renders the file with Carbone IO using a template from Azure Blob Storage, and stores the finished report back in storage.

Enlarge

In the delivery flow, the report engine saves its result and creates a delivery task in the same database. Debezium publishes it to Kafka, where the dispatcher service consumes it, reads the channel settings and the finished report, delivers it through SFTP, email or other channels, and saves the delivery result back to the task. Because every step is a persisted task, failures are visible and can be monitored and retried.

Enlarge

My contribution

• Designed the architecture of the reporting system.
• Built the report engine and integrated it with Carbone IO for rendering.
• Created the report templates for Carbone IO.
• Built the dispatcher service and integrated it with delivery providers such as SendGrid.
• Set up monitoring, dashboards and tracing for the whole system.
• Own the system end to end, including full maintenance.

Outcomes

• Users from any tenant create and manage their own reports from the UI, with no developer involvement.
• A scalable system that replaced a single-tenant service limited to about three reports.
• Fault tolerance and extensive monitoring and alerting, so failures in report generation or delivery are caught early.
• Reports reach customers through the channel they choose (SFTP, email or others), so they can keep them as long as their standards require, for example 3 years for HACCP reports.