What pain does the project solve?
Ever since I took over the development area at TCS Terminal de Cargas de Sarzedo, we had been facing almost “scheduled” problems in the invoice integration system with clients.
At least once a month, we had a severe instability or a total outage in the system. This directly impacted the terminal’s operation, taking hours, or even days, to restore normal functioning.
Initial proposal
Faced with this scenario, I began architecting a solution. At first, I tried to keep the existing system’s foundation, but separating the three critical parts:
- Ingestion: Reading the email inbox and processing XML attachments;
- Processing: Handling and formatting the data for the internal operational system;
- Integration: Active communication with the clients’ systems.
However, this quickly proved unfeasible. The legacy architecture couldn’t support that separation efficiently. Insisting on a monolith, with several other functions beyond integration, for this kind of processing would be a mistake; creating an “arm” of the monolith for each new client would turn maintenance into a nightmare of cyclopean complexity.
The Turning Point
On August 28, 2025, the definitive breakdown happened. The system stopped completely. At that moment, we still didn’t have the solution ready, although I had already been mentally preparing for that day. After a few tense hours, I managed to get the system back up in a precarious way, but enough to put out the immediate fire.
With the solution already envisioned, I started the work. On September 2, 2025, the board gathered the department and gave us an ultimatum: we needed to solve the problem at its root.
With the full support of the IT department and the Operations Manager, I stopped work on all other projects, froze the backlog, and dove exclusively into building the definitive solution.
Architecture
I opted for a microservices-based architecture using NestJS Microservices. The goal was to have extremely well-defined responsibilities, eliminating coupling.
For asynchronous communication and resilience, I adopted RabbitMQ. This ensures that if one end of the system goes down, messages stay queued and nothing is lost.
The main architecture was split into two cores:
- Nfe Service: Email reading and raw XML processing;
- Operacional Service: Formatting and sending to the internal system.
In addition, we adopted an isolation strategy for clients: each client has its own dedicated microservice.
To streamline development, I created base microservices (e.g., XMINING-basic), containing the essential code for repetitive integrations, allowing quick reuse for clients using the same ERP.
Email Reading and Processing (Nfe Service)
This microservice monitors the Outlook inbox in real time. It automatically manages the folders, processes attachments asynchronously, and ensures the flow never stalls, regardless of the incoming volume.
We also implemented an intelligence layer: automatic identification of invalid emails or clients, with automatic classification and rejection. This data is stored for future Machine Learning model training. The goal is to shield the system: it doesn’t matter if the layout of the government’s XML changes or if garbage comes in, the system will know how to handle it.
Processing for Operations (Operacional Service)
This is the heart of data translation. It receives inputs from both the client microservices and the Nfe Service and converts everything into the format of OpenGTM (a third-party system used by operations).
At first glance it seems simple, but the complexity is high. OpenGTM has a rigid structure based on SOAP, but done via JSON requests with extremely specific formatting. A single misplaced comma or data type results in rejection.
Unfortunately, for compliance reasons, I can’t share the code, but it was a very interesting engineering and data-structuring challenge to solve.
Client Microservices
The philosophy here is fault isolation. As long as the client’s system has an endpoint that returns the data TCS needs, it gets its own microservice.
This ensures that if “Client A’s” system goes down, “Client B” and the rest of the Integrator keep operating normally.
These services rely on standardized basic primitives:
- Send queue and retries;
- CRUD of integration data;
- Status monitoring.
When a new client comes in, you just “clone” the basic primitives’ structure and adjust the specific business rule.
Monitoring System
Getting out of the dark was a priority. The system has a monitoring microservice that dispatches alerts based on scope:
- Code/infra errors: Notify me and the IT team directly.
- Business errors: Notify the operators in real time (e.g., an invoice with no linked vehicle).
Prometheus and Grafana
We implemented detailed metrics for observability:
- Overall health of the Integrator flow;
- Resource consumption (CPU/RAM) per container;
- Success/failure statistics on the integration with OpenGTM;
- Throughput of each microservice and client.
Frontend
The visual interface was split into two strategic fronts:
1. Module in the SIG (Integrated Management System)
The SIG is our corporate “super-app” that unified several legacy subsystems. (Deserving of its own dedicated post soon.) The Integrator got its own module inside it with two main screens:
Screen 1: Integration Monitor Used by operators (weighbridge staff) to follow the flow in real time: barcode, date, status and friendly error logs.
Screen 2: Invalid Emails Lets operations see what was automatically rejected and why.
2. Client Portal
A separate project (still under construction) where clients themselves will be able to follow their integration processes, reducing the support load on our team.
Deployment and DevOps
The official switch happened on September 12, 2025. We retired the old system and migrated 100% of the flow.
To manage this complexity, I adopted a Monorepo strategy. Our CI is smart: it detects which folders/services changed and runs builds and tests only on the affected parts, drastically optimizing deploy time. Following industry standards.
Benefits Achieved
The architecture change brought tangible gains:
- Fault Isolation: An error in one module doesn’t bring down the entire system.
- Cost Reduction: Lower maintenance complexity and faster fixes mean fewer man-hours spent “firefighting” and less downtime.
- High Performance: Asynchronous processing allows handling load peaks without bottlenecks.
- Horizontal Scalability: What was once static now scales dynamically as demand requires.
- Observability: Before, we were flying blind; now we have a complete cockpit with metrics and dashboards.
Personal Growth
Working so intensely on this project over the past 3 months was a technical deep dive. I got to put into practice advanced concepts of distributed systems and software architecture that I had been studying, going from theory to the “trenches” of production. Especially because I had to put such a complex, distributed system into production without being able to exhaustively test every part of it.
Conclusion
Today, we complete exactly 3 months since the “turning point”. The system is stable, having gone through only two planned interruptions for scheduled maintenance.
Problems like “Error in the integration with Client X” still happen, but they are now handled, isolated and expected exceptions within the architecture, without stopping the company’s operation.
It was, without a doubt, one of the most challenging and rewarding experiences of my career so far. I’m immensely grateful to the IT department for the coverage and support during this period of total focus.
The project isn’t over. On the contrary, it’s now ready to evolve.