↓Skip to main content

Devops

From Monoliths to AI Proxies: Real-World Strategy for Testing and Evolving LLM Integrations

Ai Software Architecture Application Development Devops Software Architecture LLM Proxy Pattern AI Architecture Best-Practices Large Language Models LLM Integration Model Context Protocol (MCP) Testing Testing AI Systems
Integrating Large Language Models (LLMs) into production systems presents unique architectural, testing, and operational challenges. This article shares practical insights and solutions from real-world experience integrating LLMs into a customer interaction platform. It covers the evolution from a monolithic to a more modular AI Proxy architecture pattern, strategies for testing, deploying and monitoring LLMs, and the emerging Model Context Protocol (MCP) standard. Application developers and software architects will learn proven practices to build robust, reliable and responsible LLM-powered systems.

Monitoring your application locally with NewReclic

Programming Devops Java Performance Newrelic

The New Relic Digital Intelligence Platform provides actionable insights to drive digital business results. You can monitor your application and infrastructure performance so you can quickly resolve issues and improve digital customer experiences.

Following instruction should help you to connect your application to NewRelic platform and customize application events sent to the platform.

External Site Monitoring

Devops Web

How did you know that your site is down? From your customers? Then it’s time to start using some external monitoring tool.

Logging Policy

Programming Devops Logging

There are different points of view on how logging levels should be used in code. I will share mine.

My assumption is: “There should be no errors in logs when everything is fine.”

Developing in "Dirty Trunk"

Devops

I’m going to start a series of posts covering different aspects of DevOps.

Let’s start today with branching strategy called “dirty trunk”. Actually, this is an attempt to avoid branching at all.

Jolokia: HTTP/JSON bridge for JMX

Devops Jmx Tomcat Java

Very often there is a need to monitor the Java application server. For example, external monitoring tool, like Nagious/Zenoss/Zabbix needs to get some metrics, like heap memory usage or thread count.

Usual way to get that metrics is to setup access to application server via JMX.

But, sometimes, it is not possible to leave some other port opened for JMX and the only port available is HTTP(80 or 8080) or HTTPS(443 or 8443).

Here the Jolokia comes to rescue!