From Race Conditions to Reliable Messaging: What I Learned Building an Event-Driven System

When building software, some problems seem simple until they become complicated. Generating user IDs in a clean, predictable format, such as "abc-002," is one such challenge. At first glance, the solution seems straightforward: check the last ID in the database and increment it for the next user. This approach is practical, especially in an event-driven application.

However, that raised a question: what happens if multiple users access the same endpoint simultaneously?

The Race Condition Problem

In this scenario, known as concurrency, race conditions can occur. For instance, how can the application ensure that User A receives the ID "abc-003" if they were the first to request it?

To tackle this issue, I consulted AI for a solution. It recommended a messaging service, and because my application is written in C#, Azure Service Bus was suggested. After switching my code editor to agent mode, I implemented the fix, and it worked well.

However, I realised I didn't fully understand the system behaviours behind the solution, such as race conditions and idempotency, which are crucial for reliable messaging systems.

Although the solution was effective, I needed to understand the underlying concepts to grow as a software engineer.

The Modern System Reality

Modern systems are typically event-driven, asynchronous, and inherently unreliable. As a result, code execution does not follow a linear path. When an event triggers a message, it might:

Arrive late
Be duplicated
Come in out of order
Not arrive at all

To understand messaging effectively, familiarise yourself with key concepts such as queues and topics:

Queue: One-to-one communication using First In, First Out (FIFO) message delivery, where messages are processed in the order they were added. This pattern is especially beneficial for background tasks.
Topic: One-to-many communication, allowing multiple consumers to receive the same message (e.g., a welcome email).

Knowing which messaging pattern to use is essential because it can significantly affect application performance and user experience.

1. Ensuring Idempotency in a Messaging System (Practical Playbook)

Idempotency means that processing the same message more than once does not change the result. With that in mind, the following practices help make it reliable.

Core Best Practices

1. Use a Unique, Deterministic Message ID

Every message must have a stable identifier:

event_id
message_id
order_id + version
aggregate_id + sequence_number

This ID must be:

Generated once
Propagated across retries
Persisted

Never generate a new ID on retry.

2. Store Processed Message IDs (Inbox Table)

Before handling a message:

Check if message_id exists.
If yes → skip processing.
If no → process and store the ID.

Typical schema:

inbox (
  message_id UUID PRIMARY KEY,
  processed_at TIMESTAMP
)

This is the Inbox Pattern in its simplest form.

3. Make Handlers Idempotent by Design

Your business logic should tolerate replays.

Bad:

INSERT INTO users (email) VALUES ('a@b.com')

Good:

INSERT INTO users (email)
VALUES ('a@b.com')
ON CONFLICT (email) DO NOTHING

Other strategies:

Use UPSERT
Version checks (WHERE version = expected)
State transitions instead of actions

4. Use Database Constraints as Safety Nets

Let the database protect you:

Unique constraints
Foreign keys
Conditional updates

Example:

UPDATE orders
SET status = 'PAID'
WHERE id = :id AND status = 'PENDING'

If it runs twice, the second does nothing.

5. Treat Retries as the Default, Not the Exception

Design assuming:

Network failures
Consumer crashes
Broker redeliveries

If your handler breaks on retries → it's not production-ready.

2. When to Implement the Outbox and Inbox Patterns

Outbox Pattern (Producer Side)

Problem it solves: You update the database but fail to publish the message (or vice versa).

When to use it:

Write to a DB AND
Publish an event/message
Need guaranteed consistency

Classic failure scenario:

DB commit ✅
Message publish ❌
→ Downstream systems never see the change

Outbox solution:

Write business data
Write the event to the outbox table
Commit transaction
Background worker publishes events
Mark them as sent

Schema:

outbox (
  id UUID,
  aggregate_id,
  event_type,
  payload,
  published BOOLEAN
)

Rule of thumb: If state change and event publish must succeed together → use Outbox.

Once that is clear, consider the consumer side.

Inbox Pattern (Consumer Side)

Problem it solves: The message is delivered more than once.

When to use it:

Consume messages from a broker
Use at-least-once delivery
Cannot tolerate duplicate side effects

Rule of thumb: If duplicates would cause real damage → use Inbox.

When to Use BOTH

Use Outbox + Inbox when:

You have distributed services
Events drive state changes
You care about correctness over simplicity

This combo is ubiquitous in:

Financial systems
Order processing
Event-driven microservices

3. Delivery Semantics: Real-World Implications

Let's cut through the marketing.

At-Most-Once

The message is delivered 0 or 1 time.

Characteristics:

No retries
Fast
Simple

Trade-offs:

Messages can be lost
Not reliable

Use when: Logging, metrics, non-critical notifications. If it fails, you don't care.

At-Least-Once (Most Common)

The message is delivered 1 or more times.

Characteristics:

Retries enabled
Reliable
Duplicates possible

Trade-offs:

Requires idempotency
More engineering effort

Use when: Payments, orders, emails, user actions. This is the default for Kafka, RabbitMQ, and SQS.

Exactly-Once (The Myth)

The message is processed once and only once.

Reality check:

Usually not truly end-to-end
Achieved via:
- Transactions
- Deduplication
- Idempotent consumers

Kafka's "exactly-once" = effectively-once within Kafka + consumer group, not your DB.

Trade-offs:

Complex
Slower
Hard to debug

Use when: Financial ledgers, critical accounting flows.

Most "exactly-once" systems are really at-least-once + idempotency.

How to Choose in Practice

Scenario	Recommendation
Non-critical data	At-most-once
Business workflows	At-least-once + idempotency
Financial correctness	At-least-once + Outbox + Inbox

Marketing "exactly-once"? Verify what it really means.

Idempotency vs. Deduplication

When designing an event-driven application, you'll encounter terms like "idempotency" and "deduplication." To make the difference clear, start with idempotency:

Idempotency: The ability to run the same operation multiple times while yielding the same result.
Deduplication: Detecting and managing duplicate operations.

Standard techniques for implementing these concepts include:

Using message IDs
Time-window-based deduplication

The key distinction is that idempotency ensures safe behaviour, while deduplication serves as a detection mechanism.

My Application and the Three Delivery Modes

The application I was developing is an in-process domain event publisher that provides immediate consistency but does not guarantee delivery across service boundaries. That context shaped how I looked at the three delivery modes.

I researched three delivery modes for messaging systems:

"At-most-once" — The fastest but risks losing messages.
"At-least-once" — Ensures messages are never lost but requires idempotency to handle potential duplicates.
"Exactly-once" — Efficient, but it is the most complex and expensive.

Deciding which model to use involves weighing the trade-offs between simplicity and correctness.

Do You Need Outbox and Inbox Patterns?

When I first learned about this, I questioned whether my application required these patterns. The Outbox and Inbox patterns can be complex and may be overkill for some applications. They are worth considering when the trade-offs demand it, but it is just as important to recognise when they are unnecessary to keep your system simple and maintainable. For example, if your application is entirely monolithic, runs on a single process, does not require distributed messaging, and side effects can safely be retried without harm, adding these patterns may introduce needless complexity.

A basic checklist to help decide if Outbox/Inbox patterns are unnecessary:

Your system does not publish or consume external messages or events.
All operations occur within a single data store and process.
Occasional duplicate handling or message loss is acceptable for your use case.
You do not require strict audit trails or delivery guarantees.

If most of these apply, you can likely avoid these patterns and opt for simpler solutions.

However, consider implementing it if your application:

Has distributed messaging needs
Uses long-running operations
Requires idempotency guarantees for background job processing
Needs to meet audit compliance by ensuring that every event is persisted before publishing

The Real Lesson: Curiosity Over Convenience

This experience reminded me that modern software systems are rarely linear or straightforward. Instead, they are asynchronous, event-driven, and full of trade-offs. Concepts like idempotency, delivery guarantees, and messaging patterns aren't just academic; they directly influence an application's reliability and scalability.

AI Tools: Productivity vs Understanding

AI tools can dramatically improve productivity. In my case, they helped me quickly resolve a real production problem. However, they can also hide essential complexity behind working solutions, which makes understanding the result just as important as obtaining it.

If we don't take the time to understand why something works, we risk turning powerful tools into black boxes.

Transform Working Solutions into Knowledge

The real lesson wasn't about Azure Service Bus or user ID generation; it was about curiosity.

Using AI improved my productivity and helped me unblock a real-world problem. However, I made sure not to accept the AI-generated solution blindly. Before integrating the suggested changes, I thoroughly reviewed the code, wrote targeted unit tests, and validated the new approach in a development environment to check for unexpected side effects and ensure system stability. In doing so, I reinforced an important lesson: AI can provide solutions, but understanding and carefully verifying them remains the engineer's responsibility.

Concepts like messaging patterns, idempotency, delivery guarantees, and concurrency are not just academic terms; they define how systems behave under load, failure, and scale. Without understanding them, it's easy to ship something that works today but fails unpredictably tomorrow.

What changed for me was not just the implementation, but my mindset. I now approach AI-generated solutions with curiosity rather than blind trust. When I encounter a term I don't understand, I treat it as a signal to dig deeper. That habit of questioning why something works is what turns fixes into learning and tools into mastery.

In modern software engineering, reliability does not come from writing perfect code. It comes from designing systems that assume failure, embrace concurrency, and handle repetition safely. Understanding these principles is what allows us to build systems we can confidently scale, evolve, and depend on.

The solution worked. Understanding why it worked is what made it valuable.

The goal isn't just to build systems that work. It's to build systems you understand.

If you want to put these ideas into practice, here's a simple exercise: review a recent project or piece of code you have worked on and check whether key operations are idempotent and whether delivery guarantees are clear. Ask yourself what would happen if a message or API call were retried or duplicated. Would the result still be correct? This small step can help turn these concepts into real habits and deepen your engineering intuition.

Key Takeaways

Race conditions are real — Concurrent access requires thoughtful design.
Idempotency is non-negotiable — Build it in from the start.
Choose your delivery semantics wisely — At-least-once + idempotency is usually the sweet spot.
Patterns like Outbox/Inbox solve real problems — But only implement them if you need them.
Understanding matters — Don't treat solutions as black boxes; invest time in learning why they work.
Modern systems are inherently distributed — Plan for failures, duplicates, and ordering issues.

The next time you use an AI-suggested solution, take a moment to understand it. Your future self (and your team) will thank you.