Hybrid AI for Pac-Man
Capture the Flag

The problem

Capture the Flag is a two-vs-two adversarial game: each team simultaneously raids the opponent's food and defends its own, under partial observability (exact enemy positions are only known when they're close; otherwise distances are noisy) and a hard per-move compute budget. A single fixed strategy loses. A strong agent has to keep re-deciding what it should be doing (grab food, bank it, chase an invader, grab a capsule, flee a ghost) and then execute that decision well. I wanted a design that cleanly separated high-level intent from low-level control so each could be improved on its own.

The approach: a two-layer hybrid

        ┌─────────────────────────────────────────────┐
        │  HIGH LEVEL - Symbolic PDDL planner           │
        │  game state → predicates → goal → plan         │
        │  attack · defend · go-home · patrol · flank    │
        └───────────────────────┬───────────────────────┘
                                │  chosen high-level action
        ┌───────────────────────▼───────────────────────┐
        │  LOW LEVEL - feature-based Q-learning          │
        │  Q(s,a) = Σ wᵢ·fᵢ(s,a), pick the best move      │
        │  offensive · escape · capsule · defensive       │
        └───────────────────────┬───────────────────────┘
                                │  concrete move (N/S/E/W/Stop)
                                ▼
                         Pac-Man game engine

High level: a PDDL planner decides what to do

Each tick, the raw game state is translated into logical predicates (a custom pacman_bool STRIPS domain) and planned over. The planner chooses among high-level actions:

• attack: food is available and no enemy is close, so go raid
• defence: an enemy is close and invading, so intercept
• go_home_with_food: carrying enough food in enemy territory, so bank it
• go_home_with_enemy_nearby: carrying food with an enemy near, so retreat
• go_home: on the enemy side with food, return even if no threat
• patrol: holding a score lead, guard territory
• flank: coordinate with the ally to out-position the opponent
• move_away: a fallback to reposition instead of sitting idle

Low level: Q-learning decides how to do it

Once a high-level action is chosen, a feature-based Q-learning controller picks the concrete move by scoring candidates as a linear combination of engineered features and learned weights, choosing the highest Q-value. Separate behaviours (offensive, escape, capsule, defensive, move-away) each have their own features and reward shaping.

Reinforcement learning details

• Representation: linear function approximation, Q(s,a) = Σ wᵢ · fᵢ(s,a)
• Offensive features: closest-food, #-of-ghosts-1-step-away, chance-return-food, successorScore, bias
• Defensive features: invaderDistance, onDefense, numInvaders, stop, reverse, teamDistance
• Learning: temporal-difference weight updates with learning rate α and discount γ, plus a training flag that toggles ε-greedy exploration and weight updates on/off
• Reward shaping: per-behaviour reward functions so each policy optimizes the right objective: banking food, widening distance from ghosts, or closing on an invader

Trained offensive weights (learned, not hand-set):

successorScore:           +26.2      # push up the score
chance-return-food:       +25.5      # carry food toward home
#-of-ghosts-1-step-away:   -3.7      # avoid stepping next to a ghost
closest-food:             -11.8      # don't dawdle near food without progressing
bias:                    -486        # baseline

These are interpretable and sensible: the agent learned to value scoring and safe food-return while avoiding ghosts.

Iterating on strategy

Approach 1: fixed roles

The first version hard-assigned one attack agent and one defense agent. It worked, but had real weaknesses: fixed roles caused idle downtime (an attacker with no nearby food, or a defender with no threat, wasted turns), and when both agents were eaten they respawned with no adaptive recovery, and the Berkeley baseline exploited those gaps to bank points.

Approach 2: a semi-defensive, "Kabaddi-inspired" strategy

To fix that, both agents default to a defensive stance in their own half, forming a hard-to-breach line, then opportunistically dart out to grab reachable food when it's safe before falling back, much like the raid-and-retreat rhythm of Kabaddi. This kept both agents useful at all times and reduced the downtime the baseline had been punishing.

Results

Honest drawbacks

The biggest limitation was side-dependence: because I trained and tuned primarily on the red side under time constraints, the agent played noticeably better there (≈6/10) than when switched to blue (≈4/10). The learned weights didn't fully generalize across map orientations, a clear signal that the training regimen, not the architecture, was the bottleneck. Some later, more aggressive weight configurations also became numerically unstable during training, which reinforced the value of the stable, interpretable Approach-1 weights above.

What I'd do next

• Deep Q-Learning with a shared Q-network for both agents, to generalize across states (and map sides) instead of relying on a small set of linear features, an approach I'd previously implemented for a separate multi-agent assignment, where a shared network helped agents learn complementary behaviours.
• Balanced, both-sides training to eliminate the side-dependence.
• Dynamic role switching so an agent can flip between attack and defense based on live game state rather than a fixed assignment.

What I took away

Separating what from how was the design decision that made everything else tractable. I could change a strategic decision (fixed roles → semi-defensive) without retraining, and improve execution without touching the planning logic. And measuring against real opponents, not just intuition, is what surfaced the side-dependence problem and pointed clearly at what to fix next.

Tech

Python · PDDL (STRIPS planning) · Feature-based Q-learning (linear function approximation, TD updates, ε-greedy) · UC Berkeley Pac-Man Capture the Flag framework