System Design

Crafting Protocol-Aware LLM Prompts

To generate meaningful and effective attack strategies, BunnyFinder’s Strategy Generator (SG) relies on carefully engineered prompts that help large language models (LLMs) understand and reason through the specific dynamics of Ethereum’s Proof of Stake (PoS) consensus mechanism. The prompts are carefully crafted around three key pillars:

Environment Setting

The first step in guiding the LLM involves providing it with a structured, protocol-specific foundation. This includes the following core elements:

1. Ethereum Protocol: The LLM learns essential details about Ethereum’s network structure, such as slot and epoch intervals, checkpoint finalization, and the HLMD GHOST fork choice rule that helps determine the canonical chain.

2. Adversarial Constraints: The LLM is constrained by a set of adversarial conditions, such as the network’s synchronous nature, discrete 4-second delay units, and uniform Byzantine strategies (e.g., a coordinated group of malicious validators).

3. Incentive Rules: The model is also familiarized with Ethereum’s incentive system, which includes rewards for timely attestations, penalties for equivocation (e.g., double-signing), and conditions under which validators are slashed (i.e., punished).

Example Prompt:

"Ethereum PoS assumes that the network is partially synchronous, i.e., there exists an unknown upper bound $\Delta$ for message propagation and processing. In the paper, our analysis focuses on the attacks when the network is synchronous, assuming that $\Delta$ is known by all validators. In Ethereum, time is divided into epochs. Each epoch includes 32 slots where each slot lasts for 12 seconds. For our purpose, we call four seconds (1/3 of a slot) a time unit."

Strategy Space Definition

Once the environment is established, the next step is to clearly define the strategy space. The LLM is restricted to two primary types of manipulation, which form the basis of potential attack strategies:

• Order Manipulation: This involves delaying certain actions (e.g., block proposals) by a fixed time, with the delay expressed in discrete units (multiples of 4 seconds). The model is restricted to these increments to simplify the design space and prevent overly complex strategies.
• Content Manipulation: This focuses on altering mutable fields within Ethereum’s protocol, such as modifying block parent hashes or attestation checkpoints. These are the only targets for content manipulation to maintain consistency with Ethereum’s structure.

Example Output should like this:

{ "slot": "5", "actions": { "BlockBeforeBroadCast": "return", "AttestBeforeSign": "modifyAttestHead:0" } }

Input/Output Standardization

To ensure that the strategies generated by the LLM can be directly implemented and tested, BunnyFinder uses a structured format for input and output: the Strategy Specification Format (SSF). SSF is a JSON template that maps the generated actions to specific slots within Ethereum’s blocks and attestations.

This standardized format allows the Strategy Executor (SE) to automatically parse and apply these strategies, adjusting validator clients in real-time to simulate the attack. The executor’s ability to read SSF ensures a smooth flow of information from the LLM’s output to the actual execution environment.

Iterative Refinement

After executing the generated strategies, the State Analyzer (SA) evaluates the outcomes and metrics. The analysis focuses on various key performance indicators:

• Reward Ratios: Comparison between the rewards earned by Byzantine (malicious) and honest validators.
• Penalties: Calculation of penalties incurred by validators for equivocation (signing conflicting blocks or attestations) or missed attestations.
• Finalization Delays: The impact of the attack on the finalization process, including potential delays or even chain splits.

Based on the analysis, the LLM undergoes an iterative refinement process, where it learns from its initial strategy and adapts to improve future outcomes. Specifically, the model may:

• Prune Ineffective Strategies: Attack strategies that result in negligible reward reductions (e.g., less than 5%) are discarded or minimized in subsequent rounds.
• Amplify Successful Tactics: Strategies that show significant promise—such as increasing the delay from 1 to 3 units for a higher reward impact—are amplified and tested further.
• Test Threshold Effects: The model may explore how the attack’s success varies when certain parameters, like the number of Byzantine validators, are adjusted. For example, it might test how a strategy performs when 33% of validators are compromised versus when fewer are involved.

This iterative process helps BunnyFinder refine attack strategies and better understand Ethereum’s vulnerabilities, allowing for increasingly sophisticated attacks to be designed and tested with minimal human intervention.

This version adds more structure and clarity, making it easier for readers to follow the design steps while giving them a detailed look at how BunnyFinder generates and refines attack strategies. The key process steps, like strategy definition, prompt design, and iterative feedback, are clearly laid out to demonstrate how the system evolves with each interaction.