Slack's AI-Powered Migration from Enzyme to React Testing Library

This article details how Slack leveraged a hybrid approach combining Abstract Syntax Trees (AST) and Large Language Models (LLM) to automate their massive migration from Enzyme to React Testing Library, saving thousands of engineering hours.

The Challenge: 15,500 Tests

Migrating 15,500 tests—each taking 30-45 minutes to convert manually—presented a significant hurdle. This would have required an estimated 10,000+ engineering hours. The team at Slack knew they needed a more efficient solution.

<img src="https://imgopt.infoq.com/fit-in/1288x0/filters:quality(80)/presentations/ai-migration-large-scale/en/slides/Ser-1756815497400.jpg" alt="Robotic Arm Analogy for Code Migration">

Initial Attempts: AST and LLM Alone

Initially, Slack experimented with both AST-based codemods and LLMs independently. While the AST approach offered some automation, its success rate was limited (around 45%). The LLM approach, while promising, also had significant variability in its success rate (40-60%).

The Winning Hybrid Approach: Combining AST and LLM

The breakthrough came from combining both techniques. Their innovative pipeline involved:

1. Context Collection: Gathering file code, DOM trees, and partially converted code from the AST codemod.
2. AI API Request: Sending this comprehensive context to the LLM.
3. Response Parsing & Verification: Parsing the LLM's output, running linters, and verifying test pass/fail rates.
4. Feedback Loop (Optional): Dynamically generating improved prompts based on results.

This hybrid model dramatically increased the success rate to 80%. Key innovations included:

• DOM Tree Collection: Providing the LLM with user-perspective information (rendered HTML).
• LLM Control with Prompts and AST Annotations: Using AST to perform reliable conversions, annotating the more challenging parts for the LLM, thus improving accuracy and reducing hallucinations.

Results and Impact

The hybrid approach saved Slack an estimated 22% of developer time, demonstrating the potential of AI-assisted code migration.

Key Takeaways

• AI is a powerful tool, but it's most effective when integrated into a larger pipeline, rather than used as a standalone solution.
• Modeling the approach after how a human developer would solve the problem can significantly improve AI performance.
• Careful context collection and prompt engineering are crucial for success.

This innovative approach is transferable to other large-scale code migration projects.