AI-Assisted Code Review: What the Tools Catch and Where Humans Still Win

AI code review tools have moved from experimental to routine for a significant portion of engineering teams. GitHub Copilot’s review feature, CodeRabbit, Greptile, Graphite Automations, and several others now sit in the PR pipeline for teams that have adopted them. The question isn’t whether to use them; for most teams, some form of AI review makes sense. The question is what they’re actually useful for, what they’re not, and how to integrate them without creating noise.

This is a ground-level look at what these tools do well, where they fall short, and a workflow that gets value from both without burning reviewer attention on things AI handles better.

What AI Code Review Is Actually Doing

Under the hood, these tools send your diff to a language model along with context from the surrounding codebase. The model returns comments on specific lines, summaries of what changed, and sometimes suggestions for alternative implementations.

The quality depends on three variables:

1. The size of the context window and how much relevant code the tool includes
2. The quality of the underlying model (most use Claude Sonnet or GPT-4-class models)
3. How well the tool was prompted to review code rather than just describe it

The better tools pull in relevant files from the codebase, not just the diff. If your PR modifies UserService.create(), a good tool looks at UserService, its tests, and any calling code to give comments with actual context. A weaker tool comments only on what’s in the diff.

Where AI Review Is Genuinely Better Than Humans

Catching obvious bugs quickly. AI tools don’t miss return statement omissions, off-by-one errors, null pointer possibilities, or common async pitfalls. They catch these every time, in every PR, without fatigue. Human reviewers catch most of these too, but not all of them, especially late in a review session or on a large diff.

# AI catches this reliably:
async def get_user(user_id: int):
    user = await db.query(User).filter(User.id == user_id).first()
    return user.email  # AttributeError if user is None

Identifying missing error handling. API calls, file operations, database queries that don’t handle the failure case. AI tools flag these consistently.

Style and convention enforcement. If your codebase uses a specific naming convention or pattern, and new code deviates, AI review catches it. This is especially useful for teams with contributors across different experience levels.

Documentation completeness. Missing docstrings, public functions without parameter descriptions, complex logic without a comment explaining the why. These are things human reviewers often let slide under deadline pressure.

Security pattern violations. Obvious ones: SQL string concatenation instead of parameterized queries, eval() on user input, hardcoded credentials, MD5 for password hashing. AI tools are reliable here for the well-known patterns.

Summary generation. The PR summary feature in tools like CodeRabbit is underrated. For large PRs, an accurate summary of what changed and why, generated automatically, saves the reviewer 5-10 minutes before they look at a single line of code.

Where AI Review Falls Short

Business logic correctness. The model doesn’t know what your product is supposed to do. It can tell you that the code is syntactically correct, but it can’t tell you that your billing calculation now charges users twice when they upgrade. Domain knowledge lives in your team’s heads, not in the model.

Architectural tradeoffs. “Should this be a method on the User model or a standalone service?” requires understanding the system’s direction, the team’s conventions, and the tradeoffs of both approaches. AI tools give surface-level opinions here that often miss the actual considerations.

Test quality judgment. AI review can tell you that tests exist and that the assertions make sense structurally. It can’t tell you whether the tests cover the right cases, whether the test boundaries are meaningful, or whether the mock assumptions will break when reality diverges from them.

Performance in context. Spotting an O(n²) loop is within reach for AI tools. Understanding whether that loop matters, because it runs over 10 items in a bounded configuration rather than user data, requires context that doesn’t fit in a prompt.

Interpersonal dynamics. Whether to push back on a design decision, how to phrase a concern to a junior developer, when to ship something imperfect and iterate. These are judgment calls that require knowing the people and the project.

// AI might flag this as O(n²) needing optimization
// But if arr is always < 10 items (a config array), the comment is noise
function findMatchingRules(arr: Rule[], input: string): Rule[] {
  return arr.filter(rule => 
    arr.some(other => other.id !== rule.id && other.pattern.test(input))
  );
}

The Workflow That Actually Works

The most effective pattern separates what AI review handles from what human review handles, rather than running them in parallel and creating duplicate comments.

AI review runs first, automatically. On every PR, the AI tool posts its analysis before any human looks at the code. This is configured in CI and requires no manual triggering.

Address AI comments before requesting human review. The author resolves or explicitly dismisses the AI’s comments. Anything the AI flagged that’s actually a problem gets fixed. Anything that’s a false positive gets marked as dismissed with a one-line explanation.

Human review focuses on what AI can’t assess. The human reviewer reads the PR with the knowledge that obvious issues have been caught. Their attention goes to: Does this solve the right problem? Does it fit the system’s design direction? Are the test cases meaningful? Are there edge cases that matter in production?

Review comments from humans don’t overlap with AI. If the AI already caught a missing null check and the author fixed it, the human reviewer doesn’t also comment on null checks. The division of labor is explicit.

This workflow changes what reviewers spend time on. The time previously spent catching typos, missing returns, and style issues moves to higher-level feedback. Review quality tends to go up, not just review speed.

Configuring the Tools

CodeRabbit is configurable through a .coderabbit.yaml at the repo root. Worth spending 30 minutes on this:

# .coderabbit.yaml
reviews:
  auto_review:
    enabled: true
    drafts: false  # don't review draft PRs
  
  # focus the AI on things it's good at
  instructions: |
    Focus on: error handling, security patterns, null safety, and missing tests.
    Skip: style comments that aren't about correctness, minor naming preferences.
    This is a TypeScript/Next.js project. Assume React and Node best practices.
  
  path_filters:
    - "!**/*.md"    # skip documentation
    - "!**/*.json"  # skip config files
    - "!dist/**"    # skip build output

GitHub Copilot code review is configured at the repository level and pulls from your existing Copilot subscription. Less configurable than CodeRabbit but zero additional cost if you’re already paying for Copilot Business.

Reducing noise is the main configuration goal. AI tools that comment on everything generate reviewer fatigue. Tune for precision over recall: it’s better to miss some minor suggestions than to have reviewers ignore comments because most of them are noise.

Measuring Whether It’s Working

The metric to track: how often do AI review comments identify something that would have been caught in human review anyway? If AI catches 80% of what human reviewers catch in the first pass, reviewers can focus their time differently.

Concrete indicators that the workflow is working:

• Human reviewers spend less time per PR on style/bug catches, more on design feedback
• Bug escape rate (issues found post-merge) stays flat or decreases
• PR cycle time decreases because review rounds drop (from 3-4 back-and-forth to 1-2)

Indicators that it’s not working:

• Reviewers are dismissing AI comments without reading them
• Authors are shipping without addressing AI flags
• The same types of bugs keep appearing despite AI review

If reviewers are dismissing comments without reading them, the tool is producing too much noise. Tighten the configuration. If the same bugs keep appearing, the tool isn’t being used before human review. That’s an adoption problem, not a tool problem.

Starting Point

If you haven’t added AI code review yet, CodeRabbit’s free tier covers public repos and a limited number of private repo PRs per month. That’s enough to evaluate whether the comments are useful for your codebase before committing to a paid tier.

For teams already on GitHub Copilot Business, enable Copilot code review in the repository settings first. It’s an existing line item and takes about two minutes to enable.

The question to ask after two weeks: are reviewers reading the AI comments and finding them valuable, or are they skimming past them? The answer tells you whether you have a tool configuration problem or a workflow adoption problem, and those need different fixes.