The weight of a review

I’ve been thinking about the integrity of the systems we rely on. In academia, that system is peer review. It is the foundation that ensures the validity of scientific knowledge. But lately, I’ve noticed a shift. The friction of writing a detailed, critical review is being bypassed by Large Language Models (LLMs). We are increasingly seeing reviews that are syntactically perfect but analytically hollow.

It is a subtle kind of degradation. When we accept generic feedback or “pal-reviews” (superficial comments from colleagues), we let flawed research slip through the cracks.

In our recent paper, Classifying Scientific Peer Reviews: Distinguishing Authentic, Generic, and AI-Generated Feedback, my co-authors and I decided to tackle this problem head-on. We wanted to see if we could automatically audit and categorize peer reviews to safeguard the academic publishing ecosystem.

Building the dataset

To train a model to catch a generic or AI-generated review, you first need a high-quality dataset. We curated a specialized dataset of 399 reviews divided into three categories:

1. Authentic reviews: Sourced directly from journals that publish open peer reviews, such as MDPI and PeerJ. These serve as our baseline for human, expert-level critique.
2. AI-Generated reviews: Prompted from models like ChatGPT, DeepSeek, and Claude using published paper abstracts.
3. Generic reviews: Assembled using sentence concatenation tools to mimic low-effort “pal-reviews” (e.g., “The paper has several typos,” “The title is too long”).

Finding the right filter

We wanted to evaluate how well different transformer architectures could handle the nuanced task of classifying academic text. We tested five models: BERT, DistilBERT, XLNet, RoBERTa, and GPT-4o-nano.

For the BERT-family models, we implemented tokenization (max length: 512), optimized batch sizes, and used focal loss to handle class imbalance. We employed a rigorous training routine with k-fold cross-validation and early stopping.

graph TD
    classDef input fill:#fff,stroke:#333,stroke-width:2px;
    classDef model fill:#f9f9f9,stroke:#333;
    classDef output fill:#e1f5fe,stroke:#333;

    Input["Paper Title + Abstract + Review Text"]:::input
    
    subgraph Architectures [Evaluated Models]
        M1["RoBERTa"]:::model
        M2["DistilBERT"]:::model
        M3["XLNet"]:::model
        M4["BERT"]:::model
        M5["GPT-4o-nano"]:::model
    end
    
    Input --> Architectures
    
    Architectures --> Classify
    Classify["Three-Class Categorization"]
    
    Classify --> O1["Authentic"]:::output
    Classify --> O2["AI-Generated"]:::output
    Classify --> O3["Generic"]:::output

The results: Why GPT-4o-nano won

When evaluating on our held-out test set, the traditional BERT-family models struggled, particularly with the “Generic” class. DistilBERT and RoBERTa showed moderate performance, with RoBERTa achieving a 71.3% accuracy.

However, GPT-4o-nano completely shifted the narrative. It substantially surpassed all other models:

• Accuracy: 80.7%
• Macro-F1: 80.6%

Class-wise F1 Scores

Metric	BERT	DistilBERT	RoBERTa	XLNet	GPT-4o-nano
AI-Generated	0.690	0.770	0.840	0.820	0.880
Authentic	0.730	0.760	0.740	0.670	0.830
Generic	0.430	0.220	0.550	0.470	0.730

GPT-4o-nano demonstrated a massive leap in identifying “Generic” reviews—a category where older models failed to capture the subtle generative characteristics and lack of critical depth.

Future directions

We have made our training pipeline and the fine-tuned models openly available on the Hugging Face Model Hub and our code on GitHub for reproducibility and future research.

As LLMs continue to evolve and refine their language mimicry, classifying human and AI-generated text will become more challenging. Future work will involve exploring multi-modal approaches that incorporate reviewer history, paper metadata, or review sentiment to further enhance classification accuracy. We also plan to investigate custom classification architectures built around CNNs and advanced transformers like LONGFORMER or ELECTRA, as well as techniques like few-shot learning to apply models to new domains with limited data.

The cost of carrying everything

We need to stop computing everything. A smartphone with limited thermal headroom shouldn’t activate every parameter for a simple “hello” message. We can solve this with Mixture of Experts (MoE), an architecture where we split a model into specialized sub-networks and only fire the ones we need for a specific token. In my recent work, I converted Qwen3-0.6B into a sparse MoE (which I call TinyMoE), achieving 20 tokens/sec under mobile-equivalent constraints. The ultimate takeaway? We don’t need trillion-parameter models on our phones, we just need selective activation.

Refusing to scale normally

For years, the paradigm in machine learning has been dense computation. Every neuron fires for every input token. But human cognition doesn’t work that way (we don’t activate our entire brain just to recognize a coffee cup…). Why should our models?

With frontier models like Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.1, we’ve seen the raw power of immense scale. But maintaining that scale locally, in our pockets, is impossible using dense architectures.

A brief primer on selective activation

A Mixture of Experts architecture splits the dense layers of a neural network into multiple smaller “experts.” A routing network then decides which experts to use for a given token. It gives you the “superposition” of a massive model with the energy footprint of a small one.

graph TD
    classDef input fill:#fff,stroke:#333,stroke-width:2px;
    classDef router fill:#f9f9f9,stroke:#333;
    classDef expert fill:#fff,stroke:#333;
    classDef output fill:#f9f9f9,stroke:#333,stroke-dasharray: 5 5;

    Input["Input Tokens"]:::input
    Router["Router/Gating Network"]:::router
    E0["Expert 0"]:::expert
    E1["Expert 1"]:::expert
    E2["Expert 2"]:::expert
    E3["Expert 3"]:::expert
    Output["Weighted Aggregation"]:::output

    Input --> Router
    Router -->|"top-k=2"| E0
    Router -->|"top-k=2"| E1
    Router -.->|"not selected"| E2
    Router -.->|"not selected"| E3
    E0 --> Output
    E1 --> Output

This architectural sleight-of-hand is how today’s frontier models scale so aggressively. It’s why DeepSeek V4 can manage ~1 trillion total parameters while keeping active parameters constrained to ~37B. It’s why Kimi k2.5 destroys coding benchmarks while running efficiently natively, and it’s the foundation beneath Google’s massive Gemini 3 Pro. They act like massive-parameter models but only activate a tiny, specialized fraction per forward pass.

The burden of choosing

The catch? The router has to actually learn how to route. If it gets lazy, it sends everything to Expert 0. We call this “router collapse,” and it destroys the entire purpose of sparsity. Why learn to route when one expert is “good enough”?

Fighting the urge to rely on a single expert

We use an auxiliary loss during training. It’s essentially a penalty for imbalance. If you multiply the load (how many tokens an expert receives) by the importance (the average routing probability) and sum it up, you get a value. A high value means extreme imbalance. We force the network to minimize this number alongside the main loss, ensuring the workload stays beautifully distributed.

Shrinking the concept (TinyMoE)

I wanted to feel this constraint in my own hands. I decided to convert a pre-trained dense model (Qwen3-0.6B, representing the incredible Qwen 2.5/3 lineage) into an MoE. I call it TinyMoE.

Taking it apart to make it better

Instead of training from scratch (which requires millions of GPU hours), you upcycle. You initialize the new experts by duplicating the weights of the pre-trained dense feed-forward networks, adding a tiny amount of scaled noise to each. Then, you freeze most of the model and train only the router mechanism on your dataset.

# A conceptual snippet of TinyMoE upcycling
def upcycle_to_moe(dense_ffn, num_experts=4, noise_scale=0.01):
    experts = nn.ModuleList()
    for _ in range(num_experts): # We used 4 experts
        # Copy dense weights and add slight perturbation for diversity
        expert = copy.deepcopy(dense_ffn)
        expert.weight.data += torch.randn_like(expert.weight) * noise_scale
        experts.append(expert)
    return experts

I didn’t convert everything. I selectively upcycled only the middle layers (layers 8, 12, 16, and 20 out of 28). Early layers handle raw feature extraction and late layers handle task-specific output. The middle is where representations become diverse enough to warrant true experts.

The result? The active parameter count per token stayed incredibly low (~450M), but the absolute capacity increased by 19% to capture richer representations. Generation held at a respectable 20 tokens/second on edge hardware.

Keeping track of what matters

(And progressive disclosure…) I’m learning that MoE isn’t just a datacenter trick for training behemoths anymore. It’s an elegant hardware optimization strategy.

What if our routing was task-aware? If my phone is at 10% battery, the router could dynamically drop from top-k=2 to top-k=1, gracefully degrading its reasoning depth to extend battery life. Total energy reduction? Roughly 32% in my tests with less than 2% drop in accuracy.

This is why I do this work. Not for the benchmarks, but for the craft. It’s about “ricing” a system until the friction is completely gone. We don’t just need bigger models. We need models that know how to care about our constraints as much as we do. It’s the small wins that push us forward.

Finding Speed in Silicon

I’ve been thinking about the cost of waiting. Not the big delays, but the small slivers of time that disappear while we wait for a cursor to move. We often accept latency as a necessary tax for complexity. But after looking at Cerebras, I’m starting to think that tax might be optional.

Cerebras, based in Sunnyvale, built the Wafer-Scale Engine (WSE-3). It isn’t just a chip. It is a piece of silicon that wasn’t cut.

The silicon we choose not to cut

Standard GPUs are like separate islands connected by bridges. The interconnects are the bottleneck. You spend a lot of time just moving data between them. Cerebras is different because it is a single, contiguous continent of silicon.

900,000 cores. 44GB of SRAM. All on one piece of glass.

graph TD
    classDef wse fill:#fff,stroke:#333,stroke-width:2px;
    classDef core fill:#f9f9f9,stroke:#333;
    classDef memory fill:#fff,stroke:#333;
    classDef fabric fill:#f9f9f9,stroke:#333,stroke-dasharray: 5 5;

    WSE["Wafer-Scale Engine (WSE-3)"]:::wse

    subgraph WSE_Internal [Inside the Wafer-Scale Engine]
        Swarm["2D Swarm Interconnect Fabric"]:::fabric

        Core1["Sparse Linear Algebra Core (SLAC)"]:::core
        SRAM1["44GB Ultra-Fast Local SRAM"]:::memory
        Core1 <-->|"1 clock cycle"| SRAM1
        Core1 <-->|"High Bandwidth"| Swarm

        CoreN["... 900,000 AI-Optimized Cores ..."]:::core
        SRAMN["... Local SRAM ..."]:::memory
        CoreN <-->|"1 clock cycle"| SRAMN
        CoreN <-->|"High Bandwidth"| Swarm
    end

    WSE --> WSE_Internal

    DataFlow["Dataflow Execution Model (Event-driven)"]
    WSE_Internal --> DataFlow

When memory access happens in a single clock cycle, the latency disappears. It is the hardware version of “ricing,” which is optimizing the foundation until the friction is gone. It reminds me of trying to build the perfect local development setup. It is minimal and focused.

Speed as a foundation

We usually talk about AI reasoning, but speed is a form of intelligence. When a model responds at 2000 tokens per second, the experience changes. It stops being a tool you call and starts being a stream of thought you can actually follow. Meta uses this for their Scout model, and OpenAI uses it for Codex.

I’m interested in what happens when this becomes the standard.

Building with care is a phrase I keep coming back to. Speed isn’t just about efficiency. It is about reducing the distance between having an idea and seeing it work. I am trying to raise the bar, even if I am still learning where the bar is.

The boundaries of a single continent

A single continent has its own limits. You can’t run every model on this hardware. It requires a specific kind of architectural alignment. It is like choosing a minimalist framework. You gain speed and clarity, but you lose the flexibility of a larger, more bloated system.

Other providers take different paths. Baseten is practical. They optimize the GPU islands we already have. Mercury, from Inception Labs, rethinks the math of generation itself. They use diffusion models to generate tokens in parallel instead of one after another.

The work ahead

In the end, these are all different ways of trying to make the computer get out of the way. When the friction is gone, the tool disappears.

I am still figuring out my own approach to this. Building with care is a slow process, but it is the only way I know how to reach something that feels effortless. I am focused on the basics for now. One small win at a time. Every single day.

Approach	Philosophy	Speed
Cerebras	Build a single continent.	1000-2000+ TPS
Baseten	Optimize the bridges.	~341 TPS
Mercury	Parallelize the generation.	1000+ TPS

The polish I want will come as I keep building. One small win at a time. Every single day.