Content Chunking Quality

Content Chunking Quality is a metric (scored 0-100) that evaluates how effectively a document's content is organized into discrete, semantically coherent units.

Chunking is a fundamental preprocessing step in RAG systems, where large documents are broken into smaller pieces to be embedded and retrieved. The quality of these chunks is paramount; if a chunk splits a single idea or lacks necessary context, the retrieval system will fail, leading to inaccurate or incomplete AI-generated answers. This metric simulates a standard chunking process and then scores the resulting chunks based on their internal coherence and the logical separation between them.

Calculation Methodology

The methodology involves a two-part scoring process applied to simulated content chunks.

Simulated Chunking

• Extract the clean text content from the page's main body.
• Apply a standard, representative chunking algorithm. A good choice is a recursive character text splitter, which attempts to preserve semantic boundaries by first splitting on paragraphs (\n\n), then sentences (\n), then spaces.
• A fixed chunk size (e.g., 256 or 512 tokens) and a small overlap (e.g., 10% of chunk size) should be used.

Chunk Coherence Scoring

This measures how tightly focused each individual chunk is.

• Internal Coherence (70% weight): For each chunk, split it into its constituent sentences. Generate a vector embedding for each sentence using a sentence-transformer model. Calculate the average pairwise cosine similarity among all sentence vectors within the chunk. A higher average similarity indicates that the sentences are semantically related and the chunk is coherent.
• Boundary Coherence (30% weight): This measures how distinct adjacent chunks are from one another, which indicates good split points. For each pair of adjacent chunks (Chunk N and Chunk N+1), generate an embedding for the entire text of each chunk. Calculate the cosine similarity between the vector for Chunk N and the vector for Chunk N+1. A lower similarity score is better, indicating a clear topic shift between chunks. The score is inverted (1 - similarity) to align with the 0-100 scale.

Calculating The Content Chunking Quality Score

The calculation of the Content Chunking Quality Score is a multi-step process that involves analyzing the HTML structure of the page. Here is a simplified pseudo-code representation of how this score is calculated.

BEGIN
 FETCH and EXTRACT clean text from the webpage.
 APPLY a standard text chunking algorithm to create a list of chunks.

 // Calculate Internal Coherence
 FOR EACH chunk:
 SPLIT chunk into sentences.
 GENERATE vector embeddings for each sentence.
 CALCULATE average pairwise cosine similarity between sentence vectors.
 AVERAGE the scores across all chunks to get avg_internal_coherence.

 // Calculate Boundary Coherence
 FOR EACH adjacent pair of chunks:
 GENERATE a vector embedding for each chunk.
 CALCULATE cosine similarity between the two chunk vectors.
 INVERT the similarity score (1 - similarity).
 AVERAGE the inverted scores across all boundaries to get avg_boundary_coherence.

 CALCULATE final_score as a weighted average of avg_internal_coherence and avg_boundary_coherence.
 RETURN final_score.
END

Conclusion

Content Chunking Quality is a technical but essential metric for GEO. By optimizing your content for chunking, you are making it easier for AI systems to understand and use your content, which is a critical step in becoming a trusted source for generative engines.