AI content analytics

The quality and context of the vast amount of unstructured content (text, comments, etc.) generated by large communities can be difficult and subjective for humans to assess. Simple quantitative metrics or keyword filtering have their limitations.

Repai's solution is to leverage modern large-scale language models (LLMs) to evaluate content within a community and feed the results into a reputation score, but relying on a single model can be susceptible to bias and error. That's why Repai introduces the Hierarchical Evaluation Framework (HEF) for a more robust and fair evaluation.

How the Hierarchical Evaluation Framework (HEF) works

The HEF is comprised of three core layers, each of which plays a unique role in ensuring the accuracy and reliability of evaluations

Base Evaluation Layer

In this first layer, small, subject-specialized LLMs evaluate content for professionalism, accuracy, and originality. Each model evaluates the following aspects

• Topical relevance of the content (0-10)

• Accuracy and currency of information (0-10)

• Clarity and logical structure of the description (0-10)

• Originality and new insights (0-10)

• Alignment with community values (0-10)

Meta Verification Layer

The second layer validates the results of the foundation assessment layer and performs additional quality control. This layer consists of the following independent modules

• Plagiarism check module: generates content embeddings to calculate similarity to existing posts

• Fact-checking module: extracts facts claimed in content and checks them against trusted external sources

• Consistency assessment module: checks for logical consistency and self-contradiction within a post.

• Sentiment analysis module: Analyzes the emotional tone of con

• tent to assess whether it promotes constructive dialog or disruption.

Consensus Derivation Layer

The results from the previous two layers are synthesized using statistical methods (weighted average, ensemble learning, etc.) to produce a final reputation score.

The formula for the consensus derivation layer can be expressed as follows

Final reputation score = α × (weighted average of basic evaluation scores) + β × (meta-validation correction) + γ × (existing reputation score)

Expand the role of AI

Leverage LLM to understand the topic, tone, and logical structure of content, support multiple languages, detect plagiarism and duplication, assess sentiment and reactions, and more. Continuously improve models through community feedback and learning from best content, with AI ratings acting as an adjunct and cross-validation to human judgment