💾Technicals

As mentioned previously, Luigi is a reasoning agent, below we'll describe some of the technical aspects that brings Luigi to life.

Architecture

• Built on LangChain's StateGraph (learn more here).

• Token-Aware System utilizing 32,000 tokens for dynamic chunking and recursive summary.

• DAG workflow for modular execution and intra-agent cooperation.

• Custom APIs to feed the model pipelines with social media/market data.

• Semantic clustering allows groups' related information to preserve overall coherence across data processing stages.

• Iterative refinement employs asychronous node execution and state persistence for enhanced decision-making.

Below, we can see the flow of information in to a resulting publication out. Summarized below as,

• Raw data ingestion

• Chunking

• Market data processing

• Semantic/sentiment analysis and clustering

• Ranking

• Iterative looping and refinement

• Synthesis and publication production

First, market summary node fetches data, implements dynamic chunking, token-aware, maintaining context windows of 32,000 tokens.

Overall Market Analysis is then performed

From here, ranking system takes over via ticker and narrative rankings.

• Ticker: frequency analysis, sentiment scoring, volume metrics, historical correlates are all applied.

• Narrative: topic modeling and trend detection, as well as semantic clustering.

Next, the iterative refinement looping (three loops maximum to prevent overfitting), where Luigi refines its analysis in multiple stages: identification, verification, and storage via DAG.

Within Luigi's DAG,

• Nodes represent tastks like data digestion, rankings, and synthesis

• Edges define dependencies between tasks

• Token management ensures efficient processing of the dataset, which is important for semantic and narrative preservation.