Towards Data Science

95 articles in the Truth Foundry index. Each links out to the original.

• Master PySpark Performance by Understanding Data Movement and Shuffles · PySpark performance hinges on understanding data partitions and minimizing expensive shuffle operations.
• Memory Bottleneck, Not Compute Power, Limits Modern AI Progress · AI performance is increasingly constrained by data movement speed rather than raw computational power.
• Enterprise AI Value Requires Workflow Redesign Before Scaling Agents · Organizations must redesign workflows and define human-agent roles before deploying more AI agents to capture business value.
• Small Sample Sizes Create Spurious Correlations in Statistical Analysis · Statistical analysis reveals that small sample sizes generate high correlations even between independent variables.
• AI Autonomy Thresholds Are Prices, Not Percentages · Autonomy decisions should be based on cost ratios, not fixed confidence percentages.
• Information Theory Framework Proposed to Solve Econometric Ensemble Modeling Limitations · Author proposes using information theory to create new metrics for weighting ensemble economic forecasts.
• Granger Causal Networks Reveal Direct and Indirect Feedback in Financial Time Series · Granger Causal Networks decompose VAR model impacts into direct and indirect feedback paths for financial variables
• Measuring Stability in Econometric Models Using Rolling Validation · The author proposes using rolling validation to measure model stability in econometrics, independent of forecast accuracy.
• Upgrading RAG Pipelines: Relational PDF Parsing and Structured Retrieval · An enterprise RAG pipeline upgrades document parsing, question parsing, retrieval, and generation to handle real-world complexity.
• Proxy-Pointer RAG: Temporal Reasoning Without Semantic Precompilation · Proxy-Pointer RAG uses lazy semantic synthesis to handle temporal queries without ingestion costs.
• Metagenomics Reveals Microbial Life on International Space Station Dining Tables · Researchers used metagenomics to identify bacteria and fungi living on the ISS dining table.
• Survival Analysis Quantifies Machine Learning Model Reliability and Lifespan · The article applies survival analysis and Weibull distributions to model ML system degradation as a time-to-failure problem.
• Building Enterprise RAG: The Generation Brick and Prompt Dispatcher · The fourth brick of an enterprise RAG system assembles prompts using a dispatcher to handle different question shapes.
• PANet Architecture Walkthrough: Enhancing Object Detection with Bottom-Up Path Augmentation · PANet improves object detection by adding a bottom-up path to FPN, restoring spatial information in deep layers.
• Stop RAG Hallucinations with Typed Answer Contracts · Enterprise RAG systems use typed schemas to force models to answer only from retrieved passages.
• Long Context Models Win Only When Signal Is Scattered, Not Front-Loaded · Long context windows offer negligible gains when key information resides in the first 512 tokens of a document.
• Author Replaces LLM-Based Wiki With Deterministic Pure Python Compiler · A pure Python pipeline replaces stochastic LLM agents for building a deterministic, linked markdown wiki.
• Six Untaught Lessons on RAG Retrieval: Why Cosine Is Not the Foundation · Enterprise RAG should prioritize structured filtering and keyword matching over naive cosine similarity.
• Tokenminning: Strategies to Reduce AI Costs and Latency Without Sacrificing Quality · AI engineers should adopt tokenminning to minimize costs and latency by optimizing prompt efficiency.
• Source-Anchored Verifiers Cut AI Hallucination Rates by Half in Loops · A deterministic, source-anchored verifier halves hallucination rates in AI loops, while self-critique fails to improve accuracy.
• t0-alpha: A 102M-Parameter Probabilistic Time-Series Foundation Model · t0-alpha is a small, open-weight transformer that forecasts time series using quantiles.
• RAG Question Parsing: Structuring Queries Before Retrieval Improves Accuracy · Structured question parsing with relational schemas outperforms naive string embedding in RAG pipelines.
• Context Engineering Defines Four Typed Inputs for Reliable RAG Systems · Context engineering structures RAG pipelines using four typed inputs: system prompt, retrieved documents, question parsing, and generation schemas.
• Data Scientists Must Master Behavioral Interviews to Land Roles · Technical skill alone isn't enough; data scientists must communicate business value and handle uncertainty in interviews.
• Maximize Productivity by Using Codex Exec as a Specialized Review Agent · The author uses Codex exec as a specialized review agent to audit code and plans generated by Claude Code.
• Hybrid Local-Cloud LLM Patterns Balance Privacy and Reasoning Power · A guide to hybrid LLM patterns combining local privacy with cloud reasoning capabilities.
• Developer Builds Context Graph Layer to Fix Multi-Agent Memory Failures · A context graph architecture outperforms vector RAG and raw history for multi-agent memory retrieval
• lmxd Daemon Enables Three LLM Agents on Single 8GB GPU · A C++ daemon named lmxd manages VRAM to run three LLM agents in parallel on an aging NVIDIA GTX 1080.
• Developer Replaces Single LLM Agent with Specialized Multi-Agent Text-to-SQL Pipeline · A developer abandons a single-agent text-to-SQL system for a multi-agent architecture to handle complex queries.
• Enterprise RAG Retrieval Uses Parallel Anchors and Single LLM Arbiter · A three-stage pipeline uses parallel keyword and embedding detection followed by one LLM call to rank RAG candidates.