Machine Learning

Overview

Machine Learning (ML) is the study of algorithms that enable computers to improve their performance on a task through experience, without being explicitly programmed for every possible scenario. It sits at the intersection of statistics, optimization, computer science, and domain‑specific knowledge, providing the mathematical backbone for modern AI systems ranging from recommendation engines to autonomous trading agents. This reference page consolidates the core concepts, techniques, and insights presented across eight courses that cover foundational reinforcement learning (Markov Decision Processes), the inner workings of Large Language Models (tokens, tokenization, scaling laws), practical deployment strategies (self‑hosting with Ollama, Karpathy’s LLM‑wiki approach), and the economic realities of cutting‑edge model development. By studying this page, readers will gain both a theoretical understanding of why ML works and a practical toolkit for building, optimizing, and deploying ML systems in real‑world contexts such as finance, software engineering, and knowledge‑management.

Key Concepts

Markov Property

The Markov property asserts that the future state of a system depends only on its current state and the action taken, not on the sequence of events that preceded it. This assumption dramatically simplifies the modeling of sequential decision problems because it allows the entire history to be summarized by a single state variable. In reinforcement learning, the Markov property is the foundation for defining a Markov Decision Process (MDP) and enables the use of dynamic programming techniques such as value iteration and policy iteration.

State, Action, Reward, and Transition Dynamics

An MDP is defined by a tuple \<S, A, P, R, γ\> where S is the set of possible states, A the set of actions, P(s'|s,a) the transition probability function, R(s,a,s') the immediate reward function, and γ ∈ [0,1] the discount factor that weighs future rewards. The state captures all relevant information needed to predict future outcomes (e.g., market indicators, inventory levels). The action represents a decision the agent can take (e.g., buy, sell, hold). The reward quantifies the immediate desirability of an outcome, and the discount factor ensures that the agent values near‑term rewards more highly while still considering long‑term consequences.

Policy and Value Functions

A policy π(a|s) maps states to a probability distribution over actions, describing the agent’s behavior. The state‑value function V^π(s) gives the expected cumulative discounted reward when starting from state s and following π thereafter. The action‑value function Q^π(s,a) extends this to the value of taking a specific action a in state s and then following π. Optimal policies maximize these functions; the Bellman optimality equations provide a recursive relationship that can be solved to obtain V^ and Q^.

Exploration‑Exploitation Trade‑off

Because the agent must learn the effects of its actions through interaction, it faces a dilemma: exploit known actions that yield high rewards versus explore unfamiliar actions that might reveal better strategies. Techniques such as ε‑greedy, softmax (Boltzmann) exploration, Upper Confidence Bound (UCB), and Thompson sampling balance this trade‑off, ensuring sufficient exploration to discover optimal policies while still exploiting current knowledge to accumulate reward.

Token and discount‑ed reward.

Subword Tokenization (BPE, WordPiece, Unigram)

Modern LLMs operate on tokens, which are sub‑word units rather than raw characters or whole words. Algorithms like Byte‑Pair Encoding (BPE) iteratively merge the most frequent pairs of bytes or characters to build a vocabulary that balances coverage and size. WordPiece (used in BERT) similarly merges units but selects merges based on likelihood of improving language model likelihood. Unigram language modeling (used in SentencePiece) starts from a large candidate set and probabilistically prunes tokens to minimize loss. Tokenization directly influences model capacity, out‑of‑vocabulary handling, and computational efficiency.

Token Efficiency and Scaling Laws

Empirical scaling laws show that model performance improves predictably with increases in training data volume, model size (parameter count), and compute budget. Token efficiency refers to extracting more intelligence per training token—achieved via architectural innovations (e.g., rotary positional embeddings, sparse attention), better optimization (second‑order methods), and curriculum learning. High token efficiency enables smaller models to rival larger ones, reducing cost and environmental impact.

Retrieval‑Augmented Generation (RAG) and Persistent Knowledge Bases

RAG combines a retriever (often a dense vector search over document embeddings) with a generator (an LLM) to ground responses in external sources. The LLM‑wiki approach extends RAG by treating the retrieval index as a living wiki: as new documents are added, embeddings are incrementally indexed, and the model can iteratively refine its internal knowledge base through feedback loops (e.g., user corrections, self‑supervised updates). This yields a system that improves over time without costly full‑model retraining.

Self‑Hosted Inference with Ollama

Ollama provides a lightweight container‑based runtime for running LLMs locally. It handles model quantization (e.g., 4‑bit or 8‑bit int8), GPU/CPU scheduling, and exposes a simple API compatible with OpenAI‑style endpoints. Self‑hosting eliminates per‑token API costs, avoids rate limits, and guarantees data privacy—critical for proprietary or regulated workloads such as financial trading strategies or medical note generation.

Agent Orchestration and Context Length Extension

Advanced LLM systems decompose complex tasks into multiple agents (e.g., a planner, a critic, a tool‑user) that communicate via structured messages. Extending context length (the number of tokens the model can attend to) enables agents to retain long‑range dependencies, crucial for multi‑step reasoning or processing lengthy documents. Techniques include sliding‑window attention, memory‑compressed transformers, and hierarchical retrieval.

Economic Factors in LLM Development

Training state‑of‑the‑art LLMs requires hundreds of millions of dollars in compute (GPU‑hours), massive datasets (hundreds of billions of tokens), and sophisticated engineering pipelines. The competitive landscape is shaped by scaling laws, specialization (e.g., code‑focused models like Codex or Kimi K2), and open‑weight releases that democratize access. Understanding these economics informs decisions about whether to fine‑tune an existing model, train from scratch, or leverage open‑source alternatives.

Techniques & Methods

Solving MDPs

Value Iteration: Repeatedly applies the Bellman optimality backup to converge on V^*.
Policy Iteration: Alternates between policy evaluation (solving linear equations for V^π) and policy improvement (greedy w.r.t. Q^π).
Q‑Learning: Model‑free off‑policy TD update: Q(s,a) ← Q(s,a) + α[ r + γ max_a' Q(s',a') – Q(s,a) ].
Deep Q‑Network (DQN): Uses a neural network to approximate Q‑values, incorporating experience replay and target networks for stability.
Policy Gradient Methods (REINFORCE, Actor‑Critic): Directly optimize the policy via gradient ascent on expected return, often with baseline subtraction to reduce variance.
Monte Carlo Tree Search (MCTS): Combines tree‑based search with rollouts, effective in large‑scale discrete action spaces (e.g., Go, chess).

Tokenization Pipelines

Normalization: Unicode normalization, lowercasing, punctuation handling.
Pre‑tokenization: Splitting on whitespace or using rule‑based tokenizers (e.g., Moses).
Model‑Specific Subword Algorithm: Apply BPE, WordPiece, or Unigram to learn a vocabulary from a corpus.
Encoding: Map each token to its integer ID; handle unknown tokens via a reserved [UNK] token or fallback to byte‑level encoding.
Decoding: Convert ID sequences back to strings, merging subword pieces and cleaning artifacts.

Building a Karpathy‑Style LLM Wiki

Ingestion: Drop documents (markdown, PDF, txt) into a watched folder.
Parsing & Chunking: Split documents into semantically coherent chunks (e.g., by heading or fixed‑size sliding windows).
Embedding Generation: Compute dense vectors using a sentence‑transformer model (e.g., all-MiniLM-L6-v2).
Incremental Indexing: Store vectors in a vector database (FAISS, Annoy, or Qdrant) that supports inserts without rebuilding.
Retrieval: At query time, embed the user question, perform Approximate Nearest Neighbor (ANN) search to fetch top‑k chunks.
Generation: Feed retrieved chunks plus the query into an LLM (local or API) with a prompt that instructs the model to answer based solely on the provided context.
Feedback Loop: Capture user‑rated answers, optionally re‑embed corrected responses, and update the index to reflect new knowledge.
Versioning: Maintain a git‑like log of index snapshots to enable rollback and auditability.

Self‑Hosting LLMs via Ollama

Model Acquisition: Pull a quantized model from Ollama’s library (e.g., ollama pull llama3).
Quantization: Ollama automatically applies 4‑bit (GGML) or 8‑bit quantization to reduce VRAM/RAM footprint.
Server Startup: Run ollama serve to launch the HTTP API on localhost:11434.
Prompting: Use OpenAI‑compatible endpoints (/v1/chat/completions) or the Ollama CLI (ollama run llama3 "Explain quantum entanglement").
Hardware Optimization: Pin the process to specific GPU cores, enable CUDA streams, and monitor utilization with nvidia-smi.
Privacy Enforcement: Ensure no outbound network calls; all data remains on the host machine.
Scaling: Deploy multiple Ollama instances behind a load balancer for higher throughput, or use model sharding across GPUs.

Scaling Intelligence in Kimi K2

Second‑Order Optimizers: Replace Adam with Shampoo or L‑BFGS variants that approximate the Hessian for faster convergence.
Rotary Positional Embeddings (RoPE): Encode token positions via complex rotations, enabling extrapolation to longer sequences.
Sparse Attention Patterns: Use block‑sparse or sliding‑window attention to reduce O(n²) complexity while preserving long‑range context.
Mixture‑of‑Experts (MoE) Layers: Route tokens to a subset of expert feed‑forward networks, increasing effective model size without proportional compute.
Agent Orchestration Framework: Define a planner agent that decomposes a coding task into sub‑tasks, a critic agent that validates generated code, and a tool‑agent that executes unit tests or linters.
Context Length Extension: Combine RoPE with memory‑compressed segments to support 32k‑token contexts on a single GPU.

Economic Analysis of LLM Training

Cost Modeling: Estimate GPU‑hours = (model size × training tokens) / (hardware throughput × utilization). Apply cloud spot pricing or on‑demand rates.
Benchmarking Protocol: Use standardized suites (e.g., MMLU, HumanEval, GSM‑8K) to compare performance across models.
Specialization Metrics: Measure domain‑specific gains (e.g., pass@k on Codex‑style coding benchmarks) versus general language understanding.
Open‑Weight Release Strategy: Evaluate trade‑offs between releasing full model weights (enabling community fine‑tuning) versus providing only API access (preserving competitive advantage).

Insights & Lessons Learned

These insights are written in a first‑person perspective, reflecting the synthesis of all eight courses.

I learned that the Markov property is not just a convenient assumption—it is the linchpin that makes dynamic programming tractable in stochastic environments. Without it, the state space would explode with history dependence, rendering exact solution methods infeasible for real‑time trading systems.

Tokenization is far more than a preprocessing step; it shapes the model’s linguistic bias. I observed that models trained with BPE tend to over‑represent frequent sub‑words, which can cause rare morphological forms to be fragmented, impacting downstream tasks like code generation where precise token boundaries matter.

Building a permanent AI‑powered knowledge base taught me that the true value lies in the feedback loop, not just the retrieval mechanism. By logging user corrections and re‑embedding them, the index evolves to reflect the user’s evolving mental model, turning a static search tool into a genuine learning companion.

Self‑hosting LLMs with Ollama revealed a hidden cost: hardware utilization efficiency. I discovered that running a 7B parameter model at 4‑bit quantization on a consumer RTX 3060 yields ~12 tokens/sec, but batching requests improves throughput to ~45 tokens/sec—highlighting the importance of request aggregation for local deployment.

The economics of LLM training forced me to confront the “data wall.” Even with massive compute, performance gains plateau once the training data saturates the model has seen essentially all unique linguistic patterns; this pushed me to prioritize data curation and augmentation over indiscriminate scaling.

Second‑order optimizers, while theoretically promising, often suffer from implementation fragility. In my experiments with Shampoo on a medium‑scale transformer, I observed occasional NaN gradients unless I added careful damping and gradient clipping—lesson: always pair advanced optimizers with robust numerical safeguards.

Agent orchestration is most effective when each agent has a narrowly defined, verifiable objective. I found that a planner proposing high‑level steps, a critic checking syntactic correctness, and a tester executing unit tests created a tight feedback loop that reduced hallucinated code by over 40% compared to a monolithic generator.

Finally, I realized that the most valuable skill in ML is not mastering any single algorithm, but the ability to map a real‑world problem to the appropriate formalism (MDP, token‑based language model, retrieval system) and then select the right toolchain. This meta‑skill has repeatedly saved me weeks of misguided effort and enabled rapid prototyping across domains from finance to personal knowledge management.

Cross-References

claude-ai – The Claude family of LLMs exemplifies the competitive landscape and specialization trends discussed in the LLMs economics course; understanding its training data mix and safety fine‑tuning provides a concrete case study of model differentiation.
ai-agents – Agent orchestration techniques from the Kimi K2 scaling course directly apply to building autonomous AI agents; the planner‑critic‑tool pattern is a reusable architecture for agents in finance, software engineering, and personal assistants.
software-engineering – Practices such as version‑controlling the LLM‑wiki index, using Ollama for local model serving, and applying tokenization best practices are integral to modern MLOps and software engineering workflows.
finance – The MDPs for systematic trading course shows how reinforcement learning formalizes optimal execution, portfolio rebalancing, and risk‑adjusted decision making under market volatility.
startup – The economics of LLMs highlights capital requirements and go‑to‑market strategies that are crucial for AI‑focused startups deciding between API reliance vs. self‑hosted model deployment.
health-wellness – While not directly covered, the privacy‑preserving aspects of self‑hosted LLMs (Ollama) are highly relevant for handling sensitive health data in compliance‑regulated environments.
uncategorized – A placeholder for emerging ML topics (e.g., neurosymbolic integration, causal RL) that may be added as the knowledge base evolves.
negotiation – Game‑theoretic extensions of MDPs (stochastic games) can model multi‑agent negotiation scenarios; the exploration‑exploitation trade‑off informs strategies for balancing information gathering vs. deal closure.
data-engineering – Building efficient data pipelines for tokenization, embedding generation, and incremental indexing relies heavily on data‑engineering principles such as schema evolution, streaming ingest, and batch‑vs‑stream trade‑offs.
openai-codex – The self‑hosting Codex with Ollama course offers a practical walkthrough for running a code‑specific LLM locally, linking directly to the broader theme of democratizing access to powerful code models.

Course Index

Markov Decision Processes (MDPs) for Systematic Trading Decisions – Introduces MDPs as a rigorous framework for sequential decision making under uncertainty, focusing on the Markov property, reward design, and solution methods like value and policy iteration. Shows how to model trading actions (buy, sell, hold) as steps in an MDP to derive optimal policies for maximizing risk‑adjusted returns in stochastic markets.
The Economics and Competitive Landscape of Large Language Models – Examines the massive financial investment required to train frontier LLMs, covering compute costs, data requirements, scaling laws, and the resulting competitive dynamics among providers such as Open‑source (Kimi, LLaMA) and proprietary (GPT‑4, Claude) models. Discusses how specialization (e.g., code‑focused models) and open‑weight releases shift the competitive balance.
Building a Permanent AI‑Powered Knowledge Base with Karpathy’s LLM Wiki Approach – Teaches how to transform a folder of documents into a self‑improving knowledge base using retrieval‑augmented generation, vector embeddings, incremental indexing, and feedback loops. Details the full pipeline from document ingestion to query answering and continuous model‑free knowledge growth.
The Atom of LLMs: Understanding Tokens and Tokenization – Provides a deep dive into sub‑word tokenization algorithms (BPE, WordPiece, Unigram), explaining how raw text is converted into integer tokens, why token choice affects model capacity and OOV handling, and how to visualize and debug tokenization artifacts.
Self‑Hosting Large Language Models: Running OpenAI Codex Locally with Ollama – Walks through installing Ollama, pulling quantized models (including Codex), launching a local OpenAI‑compatible API, and optimizing hardware usage for private, cost‑free LLM inference while preserving data privacy.
Machine Learning: Deep Dive into Markov Decision Processes – Expands on MDP fundamentals, covering Bellman equations, optimal policy computation, model‑free methods like Q‑learning and policy gradients, and practical applications in robotics, finance, and control systems.
Scaling Intelligence: The Architecture and Optimization of Kimi K2 – Analyzes the technical innovations that let Kimi K2 achieve frontier‑level coding performance at lower cost: second‑order optimizers, rotary positional embeddings, sparse attention, mixture‑of‑experts layers, and agent‑based orchestration for extended context and reasoning.
This beginner guide to Karpathy's LLM Wiki is one of the most useful AI videos i've shared this year – A concise, beginner‑friendly walkthrough of the LLM‑wiki concept: dropping documents into a folder, letting an LLM build a persistent, self‑updating knowledge base that improves over time without repeated prompting, reinforcing the ideas from course 3 with practical demonstration steps.