Posts

All the articles I've posted.

• 7 MIN READ•Apr 29, 2026

  Partitioning, Sharding, and Data Distribution Strategies

  Hash partitioning distributes data evenly. Range partitioning enables fast range scans. Both create tradeoffs. Here is how databases divide data across storage and nodes.

  data partitioningdatabase shardingpartition pruning
• 7 MIN READ•Apr 29, 2026

  What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg

  Lakehouse catalogs store metadata pointers, manage namespaces, and enforce access control. Here is the complete catalog landscape from Polaris to Glue.

  lakehouse catalogsIceberg REST catalogApache Polaris
• 8 MIN READ•Apr 29, 2026

  Buffer Pools, Caches, and the Memory Hierarchy

  Databases use buffer pools, column caches, and result caches to keep hot data in RAM. Here is how each caching strategy works and what happens when data does not fit.

  database cachingbuffer poolcache eviction
• 8 MIN READ•Apr 29, 2026

  Writing to an Apache Iceberg Table: How Commits and ACID Actually Work

  Here is exactly how an engine writes to an Iceberg table, step by step, from data files through the atomic commit that makes ACID guarantees possible.

  writing to Apache IcebergIceberg ACID transactionsatomic commit
• 8 MIN READ•Apr 29, 2026

  Volcano, Vectorized, Compiled: How Engines Execute Your Query

  The Volcano model processes one row at a time. Vectorized execution processes batches with SIMD. Code generation fuses operators into compiled code. Here is how each works.

  query execution modelsvectorized executionvolcano iterator model
• 8 MIN READ•Apr 29, 2026

  Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans

  Iceberg's hidden partitioning separates physical layout from user queries using transform functions. Here is how it works and why it eliminates accidental full scans.

  Iceberg hidden partitioningpartition transformsaccidental full table scan
• 8 MIN READ•Apr 29, 2026

  Inside the Query Optimizer: How Engines Pick a Plan

  Query optimizers transform SQL into execution plans using rule-based rewrites, cost-based search, and adaptive runtime adjustments. Here is how each approach works.

  query optimizercost-based optimizationquery planning
• 8 MIN READ•Apr 29, 2026

  Partition Evolution: Change Your Partitioning Without Rewriting Data

  Iceberg lets you change partition schemes without rewriting data. Here is how partition evolution works internally and why Hive-style partitioning could not do this.

  Iceberg partition evolutionpartition spechidden partitioning
• 8 MIN READ•Apr 29, 2026

  B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum

  B-trees balance reads and writes for OLTP. LSM trees maximize write throughput. Bitmap indexes accelerate OLAP filtering. Here is when to use each.

  database indexing strategiesB-tree vs LSM treebitmap index
• 8 MIN READ•Apr 29, 2026

  Performance and Apache Iceberg's Metadata

  Iceberg's three-layer metadata tree eliminates directory listing and enables multi-level data skipping. Here is how scan planning actually works.

  Apache Iceberg metadata performanceIceberg scan planningmanifest pruning
• 8 MIN READ•Apr 29, 2026

  How Databases Organize Data on Disk: Pages, Blocks, and File Formats

  Databases structure data on disk as heap files, sorted files, or LSM trees, then wrap it in formats like Parquet with metadata that lets engines skip irrelevant blocks.

  data file formatsParquet file formatdatabase storage internals
• 8 MIN READ•Apr 29, 2026

  The Metadata Structure of Modern Table Formats

  Iceberg uses a metadata tree, Delta Lake uses a transaction log, Hudi uses a timeline. Here is exactly how each format organizes metadata and why it matters.

  table format metadataApache Iceberg metadataDelta Lake transaction log