Ihnat Z

All My Meters Said Innocent

Fri, 03 Jul 2026 22:56:17 +0000

Just after lunch on a Monday, an alert fired: FIX session disconnected. That was new. The session dropped at 09:30 NY time — the US market open, when the accumulated pre-market orders execute and a significant share of the day’s volume goes through in minutes. For a drop-copy session, a disconnection at the open is the expensive kind: trading carries on without you, and every trade that clears while you’re not listening has to be found and confirmed by hand the next day. At the open, that’s hundreds of trades per second. A few weeks earlier, we’d routed more traffic through this session. The service itself was unremarkable. Python 3.11, QuickFIX — C++ under the hood — handling the sessions, Postgres storing the trades. One pod, one CPU, nothing fancy. And the dashboard said everything was fine.

It ran as a single Python process. One connection per venue, each feeding FIX messages in on its own ingest thread, which buffered them and wrote them to Postgres. A batch processor read those messages back out and turned them into trades. That was the whole system: ingest threads accumulating messages, a processor draining them into trades, a database in between. It had run this way for a long time, quietly, without anyone needing to think about it.

The database is the reflex suspect, so I went there. I pulled the save-time histogram: 0–500ms, three-quarters under 100ms. It looked healthy — not pristine, but healthy. I poked the indexes, found nothing damning, ran a couple of faster-insert tricks (UNNEST won the local benchmark and changed nothing real), and moved on. The histogram already contained the answer. Batches arrived every ~330 ms and three-quarters of saves cleared in under 100 ms — the occasional 500 ms tail was debt the next couple of saves repaid. I didn’t do that arithmetic at the time — I just saw a green-ish chart and crossed the database off. That’s the habit worth naming, because I’m about to lean on it again: trust the dashboard, move on to the next suspect. Here it cost nothing — the database really was innocent. It didn’t stay free.

The next cost that grows with message volume is parsing. Each message needs to be parsed — but parsing is core to QuickFIX, it runs in the C++ extension, and it’s been polished by dozens of systems carrying far heavier loads than ours. The microbenchmarks said the same: parse 4.2µs, toString 3µs, prod-formatted log 21.6µs, call it 32µs per message with headroom. A heavy market open runs under 100,000 messages, so even at that ceiling the entire per-message compute budget for an open is around 3.2 seconds of CPU. The messages were being rejected as older than 120 seconds — QuickFIX’s default cutoff — so the backlog was at least two minutes deep. Compute fell short of the symptom by a factor of nearly forty — per-message work was not the constraint. The save tails told a different story. At market open they clustered around multiples of 100ms — 103.99, 211.37, 306.12, 325.97, 334.94ms against a 30–80ms median — the textbook CFS-throttling signature, 100ms being the scheduler period. CPU demand at open was running at about 2× the one-core allocation. So I raised the limit from one core to two and a half, on the same day.

The next day was clean. For two more days it looked solved. The following Monday, the FIX session disconnected at market open. The bump had bought three days, not a fix. And the tell was in the gap: the saturation alarm had gone quiet — CPU utilisation looked fine now — but the lag came back anyway. There was headroom, and it didn’t help. Whatever the ceiling was, it wasn’t compute.

The next suspect was logging. Every incoming FIX message was being logged — about 70k of the ~100k messages in a ten-minute open. My prior was that logging couldn’t be it — the same battle-tested pipe every system here has used for years. You don’t expect to be the one who breaks it. A local reproduction changed that: under a deliberately slow sink, lag grew with volume. Sync writes to a full pipe block, and an open is precisely when volume spikes. At one point that pushed my confidence to near 90% — high enough to warrant a definitive test, not a hunch. So I stripped it back: FIX logging off, most other levels downgraded. Logged volume fell ~20x, to ~5k total in the same window. The next open lagged just the same. Twenty times less logging, identical result — not logging.

Eventually I stopped looking for causes and built the metric that actually mattered: the time between a message leaving the venue (tag 52, its sending timestamp) and the moment QuickFIX hands the message to our application code — the fromApp callback. I called it lag_ms, and it isn’t an arbitrary metric — it’s the exact quantity QuickFIX checks when deciding a session has gone stale. The meter ran at the next market open:

30:38.095 count=100 avg=0.73ms max=9.60ms   lag_max=12092.26ms
30:45.775 count=100 avg=3.09ms max=245.95ms lag_max=16775.23ms
30:57.269 count=100 avg=0.61ms max=11.06ms  lag_max=25269.37ms
31:03.749 count=100 avg=3.05ms max=246.71ms lag_max=30047.58ms

Look at the accounting. Across these 26 seconds we processed ~400 messages at single-digit-millisecond averages — less than a second of measurable work, even counting the ugly 245ms outliers. Meanwhile the lag grew by eighteen seconds. Over 95% of the elapsed time had no explanation: not in the database, not in the handlers, not in parsing. And CPU utilisation sat at 0.4 of its 2.5-core limit — there was idle compute everywhere.

Fast work, spare CPU, and a queue growing by the second. The thread wasn’t slow at its job. It was being kept from its job — waiting on something none of my work-rate meters could see.

Every subsystem looked innocent in isolation. Either every measurement was lying, or the bottleneck existed between them.

This particular app serves a few different FIX connections, and most of the application initialisation involves setting their configurations. The long list of FIX engine initialisers ended with… FixMessageProcessor.

The FixMessageProcessor is designed to be independent of any FIX connections. It reads saved messages from the database and processes them. It’s similar to the outbox pattern. It could even be a different program with shared access to the database, but here it sits in the same process with other connections.

A single process means a single interpreter lock — the GIL; the ingest thread must reacquire it for every message it touches; a peer that’s constantly crunching wins those races here; the queue ages while the CPU idles. 2.5 cores with 0.4 utilisation, so the process is stuck around 1 core of usage. The most a single Python interpreter can ever use. The pod had spare CPU; the interpreter didn’t.

For the first time I had a hypothesis that explained both the idle CPU and the growing queue. Even if the problem wasn’t resolved, moving it out of the process would be a good architectural step. The change was trivial: swap threading.Thread for mp.Process. Another hypothesis was ready to test. I blocked out the next market open and waited.

30:30.536 count=100 avg=0.22ms max=0.54ms lag_max=1089.59ms
30:31.040 count=100 avg=0.21ms max=0.43ms lag_max=1040.26ms
30:31.679 count=100 avg=0.20ms max=0.37ms lag_max=1087.57ms
30:32.254 count=100 avg=0.20ms max=0.36ms lag_max=1087.30ms

Here we are: the lag dropped from almost two minutes to the floor of the meter — tag 52 carries whole seconds, so the max can’t settle below ~1s. The unbounded climb (the hallmark of arrival > drain) has disappeared — ingestion now keeps pace with the venue at the open. And here’s the part worth dwelling on — one extra process in the same pod, and CPU barely moved. Same messages, same compute. What collapsed was the lag, by well over an order of magnitude. The bottleneck was never the CPU. It was the wait.

It looked great, but this wasn’t the first time a change has led to short-term success.

Another Monday, 09:30 NY time, market open, I had the dashboard up, refreshing the page every few seconds. First reading: around 1s. That had happened before on a slow open, so I waited. One minute, still ~1s. Three minutes, seven minutes — processed-message count climbing, max lag flat. A win.

Production max lag at the open (log scale): thread mode before the fix, process mode after. The red climb flattens at the engine's 120s MaxLatency — anything older is rejected ("SendingTime accuracy problem"), so the meter only sees what survives. Sustained rejections end in logout after ~5 minutes; on reconnect the venue replays its buffered backlog and the lag jumps straight back, draining only once the open flow ends. Blue holds at the meter's ~1s floor throughout.

Production was fixed, but “fixed” and “understood” are different states — the GIL was still a hypothesis, and prod would never let me prove it. At this point the story splits. Operationally, the incident was over. Technically, it was just becoming interesting.

Every layer I could measure had been acquitted, and yet two threads sharing a process had cost me two minutes of lag. In production that was where the road ended: the app handles sensitive data, and attaching py-spy to it was not a conversation I was going to win. So the next step was to rebuild the failure locally.

Here is a model with a few knobs — arrival rate (RATE), the size of the processor’s work (HOLD_SIZE), the number of holds per cycle (HOLDS_PER_CYCLE) — turning those knobs reproduces the same behaviour and the same failure. An ingest thread blocks on recv for each message and records now - sent_at — the same shape as the QuickFIX callback loop: one interpreter re-entry per message. A processor thread does what the real one did — a socket round-trip, then building and json.dumps-ing a large structure to reproduce the execution profile. A paced producer and a DB stub run in separate processes, so neither can be victim or culprit. The only thing ingest and processor share is the interpreter.

The script uses Python 3.11; on 3.12 the runaway is shallower but still unbounded at this rate. It takes one argument:

thread runs the processor as a daemon thread — production before the fix;
process runs the identical function in its own process — production after.

import json, socket, struct, sys, threading, time
import multiprocessing as mp

DB_ADDR = ("127.0.0.1", 5001)
FEED_ADDR = ("127.0.0.1", 5002)
MSG = struct.Struct("d")

# knobs
RATE = 600
HOLD_SIZE = 200_000
HOLDS_PER_CYCLE = 200

def db_stub():
    srv = socket.socket()
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(DB_ADDR)
    srv.listen()

    def serve(conn):
        with conn:
            while conn.recv(1):
                time.sleep(0.002)
                conn.sendall(b"\x01")

    while True:
        c, _ = srv.accept()
        threading.Thread(target=serve, args=(c,), daemon=True).start()

def producer():
    time.sleep(0.5)
    c = socket.create_connection(FEED_ADDR)
    nxt = time.monotonic()
    while True:
        c.sendall(MSG.pack(time.monotonic()))
        nxt += 1.0 / RATE
        d = nxt - time.monotonic()
        if d > 0:
            time.sleep(d)

def processor():
    c = socket.create_connection(DB_ADDR)
    while True:
        c.sendall(b"\x01")
        c.recv(1)
        for _ in range(HOLDS_PER_CYCLE):
            json.dumps([{"id": i, "v": f"x{i}"} for i in range(HOLD_SIZE)])

def recv_exact(conn, n):
    buff = b""
    while len(buff) < n:
        chunk = conn.recv(n - len(buff))
        if not chunk:
            raise ConnectionError
        buff += chunk
    return buff

def ingest():
    srv = socket.socket()
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(FEED_ADDR)
    srv.listen()
    conn, _ = srv.accept()
    lags = []
    while True:
        (sent, ) = MSG.unpack(recv_exact(conn, MSG.size))
        lags.append(time.monotonic() - sent)
        if len(lags) == 100:
            print(
                f"lag_avg={sum(lags) / 100 * 1000:8.1f}",
                f"lag_max={max(lags) * 1000:8.1f}"
            )
            lags = []

if __name__ == '__main__':
    mode = sys.argv[1] if len(sys.argv) > 1 else "thread"
    mp.Process(target=db_stub, daemon=True).start()
    mp.Process(target=producer, daemon=True).start()
    time.sleep(0.2)
    if mode == "thread":
        threading.Thread(target=processor, daemon=True).start()
    elif mode == "process":
        mp.Process(target=processor, daemon=True).start()
    else:
        raise SystemExit(f"unknown mode: {mode}")
    ingest()

Output:

# thread
lag_avg=    90.9 lag_max=   199.4
lag_avg=   251.7 lag_max=   394.4
lag_avg=   526.5 lag_max=   641.5
lag_avg=   728.7 lag_max=   856.2
lag_avg=   916.7 lag_max=   977.5
lag_avg=  1011.9 lag_max=  1086.0

... 6 mins later

lag_avg=166577.3 lag_max=166726.6
lag_avg=166927.3 lag_max=167113.9
lag_avg=167260.8 lag_max=167445.3
lag_avg=167591.0 lag_max=167785.0
lag_avg=167930.0 lag_max=168045.9

# process
lag_avg=     0.1 lag_max=     0.1
lag_avg=     0.1 lag_max=     0.1
lag_avg=     0.1 lag_max=     0.2

... 6 mins later

lag_avg=     0.1 lag_max=     0.1
lag_avg=     0.1 lag_max=     0.1
lag_avg=     0.1 lag_max=     0.1

Notice the thread run sails past 168 seconds. Production never showed a number like that: at 120s QuickFIX starts rejecting anything older; the flat-top on the production chart is that reject ceiling, and the cliff after it is the logout. The repro has nobody to disconnect it — this is the first look at the runaway uncapped.

Two threads, one interpreter, one lock. Only the holder executes Python bytecode; the other waits. And they want opposite things from it. Ingest needs the lock briefly but constantly — a few microseconds per message to read a timestamp, hundreds of times a second; its recv calls release the GIL during the syscall and reacquire it to hand the bytes back to Python. The processor takes the lock in bulk and lets go rarely. So while the processor holds it, ingest starves: the syscalls finish quickly, but the results can’t get back into Python until the holder yields — messages age in the buffer while the CPU sits idle.

So what decides when the lock changes hands? switchinterval (default 0.005s): how long a waiting thread lets the holder run before asking it to step aside. The holder obliges at its next bytecode boundary — which means how long a thread actually keeps the GIL depends on its boundaries, not on the interval.

That’s why the processor’s two phases behave differently. Building the list is pure-Python bytecode: it hits a boundary every few instructions and yields promptly once asked. Each json.dumps call is a single C call with no boundaries inside it (Python 3.11, no indent) — it holds the GIL start to finish, and the handoff stalls until it returns. The comprehension is the cooperative chunk; the dumps are the unyielding chunks that produce the spikes.

Lowering switchinterval makes a useful probe. The dumps stay uninterruptible at any interval. The cooperative phases, though, can be interrupted ten times as often at a tenth of the interval — and each interruption costs the listcomp only the microseconds ingest actually spends. At 0.0005s:

lag_avg=    30.7 lag_max=    82.6
lag_avg=    36.9 lag_max=    82.7
lag_avg=    44.6 lag_max=    81.8

... 6 mins later

lag_avg=    46.9 lag_max=    85.9
lag_avg=    33.2 lag_max=    82.0
lag_avg=    44.6 lag_max=    86.9

The lag still spikes to roughly one json.dumps freeze (~72ms at this HOLD_SIZE, measured standalone) — but it drains in the next cooperative window instead of compounding. Flat instead of climbing. The probe has split the lag in two: the part that responded to the interval was time spent waiting to reacquire; the part that didn’t move is the uninterruptible hold.

schematic, not to scale, each `|` is a handoff to ingest
interval 5ms:    [listcomp....]|[xxx dumps xxx][listcomp....]|[xxx dumps xxx]
interval 0.5ms:  [lc]|[lc]|[lc]|[lc]|[xxx dumps xxx][lc]|[lc]|[lc]|[lc]|

It’s a probe, not a fix. A shorter interval means the GIL changes hands more often, and every handoff costs something — lowering it globally just buys overhead and slows genuinely CPU-bound work. What it bought here is a diagnosis: one lever, and the lag decomposed into reacquisition and hold. Process isolation — the change already running in production — kills both.

Takeaways

The cost was waiting, not working. A co-resident thread with a heavier payload didn’t out-compute the ingest threads — it out-held them. And the time they spent waiting for the GIL appeared on no meter, because work-rate meters count work done and waiting produces none. That’s not a gap in the tooling; it’s what those meters are, by construction.

Two modes, one sentence: at bytecode boundaries the victim pays per handoff; inside a C-extension hold the perpetrator extorts per hold. Separation kills both — which is why isolating the threads is the fix, not a tuning exercise.

Operationally: measure end-to-end age. Component throughput tells you each part is moving; only age tells you the system is keeping up.

Do I still trust the dashboards? Yes — to answer the questions they measure. I’ve just stopped asking them the one they can’t.

Fast PostgreSQL Data Generation

Thu, 07 Nov 2024 22:18:11 +0000

Improving query performance is always challenging, especially when working with large-scale databases. While adjusting indexes is a common approach, testing these optimizations becomes complex when your tables contain terabytes of data. I recently faced this challenge: I needed to test potential improvements for a large-scale table, but there was no simple way to work with terabytes of data on my laptop. The solution? Generating a representative dataset that mimics our production database structure and patterns.

In this article, I’ll explore several approaches to generating and inserting large volumes of test data into PostgreSQL on a local machine. While many of these techniques can be applied to production environments and other databases, we’ll focus specifically on local development with PostgreSQL.

Note: All benchmarks in this article were run on a modest laptop - you’ll likely see much better performance on modern hardware. The focus here is on relative improvements between different approaches rather than absolute numbers.

Table structure

We’ll work with a simple time-series-like table containing three fields, no indexes at the moment to keep it simple:

CREATE TABLE metrics (
    created     timestamp with time zone default now() NOT NULL,
    sensor_id   integer                                NOT NULL,
    temperature numeric                                NOT NULL
);

Basic case

The most straightforward approach is to leverage PostgreSQL’s built-in capabilities for data generation. Let’s start with a simple SQL script that generates 10 million rows:

INSERT INTO metrics (
    SELECT
        (NOW() + seq / 100 * interval '1 MILLISECOND') AS created,
        (seq % 32 + 1)::integer AS sensor_id,
        20.0 + RANDOM() * 500 / 100 as temperature
    FROM GENERATE_SERIES(1, 10_000_000) seq
);

This script:

Creates timestamps at 10μs intervals starting from now
Simulates 32 different sensors (IDs 1-32)
Generates temperatures between 20.0°C and 25.0°C

On my local machine this query takes 26.440s which is quite fast for 10 million rows (~0.5GB of data). This is probably the best case, since it eliminates overhead from connection handling and data type conversions - PostgreSQL handles everything internally.

Generating actual data

In our production database, sensor readings arrive in batches: multiple sensors record values at the exact same timestamp. This pattern has important implications for database performance, particularly for indexing and query optimization. Let’s implement a Rust generator that mimics this behavior:

const BATCH_SIZE: usize = 1_000; // Number of sensor readings per batch
const MAX_SENSORS: i32 = 32; // Total number of unique sensors

type Row = (DateTime<Utc>, i32, f64);

fn generate_batch(created: DateTime<Utc>, sensor_id: i32, base_temp: f64) -> (Vec<Row>, i32) {
    let mut rng = rand::thread_rng();
    let mut current_sensor_id = sensor_id;
    let batch: Vec<_> = (0..BATCH_SIZE)
        .map(|i| {
            current_sensor_id = (current_sensor_id + (i as i32)) % MAX_SENSORS + 1;
            let temperature = ((base_temp + rng.gen_range(-5.0..5.0)) * 100.0).round() / 100.0;
            (created, current_sensor_id, temperature)
        })
        .collect();
    (batch, sensor_id)
}

fn generate_data(
    start_time: DateTime<Utc>,
    base_temp: f64,
    batch_count: usize,
) -> impl Iterator<Item = (Vec<Row>, i64)> {
    let mut current_time = start_time;
    let mut sensor_id = 1;
    let mut current_tick = 0;

    (0..batch_count).flat_map(move |_| {
        current_tick += 1;
        current_time += Duration::milliseconds(100);
        let (new_batch, new_sensor_id) = generate_batch(current_time, sensor_id, base_temp);
        sensor_id = new_sensor_id;

        std::iter::once((new_batch, current_tick))
    })
}

This generator creates data that closely mirrors our production patterns:

Each batch generates 1,000 readings sharing the same timestamp
Multiple readings from each sensor (ID range 1-32) within each batch
Batches are generated at 100ms intervals
Temperatures vary randomly within ±5°C of the base temperature
Each reading is rounded to 2 decimal places

While this is a simplified version of our production data patterns, it captures the key characteristics that affect query performance: time clustering, sensor ID distribution, and value ranges.

Measuring performance

To effectively compare different data insertion approaches, we need consistent performance metrics. Let’s create a helper struct that measures execution time, data size, and throughput using RAII (Resource Acquisition Is Initialization) pattern:

struct ExecutionContext {
    t0: DateTime<Utc>,
    s0: i64,
    client: Client,
    name: String,
}

impl ExecutionContext {
    fn new(name: &str, conn_info: &str) -> Self {
        let mut client = Client::connect(conn_info, NoTls).unwrap();
        let s0 = Self::table_size(&mut client);
        let t0 = Utc::now();
        let name = name.to_string();

        ExecutionContext { t0, s0, client, name }
    }

    fn table_size(client: &mut Client) -> i64 {
        let row = client
            .query_one("SELECT pg_total_relation_size('public.metrics') as size", &[])
            .unwrap();
        row.get("size")
    }

    fn convert_bytes(bytes: f64, to: &str) -> f64 {
        let units = ["B", "KB", "MB", "GB", "TB", "PB"];
        let index = units
            .iter()
            .position(|&r| r == to.to_uppercase())
            .unwrap_or(0);
        bytes / (1024f64.powi(index as i32))
    }
}

impl Drop for ExecutionContext {
    fn drop(&mut self) {
        let s1 = Self::table_size(&mut self.client);
        let t1 = Utc::now();

        let duration = (t1 - self.t0).num_seconds();
        let size = s1 - self.s0;
        let speed = (size as f64) / (duration as f64);

        println!("\n{}:", self.name);
        println!("Speed: {:.2}MB/s", Self::convert_bytes(speed, "MB"));
        println!(" Data: {:.2}MB", Self::convert_bytes(size as f64, "MB"));
        println!(" Time: {:.2}s", duration);
    }
}

This performance measuring tool:

Tracks the start time and initial table size when created
Maintains a database connection throughout the test
Automatically calculates and prints performance metrics when dropped

Here’s how to use it:

{
    let _context = ExecutionContext::new("Simple Insert Test", conn_info);
    let stmt = "insert into metrics values (NOW(), (RANDOM() * 20)::integer, 20 + RANDOM())";
    for _ in 0..1_000 {
        client.execute(stmt, &[]).unwrap();
    }
}

// Output:
// Simple Insert Test:
// Speed: 0.01MB/s
// Data: 0.06MB
// Time: 5s

This tool will help us compare the performance of different data insertion strategies in the following sections.

Simple insert

Let’s start with the most straightforward approach: inserting records one by one within batch transactions. While this isn’t the most efficient method, it provides a baseline for our performance improvements.

fn f64_to_decimal(value: f64) -> Decimal {
    Decimal::from_str(&value.to_string()).unwrap_or_else(|_| Decimal::new(0, 0))
}

fn insert_to_postgres(
    client: &mut Client,
    table_name: &str,
    batch_data: &[Row],
    current_tick: i64,
) {
    let mut tx = client.transaction().unwrap();
    let stmt = tx
        .prepare(&format!("INSERT INTO {} VALUES ($1, $2, $3)", table_name))
        .unwrap();

    for row in batch_data {
        // PostgreSQL's decimal type requires explicit conversion from f64
        let params: [&(dyn ToSql + Sync); 3] = [&row.0, &row.1, &f64_to_decimal(row.2)];
        tx.execute(&stmt, &params).unwrap();
    }

    tx.commit().unwrap();

    if current_tick % REPORT_COUNT == 0 {
        println!("Copied {current_tick}");
    }
}

Running this with batches of 10,000 records each:

fn insert:
Speed: 0.89MB/s
 Data: 497.76MB
 Time: 560s

Compared to our initial pure SQL approach (26 seconds), this method is over 20 times slower. The performance gap comes primarily from network roundtrips - each insert requires:

Sending the execute command to PostgreSQL
PostgreSQL processing it
Getting the acknowledgment back

In the next sections, we’ll explore how to optimize these bottlenecks.

Long string insert

While network latency is typically low on local connections, the overhead of making thousands of separate requests adds up significantly. Instead, we can batch our inserts into a single SQL statement. This approach:

Combines all values into one INSERT command
Lets PostgreSQL handle data type conversions
Minimizes network roundtrips

fn insert_to_postgres_string(
    client: &mut Client,
    table_name: &str,
    batch_data: &[Row],
    current_tick: i64,
) {
    let tuples = batch_data
        .into_iter()
        .map(|row| {
            format!(
                "('{}'::timestamp with time zone, {}, {}::numeric(10, 2))",
                row.0, row.1, row.2
            )
        })
        .collect::<Vec<_>>()
        .join(",");
    let query = format!("INSERT INTO {} VALUES {}", table_name, tuples);
    client.execute(&query, &[]).unwrap();

    if current_tick % REPORT_COUNT == 0 {
        println!("Copied {current_tick}");
    }
}

This generates a query that looks like:

INSERT INTO metrics VALUES
('2024-01-01 10:00:00+00'::timestamp with time zone, 1, 20.5::numeric(10, 2)),
('2024-01-01 10:00:00+00'::timestamp with time zone, 2, 21.3::numeric(10, 2)),
...

Rather than sending 10,000 separate INSERT commands, we send one large query. While the resulting SQL isn’t as readable as individual INSERT statements, it’s much more efficient to transmit and execute. Let’s check how it performs:

fn insert-str:
Speed: 4.48MB/s
 Data: 497.75MB
 Time: 111s

By eliminating multiple requests and sending just one, we achieved a 5x speed improvement - quite impressive for such a straightforward change.

Binary COPY Protocol

Our string-based approach still has overhead: generating SQL strings and letting PostgreSQL parse them. Let’s go further and speak the native language of the database: binary data. PostgreSQL provides this capability through the COPY command.

From PostgreSQL documentation:

COPY FROM copies data from a file to a table (appending the data to whatever is in the table already)

The binary format option is particularly interesting:

The binary format option causes all data to be stored/read as binary format rather than as text. It is somewhat faster than the text and CSV formats, but a binary-format file is less portable across machine architectures and PostgreSQL versions.

Why is this relevant to us? We generate test data locally, so portability isn’t an issue. Instead, we can focus purely on performance by

Eliminating the overhead of string formatting
Reducing the amount of data transferred
Allowing PostgreSQL to skip the SQL parsing step

The binary format protocol requires careful handling of different data types:

timestamp with time zone
integer
numeric (decimal)

Rust makes this task easy - it’s both convenient for handling structured data and efficient at manipulating raw bytes. Let’s implement the binary encoding of each type, from the simplest to the most complex.

integer

Converting integers is really simple since both Rust’s i32 and PostgreSQL’s integer use the same representation. All we need to do is to say how long (pun intended, as PostgreSQL also has a long type) the value is and write it as bytes:

// sensor_id
buffer.write_i32::<BigEndian>(4)?;
buffer.write_i32::<BigEndian>(row.1)?;

timestamp with time zone

This one is more complex. How should we represent a timestamp? ISO format? Seconds since UNIX epoch?

PostgreSQL actually uses its own epoch (2000-01-01) and counts microseconds from there. Here’s how we handle it:

static POSTGRES_EPOCH: Lazy<DateTime<Utc>> = Lazy::new(|| {
    NaiveDate::from_ymd_opt(2000, 1, 1)
        .unwrap()
        .and_hms_opt(0, 0, 0)
        .unwrap()
        .and_utc()
});

fn datetime_to_postgres_binary(datetime: DateTime<Utc>) -> i64 {
    let time_delta = datetime - POSTGRES_EPOCH.to_utc();
    time_delta.num_microseconds().unwrap()
}

// created
let micros = datetime_to_postgres_binary(row.0);
buffer.write_i32::<BigEndian>(8)?;
buffer.write_i64::<BigEndian>(micros)?;

decimal

This is the most challenging part of our binary implementation. While PostgreSQL’s binary format can be very efficient, some data types (like numeric/decimal) lack clear documentation about their binary representation. This reflects a common challenge in database development - sometimes you need to dig deeper than the official docs.

To figure out the correct implementation, we need to:

Read the PostgreSQL source code: numeric.c
Look at existing implementations in other languages, like this Python example

From these sources, we can determine that each numeric value consists of:

ndigits: number of digits in base NBASE (which is 1000)
weight: weight of the first digit
sign: 0x0000 for positive, 0x4000 for negative
dscale: number of decimal digits after the decimal point

Here’s our implementation:

fn numeric_to_postgres_binary(value: f64) -> Vec<u8> {
    let mut buffer = Vec::new();
    let abs_value = value.abs();
    let sign = if value.is_sign_negative() {
        0x4000
    } else {
        0x0000
    };
    let parts: Vec<String> = abs_value.to_string().split('.').map(String::from).collect();

    let integer = &parts[0];
    let fraction = if parts.len() > 1 {
        parts[1].clone()
    } else {
        String::new()
    };

    let weight = ((integer.len() as i16 - 1) / 4) as i16;
    let padding = if integer.len() % 4 != 0 {
        4 - (integer.len() % 4)
    } else {
        0
    };
    let padded_number = format!(
        "{:0>width$}{}",
        integer,
        fraction,
        width = integer.len() + padding
    );

    let digits: Vec<i16> = padded_number
        .as_bytes()
        .chunks(4)
        .map(|c| {
            let mut value = 0;
            for (i, &digit) in c.iter().enumerate() {
                value += (digit - b'0') as i16 * 10i16.pow(3 - i as u32);
            }
            value as i16
        })
        .collect();
    let ndigits = digits.len() as i16;
    let dscale = fraction.len() as i16;

    buffer.write_i16::<BigEndian>(ndigits).unwrap();
    buffer.write_i16::<BigEndian>(weight).unwrap();
    buffer.write_i16::<BigEndian>(sign).unwrap();
    buffer.write_i16::<BigEndian>(dscale).unwrap();

    for digit in digits {
        buffer.write_i16::<BigEndian>(digit).unwrap();
    }

    buffer
}

// temperature
let numeric_bytes = numeric_to_postgres_binary(row.2);
buffer.write_i32::<BigEndian>(numeric_bytes.len() as i32)?;
buffer.write_all(&numeric_bytes)?;

This implementation handles our decimal temperature values by converting them into PostgreSQL’s internal numeric representation. The process is more complex than for integers or timestamps, but the resulting binary format is exactly what PostgreSQL expects.

Putting It All Together

Now let’s combine everything into a working solution. We’ll use Rust’s Cursor to build an in-memory buffer that follows PostgreSQL’s binary COPY protocol requirements:

A magic header to identify the format
Number of fields per row (3 in our case)
The actual data
A termination marker

fn generate_buffer(batch_data: &[Row]) -> anyhow::Result<Vec<u8>> {
    let mut buffer = Cursor::new(Vec::new());
    buffer.write_all(b"PGCOPY\n\xff\r\n\0")?;
    buffer.write_i32::<BigEndian>(0)?;
    buffer.write_i32::<BigEndian>(0)?;

    for row in batch_data {
        buffer.write_i16::<BigEndian>(3)?;

        // created
        let micros = datetime_to_postgres_binary(row.0);
        buffer.write_i32::<BigEndian>(8)?;
        buffer.write_i64::<BigEndian>(micros)?;

        // sensor_id
        buffer.write_i32::<BigEndian>(4)?;
        buffer.write_i32::<BigEndian>(row.1)?;

        // temperature
        let numeric_bytes = numeric_to_postgres_binary(row.2);
        buffer.write_i32::<BigEndian>(numeric_bytes.len() as i32)?;
        buffer.write_all(&numeric_bytes)?;
    }

    buffer.write_i16::<BigEndian>(-1)?;
    Ok(buffer.into_inner())
}

fn copy_to_postgres(client: &mut Client, table_name: &str, batch_data: &[Row], current_tick: i64) {
    let buffer = generate_buffer(batch_data).unwrap();
    let mut writer = client
        .copy_in(&format!("COPY {} FROM STDIN WITH BINARY", table_name))
        .unwrap();
    writer.write_all(&buffer).unwrap();
    writer.finish().unwrap();

    if current_tick % REPORT_COUNT == 0 {
        println!("Copied {current_tick}");
    }
}

Yes, this implementation is more complex than our previous approaches. But did the extra complexity pay off in terms of performance? Let’s find out in the next section.

Results

Let’s see if our binary implementation was worth the effort:

fn copy:
Speed: 24.89MB/s
 Data: 497.73MB
 Time: 20s

The results are impressive - 20 seconds! That’s even faster than our initial raw SQL approach. Let’s look at how our different approaches performed with the same data volume:

Simple inserts: 560s
String concatenation: 111s
Raw SQL: 26s
Binary COPY: 20s

Each optimization step brought significant improvements, with our final binary implementation performing 28x faster than individual inserts while retaining all the flexibility of a high-level programming language for generating complex data patterns.

Conclusion

Our journey from simple inserts to binary COPY protocol shows that while readable solutions are great, sometimes you need to dive into lower-level details for maximum performance. The final binary implementation may look complex, but the 28x speedup makes it worthwhile when testing databases at scale.

All code from this article is available in repository link.

Visualizing Trees and ASTs with Graphviz

Wed, 02 Aug 2017 19:09:22 +0000

Visualization is a great opportunity to see some complex structures from another side. Let’s assume we want to inspect how trees actually look. We will use Breadth-first search (BFS) to traverse graph (or tree in our case) and show connections between nodes.

def bfs(node)
  queue = [node]
  discovered = []
  while queue.length > 0 do
    current = queue.shift
    discovered << current
    current.children.each do |child|
      queue << child if child
    end
  end
  discovered
end

The awesome fact about BFS is that it traverse nodes from left to right, so, we will traverse tree level-by-level. Let’s assume we already have tree class. We start from filling a tree:

tree = Tree.new
[8, 4, 12, 2, 6, 10, 14, 1, 3, 5, 7, 9, 11, 13, 15].each do |value|
  tree.insert(value)
end

And now we will traverse all tree nodes and render it with left and right child:

([["Node", "Left", "Right"]] + bfs(tree.root).map do |node|
  [node.value, node.left&.value || "NULL", node.right&.value || "NULL"]
end).each { |n| puts n.join(" ") }

This code produces the following table (additionally I piped it through column -t):

Node  Left  Right
   4     12
   2     6
  10    14
   1     3
   5     7
  9     11
  13    15
   NULL  NULL
   NULL  NULL
   NULL  NULL
   NULL  NULL
   NULL  NULL
  NULL  NULL
  NULL  NULL
  NULL  NULL

It’s time to draw a tree! We will use graphviz for rendering.

According to Graphviz documentation we need to simply describe edges in proper format and graphviz will render the graph. Sounds easy. We will generate .dot file and then render it. To reduce complexity of manipulating with files we will write output to stdout and then pipe it into the graphviz tool. It will look like:

ruby tree.rb | dot | neato -n -Tpng -o tree.png

Additionally, to decrease feedback loop (in case if you are using iTerm), you could use build-in command imgcat, so it will look like:

ruby tree.rb | dot | neato -n -Tpng | imgcat

This code will generate our first node graph:

def render_tree(tree)
  result = []
  result << "digraph graphname {"  # graph type
  result << "node [shape=circle];" # we want pretty circles as nodes

  bfs(tree).each do |node|
    node.children.each do |child|
      next if child.nil?
      result << [node.value, child.value].join(" -> ")
    end
  end

  result << "}"
  result.join("\n")
end

It will generate Graphviz file with the following content:

digraph graphname {
node [shape=circle];
-> 4
-> 12
-> 2
-> 6
-> 10
-> 14
-> 1
-> 3
-> 5
-> 7
-> 9
-> 11
-> 13
-> 15
}

But there are some problems. Let’s assume we have more than one node with the same value ([5, 3, 7, 3, 7, 5]):

Hmm…, its not what we expect. We need more unique identificator and Ruby provides object_id method, which we can use to get unique identifier for each node.

Ok, it’s better, but still not so good, we solved one problem but there is another one. Graphviz allows you to define how to render nodes. In order to do this, you need to append a line with label’s contents.

New code:

def render_tree(tree)
  result = []
  result << "digraph graphname {"
  result << "node [shape=circle];"

  bfs(tree).each do |node|
    result << "#{node.object_id} [label=#{node.value}]"
    node.children.each do |child|
      next if child.nil?
      result << [node.object_id, child.object_id].join(" -> ")
    end
  end

  result << "}"
  result.join("\n")
end

This is file generated by our method:

digraph graphname {
node [shape=circle];
70120168878200 [label=5]
70120168878200 -> 70120168878140
70120168878200 -> 70120168878120
70120168878140 [label=3]
70120168878140 -> 70120168878100
70120168878120 [label=7]
70120168878120 -> 70120168878000
70120168878120 -> 70120168878020
70120168878100 [label=3]
70120168878000 [label=5]
70120168878020 [label=7]
}

Result is:

Ok, nice, but it’s not actually a binary tree. There is a nice answer on StackOverlow which gives you idea how to do render this tree as binary tree: https://stackoverflow.com/a/10905657.

Yes! So nice, actually what we want. But…

Hey! Node with value 6 is on the right of node with value 7 — is something wrong with the tree insertion code?

Node  Left  Right
   3     7
   NULL  4
   6     NULL
   NULL  NULL
   NULL  NULL

Seems like not. We need to try something else and there is question on Graphviz site which should help us: http://www.graphviz.org/content/FaqBalanceTree. According to that we need to add invisible nodes to our tree. Additionally we need to add invisible nodes for nodes with just one leaf. To implement it we introduce new class InvisibleNode which we inherit from Node class.

class TreeRenderer
  InvisibleNode = Class.new(Node)

  def initialize(tree)
    @tree = tree
  end

  def call
    result = []
    result << "digraph graphname {"
    result << "node [shape=circle];"

    bfs(tree).each do |node|
      result << ["#{node.object_id}", %{[label=#{node.value} fixedsize=true]}].join(" ")
      fix_missing_children(node).each do |child|
        case child
          when InvisibleNode
            result << [child.object_id, %{[label="" width=.1 style=invis]}].join(" ")
            result << [[node.object_id, child.object_id].join(" -> "), '[style=invis]'].join(" ")
          when Node
            result << [[node.object_id, child.object_id].join(" -> ")].join
        end
      end
    end

    result << "}"
    result.join("\n")
  end

  private
  attr_reader :tree

  def fix_missing_children(node)
    node
      .children
      .map { |child| child || generate_fake_child(node) }
      .insert(1, generate_fake_child(node))
  end

  def generate_fake_child(node)
    InvisibleNode.new.tap { |child| child.parent = node }
  end
end

It’s not so cool now, because, actually, it’s complex. We could delegate nodes rendering to the Node class, but I think it’s not perfect idea to change actual “business-logic” code for rendering.

Additionally it will be nice to have ability to show NULL values, for debug purposes.

class TreeRenderer
  InvisibleNode = Class.new(Node)
  NilNode = Class.new(Node)

  def initialize(tree, with_nil)
    @tree = tree
    @with_nil = with_nil
  end

  def call
    result = []
    result << "digraph graphname {"
    result << "node [shape=circle];"

    bfs(tree).each do |node|
      result << ["#{node.object_id}", %{[label=#{node.value} fixedsize=true]}].join(" ")
      fix_missing_children(node).each do |child|
        case child
          when InvisibleNode
            result << [child.object_id, %{[label="" width=.1 style=invis]}].join(" ")
            result << [[node.object_id, child.object_id].join(" -> "), '[style=invis]'].join(" ")
          when NilNode
            result << [child.object_id, %{[label="NULL" shape="box" style="#{with_nil ? '' : 'invis'}"]}].join(" ")
            result << [[node.object_id, child.object_id].join(" -> "), %{[style="#{with_nil ? '' : 'invis'}"]}].join(" ")
          when Node
            result << [[node.object_id, child.object_id].join(" -> ")].join
        end
      end
    end

    result << "}"
    result.join("\n")
  end

  private
  attr_reader :tree, :with_nil

  def fix_missing_children(node)
    node
      .children
      .map { |child| child || generate_nil_child(node) }
      .insert(1, generate_fake_child(node))
  end

  def generate_nil_child(node)
    NilNode.new.tap { |child| child.parent = node }
  end

  def generate_fake_child(node)
    InvisibleNode.new.tap { |child| child.parent = node }
  end
end

And the result is:

How can you use it? With minimal changes you can render any binary tree. For example here is RB-tree rendered this way.

I used this code to debug my RB-tree implementation. There are dotted edges which represents child-parent connection. So, I think you can easily find what’s wrong here.

The same renderer works on any tree structure. Let’s apply it to something more practical.

Parsing Ruby ASTs

An AST is just a tree, so the renderer above works with minor adjustments. The Ruby parser gem exposes the AST for any source file. The canonical approach would be the Visitor pattern, but we can keep it simpler:

require 'parser/current'

class ASTRenderer
  def initialize(tree)
    @tree = tree
  end

  def call
    result = []
    result << "digraph graphname {"

    bfs(@tree).each do |node|
      result << ["#{node.object_id}", %Q{[label="#{present(node)}"]}].join(" ")
      node.children.each do |child|
        next unless child.is_a?(Parser::AST::Node)
        result << [[node.object_id, child.object_id].join(" -> ")].join
      end
    end

    result << "}"
    result.join("\n")
  end

  private

  def present(node)
    return node.to_s unless node.is_a?(Parser::AST::Node)
    case node.type
    when :begin  then  "(begin)"
    when :if     then  "(if)"
    when :array  then  "(array)"
    when :send   then  "(send :#{node.to_a[1]})"
    when :lvar   then  "(lvar :#{node.to_a[0]})"
    when :def    then  "(def :#{node.to_a[0]})"
    when :str    then  "(str :#{node.to_a[0]})"
    when :args   then  "(args)"
    when :return then  "(return)"
    else node.to_s
    end
  end

  def bfs(node)
    queue = [node]
    discovered = []
    while queue.length > 0 do
      current = queue.shift
      next unless current.is_a?(Parser::AST::Node)
      discovered << current
      current.children.each do |child|
        queue << child if child
      end
    end
    discovered
  end
end

source_code = <<RUBY
  def factorial(n)
    return 1 if [1, 2].includes?(n)
    factorial(n - 1) + factorial(n - 2)
  end
RUBY
ast = Parser::CurrentRuby.parse(source_code)
puts ASTRenderer.new(ast).call

Result:

Result with comments:

The changes from the binary tree renderer are minimal: no binary tree layout code, non-Parser::AST::Node children are skipped, and node labels come from the AST node type. Even on this small example the full AST is hard to read — there’s too much detail. But we can filter it down.

Let’s look at what a class definition node looks like:

class Foo < Bar
  def body; end
end

s(:class,
  s(:const, nil, :Foo),
  s(:const, nil, :Bar),
  s(:def, :body,
    s(:args), nil))

s is s-expression notation for nested lists. The class node has three arguments: the class name constant, the parent class constant, and the body.

Building a Class Dependency Diagram

Let’s catch class definitions and store them:

class KlassDefinitionsProcessor < Parser::AST::Processor
  attr_reader :klass_definitions

  def initialize(*)
    super
    @klass_definitions = []
  end

  def on_class(node)
    klass_konst, _parrent, _body = *node
    _nested_konst, konst_name = *klass_konst
    @klass_definitions << konst_name
    super
  end
end

source_code = <<RUBY
class Foo; end
class Bar; end
RUBY

ast = Parser::CurrentRuby.parse(source_code)
processor = KlassDefinitionsProcessor.new
processor.process(ast)
p processor.klass_definitions
# => [:Foo, :Bar]

Good start. But namespaced constants like Foo::Bar::Baz get truncated. Fix: use klass_konst.loc.expression.source instead of destructuring:

  def on_class(node)
    klass_konst, _parrent, _body = *node
    @klass_definitions << klass_konst.loc.expression.source
    super
  end

class Foo::Bar::Baz; end
class Oof::Zab; end
# => ["Foo::Bar::Baz", "Oof::Zab"]

Now for the dependency edges. We need to track which constants are referenced inside each class. The problem: standard AST processing is top-down, but we need to know the enclosing class for any given constant. The solution is a nesting stack — push on entry, pop on exit:

  def process(node)
    @nesting.push(node)
    super
    @nesting.pop
  end

Now we can easily access the current node’s parent:

  def parent
    @nesting.last(2).first
  end

and find in which class we are processing current node:

  def context_klass
    @nesting.reverse.select { |x| x.type == :class }.first
  end

Which gives us useful helpers:

  def klass_name_for(node)
    klass_konst, _, _ = *node
    source_for(klass_konst)
  end

  def source_for(node)
    node.loc.expression.source
  end

Wiring it together — collect class definitions and constant references:

  def on_class(node)
    @klass_definitions << klass_name_for(node)
    super
  end

  def on_const(node)
    konst_name = source_for(node)
    current_klass = klass_name_for(context_klass)
    (@klass_dependencies[current_klass] ||= []) << konst_name
    super
  end

class Foo::Bar::Baz; end
class Oof::Zab; def call; Foo::Bar::Baz.new; end; end

# processor.klass_dependencies
# => {
# "Foo::Bar::Baz"=>["Foo::Bar::Baz", "Foo::Bar", "Foo"],
# "Oof::Zab"=>["Oof::Zab", "Oof", "Foo::Bar::Baz", "Foo::Bar", "Foo"]
# }

Too noisy — nested constants trigger on_const for each segment, and the class name itself is included. Skip those cases:

  def on_const(node)
    unless %i[class const].include?(parent.type)
      konst_name = source_for(node)
      if context_klass
        current_klass = klass_name_for(context_klass)
        (@klass_dependencies[current_klass] ||= []) << konst_name
      end
    end
    super
  end

class Foo::Bar::Baz; end
class Oof::Zab; def call; Foo::Bar::Baz.new; end; end
# {"Oof::Zab"=>["Foo::Bar::Baz"]}

Now render it with Graphviz:

class GraphRenderer
  def initialize(relations, klasses)
    @relations = relations
    @klasses = klasses
  end

  def call
    [
      begin_header,
      klasses_info,
      relations_info,
      end_header
    ].flatten.join("\n")
  end

  private

  def begin_header
    [
      %(digraph graphname {),
      %(rankdir="LR")
    ]
  end

  def klasses_info
    @klasses.map do |klass|
      [wrap(klass), %Q{[label="#{klass}"]}].join(" ")
    end
  end

  def relations_info
    @relations.map do |klass, dependents|
      dependents.map do |child|
        [wrap(klass), wrap(child)].join(' -> ')
      end
    end
  end

  def end_header
    [%(})]
  end

  def wrap(text)
    '"' + text + '"'
  end
end

One remaining limitation: Ruby’s namespace resolution means B::C inside module A and A::B::C from outside resolve to the same class, but our processor treats them as different constants. Fully resolving this requires implementing Ruby’s constant lookup rules — left as a future exercise.

Full-text Fuzzy Search with pg_trgm and Triggers

Sat, 05 Dec 2015 19:21:55 +0000

SQL triggers is something that everybody heard a lot about. Before start we need to talk about benefits and burdens.

Pros

It’s very fast. If you are using any kind of adapter you pay for all the wrappers and connectors. With triggers, all operations are very cheap because they are handled by PostgreSQL directly.
It works like magic. There is no code in your repository, no callbacks on your models.

Cons

It works like a magick. You can’t easy determine all operations. All this \df+ stuff and so on.
It’s hard to change. Really if you want change trigger you need to drop it and hang it again. With full trigger code. You can use hair_trigger to simplify it but anyway.
It increases requirements on your developers. They should know SQL. In perfect world your team is awesome. But it’s business.
It will cause some problems in cause if you are using STI because you can’t define model by it’s tablename.
It can cause problems on backups restoring. On restoring large tables somebody can use --disable-triggers to speed up process and it’s not that you are expecting.

If you are still want it let’s continue. You have MobilePhone, Laptop and Ad. Ad contains info about Product (in this case MobilePhone or Laptop). You want to provide some kind of faceted search for your products. We will use extension pg_trgm for searching products with name similar to provided by user.

What we need to implement it?

1) pg_trgm extension.

CREATE EXTENSION IF NOT EXISTS pg_trgm;

2) Table to store all this stuff

CREATE TABLE IF NOT EXISTS product_names (
  product_name  text    NOT NULL,
  product_type  text    NOT NULL,
  product_id    integer NOT NULL
);
CREATE INDEX product_name_gin_index ON product_names
  USING gin(product_name gin_trgm_ops);

3) Function that will process data. Based on this example. It’s pretty simple. We change our table curresponding to actions on product`s tables.

CREATE OR REPLACE FUNCTION collect_product_names() RETURNS TRIGGER AS $body$
BEGIN
  IF (TG_OP = 'INSERT') THEN
      INSERT INTO product_names (product_type, product_id, product_name)
      VALUES (TG_TABLE_NAME::TEXT, NEW.id, NEW.name);

      RETURN NULL;
  ELSIF (TG_OP = 'UPDATE') THEN
      UPDATE product_names
         SET product_name = NEW.name
       WHERE  product_id = OLD.id
         AND product_type = TG_TABLE_NAME::TEXT;

      RETURN NULL;
  ELSIF (TG_OP = 'DELETE') THEN
      DELETE
      FROM   product_names
      WHERE  product_type = TG_TABLE_NAME::TEXT
        AND  product_id = OLD.id;

      RETURN NULL;
  ELSE
      RAISE WARNING '[AUDIT.collect_product_names] - Other action occurred: %, at %',TG_OP,now();
      RETURN NULL;
  END IF;
END;
$body$
LANGUAGE plpgsql
SECURITY DEFINER;

4) Add our function for product`s table

DROP TRIGGER IF EXISTS collect_product_names ON mobile_phones;
DROP TRIGGER IF EXISTS collect_product_names ON laptops;

CREATE TRIGGER collect_product_names
  AFTER INSERT OR DELETE OR UPDATE ON mobile_phones
  FOR EACH ROW EXECUTE PROCEDURE collect_product_names();
CREATE TRIGGER collect_product_names
  AFTER INSERT OR DELETE OR UPDATE ON laptops
  FOR EACH ROW EXECUTE PROCEDURE collect_product_names();

5) Migrate existing data to our new table

[MobilePhone, Laptop].each do |klass|
  klass.pluck(:name, :id).each do |(name, id)|
    ProductName.create!(product_name: name,
      product_id: id, product_type: klass.table_name)
  end
end

All in one

class AddProductNamesTrigger < ActiveRecord::Migration
  def up
    execute %Q{
      CREATE EXTENSION IF NOT EXISTS pg_trgm;

      CREATE TABLE IF NOT EXISTS product_names (
          product_name  text    NOT NULL,
          product_type  text    NOT NULL,
          product_id    integer NOT NULL
      );
      CREATE INDEX product_name_gin_index ON product_names USING gin(product_name gin_trgm_ops);

      CREATE OR REPLACE FUNCTION collect_product_names() RETURNS TRIGGER AS $body$
        BEGIN
          IF (TG_OP = 'INSERT') THEN
              INSERT INTO product_names (product_type, product_id, product_name)
              VALUES (TG_TABLE_NAME::TEXT, NEW.id, NEW.name);

              RETURN NULL;
          ELSIF (TG_OP = 'UPDATE') THEN
              UPDATE product_names
                 SET product_name = NEW.name
               WHERE  product_id = OLD.id
                 AND product_type = TG_TABLE_NAME::TEXT;

              RETURN NULL;
          ELSIF (TG_OP = 'DELETE') THEN
              DELETE
              FROM   product_names
              WHERE  product_type = TG_TABLE_NAME::TEXT
                AND  product_id = OLD.id;

              RETURN NULL;
          ELSE
              RAISE WARNING '[AUDIT.collect_product_names] - Other action occurred: %, at %',TG_OP,now();
              RETURN NULL;
          END IF;

        END;
      $body$
      LANGUAGE plpgsql
      SECURITY DEFINER;

      DROP TRIGGER IF EXISTS collect_product_names ON mobile_phones;
      DROP TRIGGER IF EXISTS collect_product_names ON laptops;

      CREATE TRIGGER collect_product_names
        AFTER INSERT OR DELETE OR UPDATE ON mobile_phones
        FOR EACH ROW EXECUTE PROCEDURE collect_product_names();
      CREATE TRIGGER collect_product_names
        AFTER INSERT OR DELETE OR UPDATE ON laptops
        FOR EACH ROW EXECUTE PROCEDURE collect_product_names();
    }
    [MobilePhone, Laptop].each do |klass|
      klass.pluck(:name, :id).each do |(name, id)|
        ProductName.create(product_name: name, product_id: id, product_type: klass.table_name)
      end
    end
  end

  def down
    execute %Q{
      DROP TABLE IF EXISTS product_names;
      DROP TRIGGER IF EXISTS collect_product_names ON mobile_phones;
      DROP TRIGGER IF EXISTS collect_product_names ON laptops;
      DROP FUNCTION IF EXISTS collect_product_names();
    }
  end
end

Our databases prepared. Now we need to prepare rails app.

class ProductName < ActiveRecord::Base
  scope :by_name,    -> (pattern) { where("product_name % ?", pattern) }
  scope :statistics, -> { group(:product_type).count }
end

How to use it?

ProductName.by_name("Apple").statistics
#=> {"laptops"=>1, "mobile_phones"=>3}

P.S. Now you can see your function with \df+ on PostgreSQL terminal.

Finding Leaf Nodes with ltree

Thu, 06 Aug 2015 22:20:35 +0000

Let’s assume you want to find all node leaves. To keep it simple for all examples we will use DB from previous post. Leaf is a node without children. If you have parent_id it seems easy to find all leaves

SELECT * FROM "properties" WHERE "id" NOT IN
  (SELECT DISTINCT "parent_id" FROM "properties" WHERE "parent_id" IS NOT NULL);

We just excluding all nodes that are parents for any node and that’s it. But what if you are prefer to use pure ltree without parent_id? And here come troubles. All that I can find by googling this problem is how to find node with most children count:

SELECT subpath("path", 0, 1), count(*)
  FROM "properties"
  GROUP BY 1 ORDER BY 2 DESC;

And to be honest, I envy whoever invented it. Pretty simple: with subpath we retrieve the root element for each node, group by it, and count how many such elements exist. But that is not our problem.

As I said before, leaf is a node without children. We can find all children nodes with ltree <@ ltree.

ltree <@ ltree boolean is left argument a descendant of right (or equal)?

It will return all children and self. As leaf have zero children and self it will be one result for leaf ancestors. As we have statement for filtering leaves we need to check each node. As I just learned LATERAL queries my first thought was to use it. From documentation:

When a FROM item contains LATERAL cross-references, evaluation proceeds as follows: for each row of the FROM item providing the cross-referenced column(s)

Each row, see?

SELECT * FROM (
  SELECT "path", "children_count" FROM "properties",
  LATERAL (
    SELECT COUNT("path") AS "children_count"
    FROM  "properties" AS "subproperties"
    WHERE "properties"."path" @> "subproperties"."path"
  ) AS "subproperties"
) AS "path_with_children" WHERE "path_with_children"."children_count" = 1
GROUP BY 1, 2;

But it seems that there are too much overhead. Something is wrong here. Let’s try to use INNER JOIN with statement instead of additional WHERE.

SELECT * FROM "properties"
  INNER JOIN LATERAL (
    SELECT COUNT("path") AS "children_count"
    FROM  "properties" AS "subproperties"
    WHERE "properties"."path" @> "subproperties"."path"
  ) AS "path_with_children" ON "path_with_children"."children_count" = 1 ;

Seems better but still complex. Wait, is it really necessary to calculate table and then filter values from it? Seems like we can just filter each node with simple query.

SELECT count("path") = 1 FROM "properties" WHERE "path" <@ '1'; -- f
SELECT count("path") = 1 FROM "properties" WHERE "path" <@ '1.4'; -- t
SELECT count("path") = 1 FROM "properties" WHERE "path" <@ '1.2.3'; -- f

And apply it for each node:

SELECT * FROM "properties" AS "main" WHERE (
  SELECT count("path") = 1 FROM "properties" WHERE "path" <@ "main"."path");

Simple and elegant way, what we need!

When I showed it to Andrei, he noticed that it will not work for nodes with the same path. And if it has not so much sense for path with ids it’s normal for path with hierarchical structures.

To reproduce this bug on our data we will add node with same path but another value.

insert into properties (name, path, parent_id) VALUES ('H', '1.3.7', 9);

 id | name | path  | parent_id
----+------+-------+-----------
  4 | D    | 1.4   |         1
  5 | E    | 1.2.5 |         2
  6 | F    | 1.2.6 |         2

We missed two leaves with path 1.3.7 because <@ operator will return all ancestors including self and nodes with same paths. To fix it we just need to add DISTINCT keyword in right place.

SELECT * FROM "properties" AS "main" WHERE (
  SELECT count(DISTINCT "path") = 1 FROM "properties" WHERE "path" <@ "main"."path");

 id | name | path  | parent_id
----+------+-------+-----------
  4 | D    | 1.4   |         1
  5 | E    | 1.2.5 |         2
  6 | F    | 1.2.6 |         2
  7 | G    | 1.3.7 |         3
  8 | H    | 1.3.7 |         9

As a consequence: if your solution is too complex, it’s usually not the right one.

P.S. If you want to see nice example of using LATERAL keyword you can check this article.

Tree Traversal in PostgreSQL: ltree, GiST, and Recursive CTEs

Tue, 21 Jul 2015 09:49:35 +0000

PostgreSQL provides you ltree extension to organize tree-like structures. You can enable this extension with command CREATE EXTENSION IF NOT EXISTS ltree; or you can wrap it with ActiveRecord

def up
  execute "CREATE EXTENSION IF NOT EXISTS ltree"
end

In Ruby, you can use ltree_hierarchy gem to simplify main operations for manipulating with tree’s nodes. It updates path column, that contains all hierarchy for each node. For example, if node parents are 1, 3, 5 and node id is 9, then node path will contain 1.3.5.9. It allows you to easily filter results. Note that the path format differs slightly from the raw PostgreSQL documentation.

As sandbox we will use simple database that contains two tables, one for hierarchical structure and second for some additional data connected to each node of tree. To emulate ltree_hierarchy gem behaviour we fill the path column manually. Don’t forget to enable ltree extension.

create temp table properties (
  id bigserial primary key,
  name varchar(256) NOT NULL,
  path ltree,
  parent_id integer
);

create temp table reports (
  id bigserial primary key,
  property_id integer,
  value integer
);

insert into properties (name, path) VALUES ('A', '1'); -- NULL parent_id for root
insert into properties (name, path, parent_id) VALUES ('B', '1.2', 1);
insert into properties (name, path, parent_id) VALUES ('C', '1.3', 1);
insert into properties (name, path, parent_id) VALUES ('D', '1.4', 1);
insert into properties (name, path, parent_id) VALUES ('E', '1.2.5', 2);
insert into properties (name, path, parent_id) VALUES ('F', '1.2.6', 2);
insert into properties (name, path, parent_id) VALUES ('G', '1.3.7', 3);

insert into reports (property_id, value) VALUES (1, 1);
insert into reports (property_id, value) VALUES (2, 4);
insert into reports (property_id, value) VALUES (3, 19);
insert into reports (property_id, value) VALUES (4, 21);
insert into reports (property_id, value) VALUES (5, 9);
insert into reports (property_id, value) VALUES (6, 11);

If you want more data to fill difference between queries speed you can use my file with 4.5k+ records. (You need download it, connect to schema through psql and run \i path_to_file) Let’s build named paths for each node. Since the path column stores all parent ids, we can easily filter unrelated nodes with like queries.

# select '1.2.3' like '1.2.3%';   -- t
# select '1.2.3.4' like '1.2.3%'; -- t
# select '1.2.3.5' like '1.2.3%'; -- t
# select '1.2.4.5' like '1.2.3%'; -- f

As path type is ltree we need to cast it to varchar with ::varchar, because you can’t concatenate text and ltree path column. Put it all together.

select "properties"."id",
       "properties"."name",
       "properties"."path",
       "subproperties"."name" as "named_path"
  from "properties"
  left join "properties" as "subproperties"
    on ("properties"."path"::varchar) like ("subproperties"."path"::varchar || '%')
  group by 1, 2, 3, 4 order by 1;

 id | name | path  | named_path
----+------+-------+------------
  1 | A    | 1     | A
  2 | B    | 1.2   | A
  2 | B    | 1.2   | B
  3 | C    | 1.3   | A
  3 | C    | 1.3   | C
  4 | D    | 1.4   | A
  4 | D    | 1.4   | D
  5 | E    | 1.2.5 | A
  5 | E    | 1.2.5 | B
  5 | E    | 1.2.5 | E
  6 | F    | 1.2.6 | A
  6 | F    | 1.2.6 | B
  6 | F    | 1.2.6 | F
  7 | G    | 1.3.7 | A
  7 | G    | 1.3.7 | C
  7 | G    | 1.3.7 | G
(16 rows)

We have all node names, but to get a named path we need to concatenate them. We can achieve it with string_agg aggregate function.

string_agg(expression, delimiter) - input values concatenated into a string, separated by delimiter

Seems like what we need

select "properties"."id",
       "properties"."name",
       "properties"."path",
        string_agg("subproperties"."name", '->') as "named_path"
  from "properties"
  left join "properties" as "subproperties"
    on ("properties"."path"::varchar) like ("subproperties"."path"::varchar || '%')
  group by 1, 2, 3 order by 1;

 id | name | path  | named_path
----+------+-------+------------
  1 | A    | 1     | A
  2 | B    | 1.2   | A->B
  3 | C    | 1.3   | A->C
  4 | D    | 1.4   | A->D
  5 | E    | 1.2.5 | A->B->E
  6 | F    | 1.2.6 | A->B->F
  7 | G    | 1.3.7 | A->C->G

Looks correct. Let’s try it against the production database.

 id |   name   |     path    |           named_path
----+----------+-------------+------------------------------
  1 | a450ceb3 | 1           | a450ceb3
  2 | 91b2f902 | 1.2         | a450ceb3->91b2f902
  3 | e4f25325 | 1.2.3       | a450ceb3->91b2f902->e4f25325
  4 | d5fdd885 | 1.2.4       | 91b2f902->d5fdd885->a450ceb3
  5 | 45160d77 | 1.2.5       | a450ceb3->91b2f902->45160d77

Look at record with id 4 — the named path starts from the second record, not the root. Let’s read the documentation more carefully.

This ordering is unspecified by default, but can be controlled by writing an ORDER BY clause within the aggregate call

select "properties"."id",
       "properties"."name",
       "properties"."path",
        string_agg("subproperties"."name", '->'
          order by "subproperties"."path") as "named_path"
  from "properties"
  left join "properties" as "subproperties"
    on ("properties"."path"::varchar) like ("subproperties"."path"::varchar || '%')
  group by 1, 2, 3 order by 1;

 id |   name   |     path    |           named_path
----+----------+-------------+------------------------------
  1 | a450ceb3 | 1           | a450ceb3
  2 | 91b2f902 | 1.2         | a450ceb3->91b2f902
  3 | e4f25325 | 1.2.3       | a450ceb3->91b2f902->e4f25325
  4 | d5fdd885 | 1.2.4       | a450ceb3->91b2f902->d5fdd885
  5 | 45160d77 | 1.2.5       | a450ceb3->91b2f902->45160d77

Got it! All works, but… We have ltree column while we are using like query. A-ha! We can use ltree @> operator.

ltree @> ltree boolean is left argument an ancestor of right (or equal)?

Let’s enable timing to feel difference between queries.

Operating System
  \timing [on|off]       toggle timing of commands (currently off)

Current query on table with 4644 records takes 8584,278 ms. Ok, what will we have by replacing like operator with ltree @>?

select "properties"."id",
       "properties"."name",
       "properties"."path",
        string_agg("subproperties"."name", '->' order by "subproperties"."path") as "named_path"
  from "properties"
  left join "properties" as "subproperties"
    on "subproperties"."path" @> "properties"."path"
  group by 1, 2, 3 order by 1;

Two times faster. Good! Let’s see, what happens.

                                                                   QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=772405.45..772733.09 rows=4644 width=1104) (actual time=5143.003..5164.441 rows=4644 loops=1)
   ->  Sort  (cost=772405.45..772459.37 rows=21567 width=1104) (actual time=5142.988..5146.275 rows=22902 loops=1)
         Sort Key: properties.id, properties.name, properties.path
         Sort Method: external sort  Disk: 2520kB
         ->  Nested Loop Left Join  (cost=0.00..750063.00 rows=21567 width=1104) (actual time=0.018..5095.446 rows=22902 loops=1)
               Join Filter: (subproperties.path @> properties.path)
               Rows Removed by Join Filter: 21543834
               ->  Seq Scan on properties  (cost=0.00..103.44 rows=4644 width=556) (actual time=0.008..0.511 rows=4644 loops=1)
               ->  Seq Scan on properties subproperties  (cost=0.00..103.44 rows=4644 width=548) (actual time=0.001..0.364 rows=4644 loops=4644)
 Time: 3610,234 ms

It filters records with @> operator. But we can use GiST index for such operator to make it faster.

GiST index over ltree[]: ltree[] <@ ltree, ltree @> ltree[], @, ~, ?

Ok, create index.

CREATE INDEX gist_path_index ON "properties" USING gist("path");

And let’s try again.

                                                                           QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=24341.43..24669.07 rows=4644 width=1104) (actual time=148.255..169.519 rows=4644 loops=1)
   ->  Sort  (cost=24341.43..24395.35 rows=21567 width=1104) (actual time=148.212..151.562 rows=22902 loops=1)
         Sort Key: properties.id, properties.name, properties.path
         Sort Method: external sort  Disk: 2520kB
         ->  Nested Loop Left Join  (cost=0.15..1998.98 rows=21567 width=1104) (actual time=0.033..98.501 rows=22902 loops=1)
               ->  Seq Scan on properties  (cost=0.00..103.44 rows=4644 width=556) (actual time=0.007..0.509 rows=4644 loops=1)
               ->  Index Scan using gist_path_index on properties subproperties  (cost=0.15..0.36 rows=5 width=548) (actual time=0.013..0.020 rows=5 loops=4644)
                     Index Cond: (path @> properties.path)
 Total runtime: 177.499 ms

Wow! Instead of filtering records it’s just using our index. It’s 20 times faster than before. Keep it so.

Time goes on and you have new task. You need to find sum of each children for each node. Seems like not big problem, just little modify our previous query.

select "properties"."id",
       "properties"."name",
       "properties"."path",
        sum("reports"."value")
  from "properties"
  left join "properties" as "subproperties"
    on "subproperties"."path" @> "properties"."path"
  inner join "reports"
    on "reports"."property_id" = "subproperties"."id"
  group by 1, 2, 3 order by 1;

 id | path  | name | sum
----+-------+------+-----
  1 | 1     | A    |   1
  2 | 1.2   | B    |   5
  3 | 1.3   | C    |  20
  4 | 1.4   | D    |  22
  5 | 1.2.5 | E    |  14
  6 | 1.2.6 | F    |  16
  7 | 1.3.7 | G    |  20

No problems. 115,062 ms on 4.5k+ records. One day later you need both data, sum and named path. Argh! Roll up your sleeves.

select "properties"."id",
       "properties"."name",
       "properties"."path",
        sum("reports"."value"),
        string_agg("subproperties"."name", '->' order by "subproperties"."path") as "named_path"
  from "properties"
  left join "properties" as "subproperties"
    on "subproperties"."path" @> "properties"."path"
  inner join "reports"
    on "reports"."property_id" = "subproperties"."id"
  group by 1, 2, 3 order by 1;

 id | path  | name | named_path | sum
----+-------+------+------------+-----
  1 | 1     | A    | A          |   1
  2 | 1.2   | B    | A->B       |   5
  3 | 1.3   | C    | A->C       |  20
  4 | 1.4   | D    | A->D       |  22
  5 | 1.2.5 | E    | A->B->E    |  14
  6 | 1.2.6 | F    | A->B->F    |  16
  7 | 1.3.7 | G    | A->C->G    |  20

Exactly what we need. But let’s look a little deeper at what’s happening.

                                                                                 QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate  (cost=24954.00..25332.63 rows=4644 width=1108) (actual time=102.756..125.912 rows=4644 loops=1)
   ->  Sort  (cost=24954.00..25007.43 rows=21372 width=1108) (actual time=102.738..106.177 rows=22901 loops=1)
         Sort Key: properties.id, properties.name, properties.path
         Sort Method: external sort  Disk: 2648kB
         ->  Nested Loop  (cost=352.44..2668.99 rows=21372 width=1108) (actual time=1.949..56.588 rows=22901 loops=1)
               ->  Merge Join  (cost=352.29..790.59 rows=4602 width=552) (actual time=1.924..5.101 rows=4643 loops=1)
                     Merge Cond: (subproperties.id = reports.property_id)
                     ->  Index Scan using properties_pkey on properties subproperties  (cost=0.28..357.94 rows=4644 width=556) (actual time=0.015..0.737 rows=4644 loops=1)
                     ->  Sort  (cost=352.01..363.51 rows=4602 width=8) (actual time=1.905..2.367 rows=4644 loops=1)
                           Sort Key: reports.property_id
                           Sort Method: quicksort  Memory: 410kB
                           ->  Seq Scan on reports  (cost=0.00..72.02 rows=4602 width=8) (actual time=0.009..0.961 rows=4644 loops=1)
               ->  Index Scan using gist_path_index on properties  (cost=0.15..0.36 rows=5 width=556) (actual time=0.009..0.010 rows=5 loops=4643)
                     Index Cond: (subproperties.path @> path)
 Total runtime: 132.957 ms
(15 rows)

loops=4643 — a self join processes N² records. We need something else, and PostgreSQL has it: WITH Queries. Pay attention to the RECURSIVE keyword.

The optional RECURSIVE modifier changes WITH from a mere syntactic convenience into a feature that accomplishes things not otherwise possible in standard SQL. Using RECURSIVE, a WITH query can refer to its own output. The general form of a recursive WITH query is always a non-recursive term, then UNION (or UNION ALL), then a recursive term, where only the recursive term can contain a reference to the query’s own output

We have a tree, we have roots that can be used as initial term and children that can be traversed recursive.

with recursive nodes("id", "name", "path", tree_sum, named_path) as (
  select roots."id", roots."name", roots."path", "reports"."value" as tree_sum, cast (roots."name" AS varchar(255)) as named_path
    from properties roots
    inner join reports on ("property_id" = roots."id")
    where (roots."parent_id" is NULL)
  union all
  select children."id", children."name", children."path", (tree_sum + "reports"."value"), cast (nodes.named_path ||'->'|| children."name" AS varchar(255))
    from properties children
    inner join nodes on (nodes."id"= children."parent_id")
    inner join reports on (reports."property_id" = children."id")
  )
  select * from nodes order by 1;

 id | name | path  | tree_sum | named_path
----+------+-------+----------+------------
  1 | A    | 1     |        1 | A
  2 | B    | 1.2   |        5 | A->B
  3 | C    | 1.3   |       20 | A->C
  4 | D    | 1.4   |       22 | A->D
  5 | E    | 1.2.5 |       14 | A->B->E
  6 | F    | 1.2.6 |       16 | A->B->F

                                                                               QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=66902.48..67034.87 rows=52953 width=1076) (actual time=31.521..32.130 rows=4643 loops=1)
   Sort Key: nodes.id
   Sort Method: external sort  Disk: 576kB
   CTE nodes
     ->  Recursive Union  (cost=103.73..11729.62 rows=52953 width=1080) (actual time=0.817..26.962 rows=4643 loops=1)
           ->  Hash Join  (cost=103.73..193.29 rows=23 width=560) (actual time=0.814..2.210 rows=48 loops=1)
                 Hash Cond: (reports.property_id = roots.id)
                 ->  Seq Scan on reports  (cost=0.00..72.02 rows=4602 width=8) (actual time=0.011..0.675 rows=4644 loops=1)
                 ->  Hash  (cost=103.44..103.44 rows=23 width=556) (actual time=0.788..0.788 rows=48 loops=1)
                       Buckets: 1024  Batches: 1  Memory Usage: 3kB
                       ->  Seq Scan on properties roots  (cost=0.00..103.44 rows=23 width=556) (actual time=0.008..0.775 rows=48 loops=1)
                             Filter: (parent_id IS NULL)
                             Rows Removed by Filter: 4596
           ->  Hash Join  (cost=359.76..1047.73 rows=5293 width=1080) (actual time=1.000..3.967 rows=766 loops=6)
                 Hash Cond: (children.parent_id = nodes_1.id)
                 ->  Merge Join  (cost=352.29..790.59 rows=4602 width=564) (actual time=0.273..3.054 rows=4643 loops=6)
                       Merge Cond: (children.id = reports_1.property_id)
                       ->  Index Scan using properties_pkey on properties children  (cost=0.28..357.94 rows=4644 width=560) (actual time=0.006..0.638 rows=4644 loops=6)
                       ->  Sort  (cost=352.01..363.51 rows=4602 width=8) (actual time=0.266..0.557 rows=4644 loops=6)
                             Sort Key: reports_1.property_id
                             Sort Method: quicksort  Memory: 410kB
                             ->  Seq Scan on reports reports_1  (cost=0.00..72.02 rows=4602 width=8) (actual time=0.004..0.807 rows=4644 loops=1)
                 ->  Hash  (cost=4.60..4.60 rows=230 width=528) (actual time=0.216..0.216 rows=774 loops=6)
                       Buckets: 1024  Batches: 1  Memory Usage: 207kB
                       ->  WorkTable Scan on nodes nodes_1  (cost=0.00..4.60 rows=230 width=528) (actual time=0.001..0.094 rows=774 loops=6)
   ->  CTE Scan on nodes  (cost=0.00..1059.06 rows=52953 width=1076) (actual time=0.820..29.086 rows=4643 loops=1)
 Total runtime: 36.068 ms

4x time faster. And it will not slow down with table growing, it only depends on tree height.

What is the point? You need to use suitable instruments. If you need traverse tree – use WITH RECURSIVE, if you need to work with single node relations – ltree is what you need.

Materialized Views and Concurrent Refresh

Fri, 19 Jun 2015 17:49:35 +0000

Materialized view is an object that contains the query’s results. Unlike a database table it doesn’t support INSERT/UPDATE/DELETE operations. Since these operations are unsupported, to update a materialized view you need to call a refresh operation. In PostgreSQL materialized views support was introduced in version 9.3. From PostgreSQL documentation you can see how to create materialized view. So, you need query that you want to materialize and… that all.

Let’s assume you have next structure: books, authors and feedbacks. Each author has many books, each book has many feedbacks

# author.rb
class Author < ActiveRecord::Base
  has_many :books
  has_many :feedbacks, through: :books
end

# book.rb
class Book < ActiveRecord::Base
  belongs_to :author
  has_many :feedbacks
end

# feedback.rb
class Feedback < ActiveRecord::Base
  belongs_to :book
end

And let’s assume you need to we need to show top authors by their feedbacks. We have Rails ActiveRecord and it seems easy

Author.joins(:feedbacks, :books)
      .group("authors.name")
      .order("sum(feedbacks.mark) DESC")
      .sum("feedbacks.mark")
# {"Alexander Pushkin"=>360, "Mikhail Lermontov"=>330, "Nikolai Gogol"=>180}

But three INNER JOINS, isn’t too much? Let’s use our materialized views! Starting from migration:

class CreateAuthorsFeedbacks < ActiveRecord::Migration
  def self.up
    query = %Q{
      CREATE MATERIALIZED VIEW authors_feedbacks  AS
      #{Author.joins(:feedbacks, :books)
          .select("sum(feedbacks.mark)", "authors.name", "authors.id as author_id")
          .group("authors.name", "authors.id")
          .order("sum(feedbacks.mark) DESC")
          .to_sql};
    }
    execute query
    add_index :authors_feedbacks, :author_id, unique: true
  end

  def self.down
    execute <<-SQL
      DROP MATERIALIZED VIEW IF EXISTS authors_feedbacks;
    SQL
  end
end

To be honest, I’m not a big fan of using Rails to_sql method and so on and prefer to write pure SQL for such migrations. But it’s sample and we will keep it so. Each selected column will be materialized view column, that’s why we used as for authors.id, in our table “authors.id” will be stored in “author_id” column. Unlike simple views, we can index any materialized view column, additionaly, we will make it index unique. As I said before, to actualize data in view we need to call refresh view. In pure PostgreSQL it will be:

REFRESH MATERIALIZED VIEW authors_feedbacks

But in PostgreSQL 9.4 we can do it concurrently! It will refresh the materialized view without locking out concurrent selects on the materialized view, but… You need uniq index on materialized view for this. Wait a minute, we alredy have one!

REFRESH MATERIALIZED VIEW CONCURRENTLY authors_feedbacks

As I can see, concurrently refreshing much more quick if your index very simple. With complex index it can be even slower then unconcurrent refreshing. As it behave like table we can even use it as ActiveRecord model.

# authors_feedback.rb
class AuthorsFeedback < ActiveRecord::Base
  include ReadOnlyModel

  belongs_to :author

  # Refresh materialized view by reaggregating data from each connected table
  def self.refresh_view
    connection = ActiveRecord::Base.connection
    connection.execute("REFRESH MATERIALIZED VIEW #{table_name}")
  end

  # Concurrently refresh materialized view by reaggregating data from each
  # connected table
  def self.refresh_view_concurrently
    connection = ActiveRecord::Base.connection
    connection.execute("REFRESH MATERIALIZED VIEW CONCURRENTLY #{table_name}")
  end
end

As I said before, materialized view doesn’t support any editing operations, for this we will use concern ReadOnlyModel

module ReadOnlyModel
  extend ActiveSupport::Concern

  included do
    # Required to block update_attribute and update_column
    attr_readonly(*column_names)
  end

  def readonly?
    # Does not block destroy or delete
    true
  end

  def destroy
    raise ActiveRecord::ReadOnlyRecord
  end

  def delete
    raise ActiveRecord::ReadOnlyRecord
  end
end

And we need to add association for authors

class Author < ActiveRecord::Base
  has_many :books
  has_many :feedbacks, through: :books
  has_one  :authors_feedback
end

And that’s all, just use it.

Author.last.authors_feedback
# => #<AuthorsFeedback sum: 180, name: "Nikolai Gogol", author_id: 3>
AuthorsFeedback.all
#=> #<ActiveRecord::Relation [
  #<AuthorsFeedback sum: 360, name: "Alexander Pushkin", author_id: 1>,
  #<AuthorsFeedback sum: 330, name: "Mikhail Lermontov", author_id: 2>,
  #<AuthorsFeedback sum: 180, name: "Nikolai Gogol", author_id: 3>
#]>

P.S. Don’t forget to change your dump format, schema.rb can’t store materialized view structure, so, you need to store all this in SQL. Change in application.rb

config.active_record.schema_format = :sql