Skip to content

Get started

Question this section answers: How do I go from zero to a working produce/fetch loop?

Install

Requires Rust 1.99+ (edition 2024).

[dependencies]
object-log = "0.4"

Optional features:

FeatureWhat you get
s3S3BlobStore (AWS S3, MinIO, Garage, …)
cliobject-log diagnostics / produce-consume binary
cargo add object-log
# or
cargo add object-log --features s3
cargo install object-log --features cli

Minimal produce / fetch

use object_log::{
    Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey,
};
use bytes::Bytes;
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), object_log::ObjectLogError> {
    let engine = LogEngine::new(
        Arc::new(MemoryBlobStore::new()),
        Arc::new(InMemorySequencer::new()),
        FlushConfig::default(),
        "log/",
    );
    let p = PartitionKey("events-0".into());

    let out = engine
        .produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
        .await?;
    assert_eq!(out.base_offset, Some(0));

    let batches = engine.fetch(&p, 0, 1 << 20).await?;
    assert_eq!(batches[0].payload, "hello");
    Ok(())
}

Notes:

  • MemoryBlobStore is not crash-durable—tests and demos only.
  • For durability on disk use LocalBlobStore; for S3 enable feature s3.
  • For a crash-durable standalone index use ManifestSequencer instead of InMemorySequencer.

Durability levels

LevelResolves whenOffsets?
BufferedBatch is in the flush bufferNo
DurableContaining object PUT succeededNo
SequencedPUT and Sequencer::commit succeededYes

After Buffered produces, call engine.flush() to barrier work already enqueued.

CLI black-box (optional)

cargo install --path . --features cli   # from a checkout
# or: cargo install object-log --features cli

printf 'a\nb\nc\n' | object-log produce --root /tmp/olog --partition demo --lines
object-log consume --root /tmp/olog --partition demo --lines

Next