Get started
Question this section answers: How do I go from zero to a working produce/fetch loop?
Install
Requires Rust 1.99+ (edition 2024).
[dependencies]
object-log = "0.4"Optional features:
| Feature | What you get |
|---|---|
s3 | S3BlobStore (AWS S3, MinIO, Garage, …) |
cli | object-log diagnostics / produce-consume binary |
cargo add object-log
# or
cargo add object-log --features s3
cargo install object-log --features cliMinimal produce / fetch
use object_log::{
Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey,
};
use bytes::Bytes;
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), object_log::ObjectLogError> {
let engine = LogEngine::new(
Arc::new(MemoryBlobStore::new()),
Arc::new(InMemorySequencer::new()),
FlushConfig::default(),
"log/",
);
let p = PartitionKey("events-0".into());
let out = engine
.produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
.await?;
assert_eq!(out.base_offset, Some(0));
let batches = engine.fetch(&p, 0, 1 << 20).await?;
assert_eq!(batches[0].payload, "hello");
Ok(())
}Notes:
MemoryBlobStoreis not crash-durable—tests and demos only.- For durability on disk use
LocalBlobStore; for S3 enable features3. - For a crash-durable standalone index use
ManifestSequencerinstead ofInMemorySequencer.
Durability levels
| Level | Resolves when | Offsets? |
|---|---|---|
Buffered | Batch is in the flush buffer | No |
Durable | Containing object PUT succeeded | No |
Sequenced | PUT and Sequencer::commit succeeded | Yes |
After Buffered produces, call engine.flush() to barrier work already enqueued.
CLI black-box (optional)
cargo install --path . --features cli # from a checkout
# or: cargo install object-log --features cli
printf 'a\nb\nc\n' | object-log produce --root /tmp/olog --partition demo --lines
object-log consume --root /tmp/olog --partition demo --lines