EnclaveStack
GitHubGet started

Queues

Messages that wait for a worker, with retries and a dead-letter queue.

Overview

A queue holds messages until a worker takes them. If the worker fails, the message comes back and is tried again, so nothing is lost.

When to use it

  • Emails, image resizing or payments in the background.
  • Smoothing out bursts.
  • Handing work between services.

How it works

NATS JetStream
One broker per machine (NATS 2.15), its store on a volume, memory capped at 512 MiB.
A user per queue
Each queue has its own user and managed password.

A received message stays hidden for the visibility timeout; if it isn’t deleted by then, it comes back. After its last try it goes to the dead-letter queue, if you set one.

Architecture

Queues architectureProducers send, workers receive and delete; failed messages come back.sendreceiveafter max triesAPIsendsNATS JetStreamqueue emailsDead-letter queueWorkerreceives, deletes
Producers send, workers receive and delete; failed messages come back.
  • API to NATS JetStream: send
  • NATS JetStream to Worker: receive
  • NATS JetStream to Dead-letter queue: after max tries

Flows

A message’s life

  1. The API sends a message.
  2. A worker receives it; it is hidden for 30 seconds.
  3. The worker deletes it once done — or it comes back to be tried again.

Set it up

In the console

  1. Open Queues and choose New queue.
  2. Set the visibility timeout, tries and retention.
  3. Use the Messages tab to send and receive by hand.

With the CLI

enclave queue create emails --wait

With OpenTofu

OpenTofu configuration
resource "enclave_queue" "emails" {
  name = "emails"
}

Settings reference

Every setting and output of the OpenTofu resource; the console and the CLI use the same names in their own words. Generated from platform 4d30582.

enclave_queue
NameTypeUseWhat it does
nametextrequiredName of the queue: 3–40 lowercase letters, digits or hyphens, starting with a letter. Changing it replaces the queue.
dead_letter_queuetextoptionalQueue that takes messages past max_receives.
max_message_bytesnumberoptionalLargest body, 1024–1048576 (default 262144).
max_receivesnumberoptionalReceives before a message is dead-lettered, 0–1000 (default 0: no limit).
projecttextoptionalThe project to create it in. Defaults to the provider's project (ENCLAVE_PROJECT, else default). Changing it replaces the resource.
retention_hoursnumberoptionalHow long a message is kept, 1–336 (default 96).
visibility_timeout_secondsnumberoptionalHow long a received message stays hidden, 1–43200 (default 30).
broker_urltextoutputThe broker on this machine, nats://127.0.0.1:<port>.
idtextoutputResource ID.
internal_broker_urltextoutputThe broker inside EnclaveStack's networks, nats://es-system-broker:4222.
password_secrettextoutputName of the managed secret holding the user's password.
subjecttextoutputThe subject apps publish to.
usernametextoutputThe broker user apps sign in as.

Limits and security

Good to know

  • Messages up to 1 MiB; kept 1 – 336 hours (96 by default).

Security

  • Each queue has its own credentials.