Skip to content

API reference

potik also re-exports the core names that services use, such as Config, Producer, Runtime, and the record types. Each name is documented once, under the layer that defines it.

Runtime

potik.core

The core runtime, a minimal wrapper around confluent-kafka.

It works without the typed layer: a handler takes a raw ConsumerRecord and returns a list or tuple of ProducerRecords.

OffsetPolicy

OffsetPolicy = Callable[[OffsetContext], int]

A function that returns where a newly assigned partition starts, e.g. resume_or_earliest.

It returns the committed offset or an offset within the log, else the run fails with InvalidOffsetError. If it returns another offset than the committed one, the runtime commits it before handling, so a restart resumes from it. An offset below the committed one handles records again, whose outputs may already be committed, and an offset above it skips records. It runs on the consumer's thread during the rebalance, so it must be synchronous and return quickly. rebalance_timeout_ms bounds the whole assignment.

ConsumerRecord

ConsumerRecord = Message

The native confluent_kafka.Message, not a copy. Its fields are methods, e.g. record.key() and record.offset().

Headers

Headers = list[tuple[str, str | bytes | None]] | dict[str, str | bytes | None]

A record's headers: a list of (name, value) pairs, which keeps order and allows repeated names, or a dict.

Values are str, bytes, or None.

RecordHandler

RecordHandler = Callable[[ConsumerRecord], Awaitable[Sequence[ProducerRecord]]]

A core handler. It takes one record and returns its outputs as a list or tuple, possibly empty.

Returning does not commit. The runtime commits the outputs and the record's offset later, in one transaction with other handled records.

RecordSender

RecordSender = Callable[[ProducerRecord], Awaitable[Delivery]]

The call_next of BaseMiddleware.send. It sends one record and returns once the broker acknowledges it.

Config dataclass

Config(*, bootstrap_servers: str, group_id: str, offset_policy: OffsetPolicy = resume_or_latest, max_buffered_bytes: int = 16 * 1024 * 1024, stop_failed_partitions: bool = True, commit_batch_size: int = 1000, commit_interval_ms: int = 100, transaction_timeout_ms: int = 10000, rebalance_timeout_ms: int = 60000, revocation_grace_ms: int = 0, shutdown_grace_ms: int = 0, max_poll_interval_ms: int = 300000, transactional_id_prefix: str = 'potik', options: Mapping[str, Any] = dict(), consumer_options: Mapping[str, Any] = dict(), producer_options: Mapping[str, Any] = dict())

Runtime limits and librdkafka client options.

The runtime owns the client settings that its guarantees depend on, e.g. isolation.level and enable.auto.commit, and the options cannot override them. Callbacks in the options, e.g. error_cb, run on the runtime's client threads, so they must be thread-safe functions, not coroutines.

The producer's transactional.id is {transactional_id_prefix}-{random}, new each run. With static membership, i.e. a group.instance.id in the options, it derives from the group and instance IDs instead. A restarted instance then aborts the transaction that its crashed run left open, so read_committed readers do not wait transaction_timeout_ms for it. A second live process with the same instance ID fences the first one's producer.

Parameters:

Name Type Description Default
bootstrap_servers str

The cluster's bootstrap servers.

required
group_id str

The consumer group.

required
offset_policy OffsetPolicy

Picks the start offset of each newly assigned partition (see OffsetPolicy). resume_or_latest, the default, resume_or_earliest, and resume_or_fail resume the committed offset, and differ only for a partition without one.

resume_or_latest
max_buffered_bytes int

The memory of the fetched records that have not gone to a transaction yet, across all partitions, i.e. those that wait for their handler, run, or wait for an earlier record of their partition (see the key_concurrency of Runtime). A record counts as its key and value plus 350 bytes for its objects and a typical traceparent header, so a UUID key and 1KB of JSON count as about 1.4KB. With consumer lag, the buffers stay full, so this is steady memory, not a peak. Each partition gets an equal share, at least 16KB, so the total can exceed it with more than max_buffered_bytes / 16384 partitions. Handlers that do I/O need only a few buffered records each. Records arrive one record batch of a partition at a time, and a share smaller than the batches of the topics' producers makes partitions pause and fetch records again, which slows handlers that take about a millisecond or less. It does not count librdkafka's prefetch queue, which potik's defaults limit to about 24MB, handler outputs, or handled records that wait for their transaction, at most two commit_batch_size batches. A fetch of records larger than the previous fetch's can exceed a partition's share, by at most the prefetch queue, so the budget is a target, not a hard limit.

16 * 1024 * 1024
stop_failed_partitions bool

Whether a failure of one partition stops it, instead of failing the run. A stopped partition handles nothing past the failed record, which stays uncommitted, while other partitions keep running. The record runs again once the partition is assigned again, e.g. after a restart. A failure is a handler exception that no exception rule handles, or a fetch error, e.g. a committed offset that retention deleted.

True
commit_batch_size int

The maximum number of handled records per transaction.

1000
commit_interval_ms int

The minimum time between the starts of two transactions, unless the batch fills first. A transaction also waits for the previous one to finish, so batches grow with throughput.

100
transaction_timeout_ms int

The producer's transaction.timeout.ms, after which the broker aborts an open transaction. It also limits each commit and abort, so a broker outage longer than it fails the run, at the latest once a broker returns. At least 1000.

10000
rebalance_timeout_ms int

The longest that the runtime may take to handle an assignment or a revocation, including the revoked partitions' grace period and final commit. Exceeding it fails the run. Must be less than max_poll_interval_ms.

60000
revocation_grace_ms int

How long in-flight handlers of revoked partitions may run before the runtime cancels them. Their records run again on the partitions' next owner. The default, 0, cancels them at once. Must be less than rebalance_timeout_ms.

0
shutdown_grace_ms int

How long in-flight handlers may run on shutdown before the runtime cancels them. The default, 0, cancels them at once. Plus rebalance_timeout_ms, it limits the whole shutdown, including the final commit, and must be less than max_poll_interval_ms.

0
max_poll_interval_ms int

The consumer's max.poll.interval.ms. The runtime keeps polling while handlers run, so handlers may take longer than this.

300000
transactional_id_prefix str

The start of the producer's transactional.id. Give each app its own, so that its ACL on the prefixed TransactionalId resource covers no other app's IDs.

'potik'
options Mapping[str, Any]

librdkafka options of both clients, e.g. for the connection, TLS, SASL, OAuth, client.id, or logging.

dict()
consumer_options Mapping[str, Any]

librdkafka options of the consumer, e.g. group.remote.assignor, group.instance.id, client.rack, or fetch sizes.

dict()
producer_options Mapping[str, Any]

librdkafka options of the transactional producer, e.g. compression.type or linger.ms.

dict()

Raises:

Type Description
ValueError

If a limit is out of range, or options override a setting that the runtime owns.

client_config

client_config() -> dict[str, Any]

Return librdkafka options to connect a client of your own to the cluster, e.g. an AdminClient.

InvalidOffsetError

Bases: PotikError

Raised when an offset policy refuses a partition, e.g. resume_or_fail, or returns an invalid offset.

A valid offset is the committed one, or within the partition's log. The error fails the run.

InvalidOutputError

Bases: PotikError, PermanentError

Raised when a handler returns something other than a list or tuple of ProducerRecords.

A core handler that returns a single record raises it, for example. It comes from the handler call, so exception rules see it like any other handler exception. To stop the partition instead of letting a catch-all rule handle it, put on(InvalidOutputError, reraise()) before that rule.

PartitionsLostError

Bases: PotikError

Raised when the consumer loses partitions without a revocation, so another member may already own them.

It happens e.g. when the consumer misses its heartbeats. The error fails the run, which commits nothing more.

PermanentError

Bases: Exception

Marks a failure that would repeat if the handler ran again, e.g. a value that does not decode.

retry() and retry_forever() rules pass it to the next rule instead of retrying. Subclass it for such failures of your own. It does not derive from PotikError, so except PotikError does not catch your subclasses.

PotikError

Bases: Exception

The base of the errors that potik raises.

Errors of the Kafka client stay KafkaExceptions, and invalid arguments raise ValueError or TypeError.

BaseMiddleware

The base class of middleware, which wraps the stages of a Runtime or a Producer.

Subclass it and override any of these stages:

  • lifespan surrounds a run, or the time a Producer is open.
  • handle wraps each handler call of a run.
  • commit wraps each transaction of a run.
  • send wraps each Producer.send().

Each stage except lifespan receives call_next, the rest of the chain. A middleware can run code around it, observe its outcome, and scope state to it, e.g. a span. One class can override every stage, so that e.g. one tracing middleware serves runtimes and producers. A runtime or producer runs its middleware outermost first.

An instance serves one run or one Producer, once, so its state lasts one run. Using it a second time raises RuntimeError. Create the instances where you create the runtime or producer, and pass resources, e.g. a connection pool, to __init__.

lifespan

lifespan(scope: RunScope | ProducerScope) -> AbstractAsyncContextManager[None]

Return an async context manager that surrounds one run, or the time a Producer is open.

Write it with @asynccontextmanager. scope is a RunScope for a run, and a ProducerScope for a producer. Code before yield runs before the clients start, and a run is ready only once every lifespan has entered. Code after yield runs once the clients have closed, in reverse order. yield raises the run's failure, or the cancellation that stopped it, and the lifespan cannot suppress it. The exit is shielded from cancellation, so it may await, e.g. to close a connection pool. A stuck exit holds up shutdown, so cancel any background tasks after yield.

handle async

handle(record: ConsumerRecord, call_next: RecordHandler) -> Sequence[ProducerRecord]

Wrap one handler call, and return its outputs.

It is the only stage that may recover from a failure, e.g. by calling call_next again, or by returning outputs instead of raising. It may also change the outputs.

commit async

commit(transaction: Transaction, call_next: Callable[[Transaction], Awaitable[None]]) -> None

Wrap one transaction, including the retries when a rebalance makes the broker reject it.

Call call_next exactly once. Pass it the transaction, or a copy with other outputs made with dataclasses.replace(), which keeps the offsets. Code after call_next runs once the transaction has committed, before the next transaction starts, and before a revocation or shutdown that waits for it completes. Slow code there slows commits, and so the handlers. Raising fails the run. If code after call_next raises, the transaction stays committed. The stage cannot suppress an exception from call_next. A run failure cancels the stage, so the code after call_next may not run even when the transaction committed.

send async

send(record: ProducerRecord, call_next: RecordSender) -> Delivery

Wrap one send through a Producer, from producing record until the broker acknowledges it.

Outputs never pass through it. They go through handle and commit.

ProducerScope dataclass

ProducerScope(*, client_options: Mapping[str, Any])

Describes a Producer to a middleware's lifespan.

Attributes:

Name Type Description
client_options Mapping[str, Any]

librdkafka options to connect a client of the middleware's own to the producer's cluster.

RunScope dataclass

RunScope(*, config: Config, status: Callable[[], RuntimeStatus], client_options: Mapping[str, Any])

Describes a run to a middleware's lifespan.

Attributes:

Name Type Description
config Config

The run's config.

status Callable[[], RuntimeStatus]

Returns a snapshot of the run. Any thread may call it, e.g. a metrics exporter's. Before the lifespan yields, the run is not running yet.

client_options Mapping[str, Any]

librdkafka options to connect a client of the middleware's own to the run's cluster, e.g. an AdminClient.

OffsetContext

OffsetContext(topic: str, partition: int, committed: int | None, *, watermarks: Callable[[], tuple[int, int]])

What an offset policy decides on: a newly assigned partition, its committed offset, and its log's bounds.

Parameters:

Name Type Description Default
topic str

The partition's topic.

required
partition int

The partition's number.

required
committed int | None

The committed offset, or None if there is none.

required
watermarks Callable[[], tuple[int, int]]

Returns the log's first and end offsets. The context calls it only once a policy reads earliest or latest, and at most once, because the runtime asks the broker. In tests, pass e.g. lambda: (0, 100).

required

Attributes:

Name Type Description
topic

The partition's topic.

partition

The partition's number.

committed

The committed offset, or None if there is none.

earliest property

earliest: int

The log's first offset.

latest property

latest: int

The log's end offset, which the next record gets.

Producer

Producer(bootstrap_servers: str, *, options: Mapping[str, Any] | None = None, middleware: Iterable[BaseMiddleware] = (), close_timeout: float = _CLOSE_TIMEOUT)

A Kafka producer that sends records at once, outside any transaction.

Use it where there is no input record to commit with, e.g. in an API service, or from a handler. Open it once with async with, e.g. in an API service's lifespan, or around a run, and pass it to handlers through potik.build_runtime(context=...). Exiting waits up to close_timeout for every send to be delivered or fail, then fails the rest. On a normal exit, it then raises the first failure of a send that nobody awaited, e.g. a cancelled one. A producer opens only once.

A send from a handler is at-least-once, unlike the handler's outputs. A retried or replayed handler sends again, and readers see the record before the input record commits.

The producer is idempotent and waits for all in-sync replicas (acks=all). After a fatal error, e.g. a broken idempotent sequence, every later send fails. A callback in options that raises does the same, and also fails the sends that wait for a delivery report. Check failure in a liveness probe to restart the process then.

Parameters:

Name Type Description Default
bootstrap_servers str

The cluster's bootstrap servers.

required
options Mapping[str, Any] | None

librdkafka options, e.g. TLS, SASL, linger.ms, or message.timeout.ms, which bounds how long a send may take. Keep the partitioner the same as other producers of the same topics, so that each key stays in one partition.

None
middleware Iterable[BaseMiddleware]

Middleware instances, outermost first. A producer runs their lifespan and send stages (see BaseMiddleware). Add a TopicCheckMiddleware to check at the start that the topics exist. The producer never asks the broker to create a missing topic, unless allow.auto.create.topics is set back to true.

()
close_timeout float

How many seconds exiting waits for sends to be delivered or fail. Then it fails the rest with a _PURGE_QUEUE or _PURGE_INFLIGHT error, although a purged send that was already in flight may still be delivered. Keep it, plus the time that the rest of the shutdown takes, below the process supervisor's deadline, e.g. Kubernetes' termination grace period of 30 seconds.

_CLOSE_TIMEOUT

Raises:

Type Description
ValueError

If options override a setting that the producer owns, or close_timeout is negative.

failure property

failure: KafkaException | None

The fatal error that stopped this producer, if any.

Only a new producer, e.g. in a restarted process, can send again.

from_config classmethod

from_config(config: Config, *, middleware: Iterable[BaseMiddleware] = (), close_timeout: float = _CLOSE_TIMEOUT) -> Self

Make a producer for a runtime's cluster, with the config's options and producer_options.

send async

send(record: ProducerRecord) -> Delivery

Send record at once, and return where the broker appended it.

It returns once the broker acknowledges the record. If librdkafka's queue is full, it first waits for room. A cancelled send may still be delivered.

Raises:

Type Description
KafkaException

If the record was not delivered, or the producer has failed.

RuntimeError

If the producer is not entered.

PendingReply

PendingReply(correlation_id: str)

The reply that a ReplyListener.expect() block awaits.

wait async

wait() -> ConsumerRecord

Return the first reply with this correlation ID, which may have arrived before the call.

Bound the wait with e.g. anyio.fail_after().

Raises:

Type Description
KafkaException

If the listener failed.

RuntimeError

If the listener closed before the reply arrived.

ReplyListener

ReplyListener(reply_topic: str, *, bootstrap_servers: str, correlate: Callable[[ConsumerRecord], str | None], options: Mapping[str, Any] | None = None)

Hands each reply from a reply topic to the request in this process that awaits it.

Every process that sends requests, e.g. each replica of an API, reads every partition of the reply topic itself. The consumer belongs to no group and commits nothing. It reads from the end of each partition, as it is when the listener opens, so it never reads a reply to a request from an earlier process. It reads with read_committed, so a transactional responder's reply arrives once its request's offset commits.

The listener does not send requests, and potik sets no convention for correlation IDs. The caller puts each request's ID wherever its responder reads it, e.g. a header or a field of the value, and the responder copies it to its reply, where correlate reads it.

A reply that arrives while nothing awaits it is dropped, e.g. after its request timed out, or one to another process. A request that times out may still be handled. A responder that runs it again sends another reply, and only the first one counts. A reply whose correlation ID correlate cannot read is logged and dropped.

Open it once with async with, e.g. in an API's lifespan, before the API takes requests. Once it opens, it reads every reply to a request that the process sends. Exiting fails the requests that still await a reply. After a fatal consumer error, every awaited and later request fails. Check failure in a liveness probe to restart the process then.

Parameters:

Name Type Description Default
reply_topic str

The topic that the responders reply to.

required
bootstrap_servers str

The cluster's bootstrap servers.

required
correlate Callable[[ConsumerRecord], str | None]

Returns a reply's correlation ID, or None if it has none. It runs on the listener's thread for every record of the reply topic, so it must be thread-safe and fast.

required
options Mapping[str, Any] | None

librdkafka consumer options, e.g. TLS or SASL. group.id defaults to potik-reply-listener. The consumer never joins the group or commits its offsets, but the client requires one, and its ACL may need to allow it.

None

Raises:

Type Description
ValueError

If options override a setting that the listener owns.

failure property

failure: KafkaException | None

The error that stopped reading replies, if any. Check it in a liveness probe.

expect

expect(correlation_id: str | None = None) -> Generator[PendingReply]

Await the reply with correlation_id from entering the block until leaving it.

Enter the block before sending the request, so that a fast reply cannot arrive first. Send PendingReply.correlation_id with the request in any way the responder reads it, e.g. a header, a field of its value, or an HTTP call that makes the responder reply to Kafka.

Parameters:

Name Type Description Default
correlation_id str | None

The request's ID, unique among this process's requests. Defaults to a new UUID.

None

Raises:

Type Description
RuntimeError

If the listener is not open.

KafkaException

If the listener failed.

ValueError

If another block in this process awaits the same ID.

Runtime

Runtime(config: Config, topics: Iterable[str], handler: RecordHandler, *, middleware: Iterable[BaseMiddleware] = (), key_concurrency: Mapping[str, int] | None = None)

Consumes topics, calls a handler for each record, and commits its outputs with the record's offset.

Each transaction produces a batch of outputs and commits their input records' offsets atomically, so each record's outputs appear exactly once to read_committed readers. A run uses one consumer, with the consumer group protocol of Kafka 4.0+, and one transactional producer. Each assigned partition handles its records one at a time, in offset order, unless its topic runs by key (see key_concurrency), and partitions run concurrently. The consumer keeps polling while handlers run, so a slow handler does not make it leave the group.

On a revocation or shutdown, the runtime lets in-flight handlers finish within a grace period, then cancels them, and commits the handled records before it gives up the partitions. A failure of one partition stops it (see Config.stop_failed_partitions). Any other failure, e.g. a failed transaction or lost partitions, stops the run without committing more, and the uncommitted records run again on the next run.

Cancelling run() requests a graceful shutdown and waits for it. If the shutdown fails, run() raises its error instead of the cancellation. The runtime never abandons a blocking client call, so the process supervisor must enforce a hard deadline.

A runtime runs once. Make a new one, with new middleware, for another run. potik.build_runtime() builds one from routers of typed handlers.

Parameters:

Name Type Description Default
config Config

The runtime limits and client options.

required
topics Iterable[str]

The names of the topics to subscribe to.

required
handler RecordHandler

Returns the outputs of one record of any of topics, as a list or tuple of ProducerRecords. Any other result raises InvalidOutputError from the handler call, so middleware handles it like any other handler exception.

required
middleware Iterable[BaseMiddleware]

Middleware instances, outermost first (see BaseMiddleware). Put a TopicCheckMiddleware first to check that the topics exist before the run starts, and a ConcurrencyMiddleware last to limit concurrent handler calls.

()
key_concurrency Mapping[str, int] | None

The topics whose records run concurrently by key, each with the maximum number of its keys whose records run at once, across its partitions. While keys wait for their turn, a key with more records takes turns with them, so that every partition makes progress. Records with the same key run one at a time, in offset order, and records with other keys run concurrently. All records with a null key count as one key. A key is in progress while its records run, including their retries' backoff, so the limit is not a pool of handler calls: limit those with a ConcurrencyMiddleware. A record that finishes before an earlier record of its partition waits for it, with its outputs, so that the committed offset never skips an unfinished record. Records count toward Config.max_buffered_bytes until they go to a transaction, so a slow record pauses its partition once the finished records behind it fill the partition's share. A handler failure stops the partition at the lowest failed record, cancels the running records above it, and lets the records below it run, unless Config.stop_failed_partitions is off, in which case it cancels them all and fails the run. The records of other topics run one at a time per partition.

None

Raises:

Type Description
ValueError

If topics is empty, a topic name is invalid, or key_concurrency names a topic outside topics or a limit that is not a positive integer.

TypeError

If topics is a string, or a middleware is not a BaseMiddleware instance.

status

status() -> RuntimeStatus

Return a snapshot of the run.

Any thread may call it, e.g. a metrics exporter's. Off the event loop, the fields may come from slightly different instants.

liveness

liveness(*, slack: float = 30.0) -> str | None

Check for failures that a restart fixes, with potik.core.health.liveness.

Serve it as the liveness probe. Any thread may call it.

Parameters:

Name Type Description Default
slack float

Seconds on top of each limit.

30.0

Returns:

Type Description
str | None

Why the probe fails, or None if it passes.

readiness

readiness() -> str | None

Check that the run is running, with potik.core.health.readiness.

Serve it as the readiness probe. Any thread may call it.

Returns:

Type Description
str | None

Why the probe fails, or None if it passes.

run async

run(*, task_status: TaskStatus[None] = anyio.TASK_STATUS_IGNORED, handle_signals: bool = True) -> None

Run until stopped or failed, and raise the run's failure, if any.

Cancelling it shuts the run down gracefully.

Parameters:

Name Type Description Default
task_status TaskStatus[None]

Lets await task_group.start(runtime.run) return once the run is ready.

TASK_STATUS_IGNORED
handle_signals bool

Whether the first SIGTERM or SIGINT shuts the run down gracefully, after which run() returns. Later signals are ignored, so a stuck shutdown needs SIGKILL, which Kubernetes sends after the termination grace period. It requires the main thread on Unix, and only one run in the process may handle signals at a time. Turn it off in tests, to handle signals yourself, or to run several runtimes at once, which you then cancel on a signal.

True

Raises:

Type Description
RuntimeError

If the runtime has already run, or it handles signals outside the main thread or while another run in the process handles them.

Delivery

Bases: NamedTuple

Where the broker appended a sent record.

PartitionStatus dataclass

PartitionStatus(*, topic: str, partition: int, buffered_records: int, running_handlers: int, paused: bool, handler_started_at: float | None, failure: Exception | None)

A snapshot of one assigned partition.

Attributes:

Name Type Description
topic str

The partition's topic.

partition int

The partition's number.

buffered_records int

The number of fetched records that have not started, including those that wait for an earlier record of their key.

running_handlers int

The number of handler calls that run. It is at most 1, unless the topic runs by key (see Runtime).

paused bool

Whether fetching is paused, to limit the buffer or because the partition is stopped.

handler_started_at float | None

The time.monotonic() time when the oldest running handler call started, or None if no call runs.

failure Exception | None

The exception that stopped the partition (see Config.stop_failed_partitions), or None.

ProducerRecord

Bases: NamedTuple, Generic[_V_co]

A record to produce.

A handler returns it as an output, which commits in one transaction with the input record's offset. Producer.send() sends it at once instead.

ProducerRecord[V] marks a record that a Topic[V] made, for type checkers only. A record made directly is a ProducerRecord[Any].

Attributes:

Name Type Description
topic str

The topic's name.

key bytes | None

The encoded key, or None for a null key.

value bytes | None

The encoded value, or None for a tombstone.

headers Headers | None

The record's headers.

partition int

The record's partition, or -1 to let the producer's partitioner pick one from the key. librdkafka's default partitioner hashes keys differently from the Java client's. To match Java producers, set the partitioner option to murmur2_random.

RuntimeStatus dataclass

RuntimeStatus(*, running: bool = False, polled_at: float | None = None, oldest_uncommitted_at: float | None = None, failure: Exception | None = None, partitions: tuple[PartitionStatus, ...] = ())

A snapshot of a run, from Runtime.status().

Times are time.monotonic() values. potik.core.health checks a snapshot for liveness and readiness probes.

Attributes:

Name Type Description
running bool

Whether the run has started and is neither stopping nor stopped.

polled_at float | None

When the consumer's last poll returned, or when polling started, or None before the run starts. The runtime polls at least every 200ms, except during a rebalance or a shutdown.

oldest_uncommitted_at float | None

When the oldest batch that waits for its transaction got its first handled record or policy-selected offset, or None if no batch waits.

failure Exception | None

The error that stopped the run, or None. An error of a middleware's lifespan is not recorded here, but run() raises it.

partitions tuple[PartitionStatus, ...]

The assigned partitions.

Transaction dataclass

Transaction(offsets: Mapping[tuple[str, int], int], outputs: Sequence[ProducerRecord])

The outputs and offsets of a batch of handled records, which commit atomically.

Middleware sees it in the commit stage (see BaseMiddleware.commit).

Attributes:

Name Type Description
offsets Mapping[tuple[str, int], int]

The offset to resume from of each (topic, partition), one past its last handled record. It may instead be the start offset that the offset policy picked for a new assignment, with no handled records.

outputs Sequence[ProducerRecord]

The records to produce.

resume_or_earliest

resume_or_earliest(context: OffsetContext) -> int

Resume the committed offset, or start at the beginning of the log.

resume_or_fail

resume_or_fail(context: OffsetContext) -> int

Resume the committed offset, or fail the run with InvalidOffsetError.

resume_or_latest

resume_or_latest(context: OffsetContext) -> int

Resume the committed offset, or start at the end of the log.

potik.core.health

Liveness and readiness checks of a RuntimeStatus, to serve over any probe transport, e.g. HTTP.

Runtime.liveness() and Runtime.readiness() call them. Each check returns why the probe fails, or None if it passes. A reason names at most an exception's type, never its message, so it is safe to expose.

liveness

liveness(status: RuntimeStatus, config: Config, *, now: float | None = None, slack: float = 30.0) -> str | None

Fail the probe when a restart would help, i.e. the run failed, or it stopped polling or committing.

Polling counts as stuck when no poll has returned for rebalance_timeout_ms plus slack. Committing counts as stuck when a handled record stays uncommitted for commit_interval_ms plus twice transaction_timeout_ms, plus slack.

A broker outage alone does not fail the check, because a restart cannot fix it. The check fails only if the outage fails the run, e.g. by timing out a transaction. The check also passes while a run starts or stops, unless the run failed. A handler may run for any time, so the check ignores how long it runs. It ignores stopped partitions too, because a restart would run the failed record again, which usually fails again, and stop every other partition meanwhile. Alert on them instead, e.g. with MetricsMiddleware.

Parameters:

Name Type Description Default
status RuntimeStatus

The runtime's status, e.g. from Runtime.status().

required
config Config

The run's config.

required
now float | None

The time.monotonic() time to check at. Defaults to the current time.

None
slack float

Seconds added to each limit, to allow for a slow event loop and slow commit middleware.

30.0

Returns:

Type Description
str | None

Why the probe fails, or None if it passes.

readiness

readiness(status: RuntimeStatus) -> str | None

Pass the probe while the run is running, whether or not it has assigned or stopped partitions.

A consumer serves no requests, so readiness only paces rollouts and disruption budgets. It ignores partitions, because an instance may get none, e.g. with more replicas than partitions.

Returns:

Type Description
str | None

Why the probe fails, or None if it passes.

Middleware

potik.middleware

The built-in middleware: exception rules, the topic check, and concurrency limits.

Tracing and metrics are in potik.middleware.tracing and potik.middleware.metrics, and need the otel extra.

Backoff

Backoff = Callable[[ExceptionContext[Any]], float]

Returns the seconds to wait before the next attempt, e.g. from a Retry-After value that the exception carries.

Decision

Decision = RetryAfter | Sequence[ProducerRecord] | Reraise | None

What a rule returns.

It returns RetryAfter to call the handler again, a list or tuple of records to commit as the record's outputs, Reraise to let the exception propagate, or None to ask the next rule.

Match

Match = type[ExcT] | tuple[type[ExcT], ...] | Callable[[Exception], bool]

The exceptions that on() applies a rule to: a type, a tuple of them as in an except clause, or a predicate.

Rule

Rule = Callable[[ExceptionContext[ExcT]], Decision | Awaitable[Decision]]

A sync or async function of an ExceptionContext that returns a Decision.

ConcurrencyMiddleware

ConcurrencyMiddleware(*, max_concurrency: int | None = None, per_topic: Mapping[str, int] | None = None)

Bases: BaseMiddleware

Limits how many handler calls run at once across partitions, in total and per topic.

A call takes a slot of its topic first, then a shared one, so a call that waits for its topic holds no shared slot. Only the middleware after it runs with a slot, so put it last. Exception rules before it then do not hold a slot during retry backoff. potik.build_runtime() adds one from its own max_concurrency and its handlers', after the exception rules.

Parameters:

Name Type Description Default
max_concurrency int | None

The maximum number of handler calls that run at once across all topics, e.g. the size of a database pool, or None for no limit.

None
per_topic Mapping[str, int] | None

The maximum number of handler calls that run at once for each topic. Topics missing from it have no limit of their own.

None

Raises:

Type Description
ValueError

If a limit is not a positive integer, or a topic name is invalid.

ExceptionContext dataclass

ExceptionContext(*, exception: ExcT, record: ConsumerRecord, attempt: int)

What an exception rule decides on: the exception, the record, and the attempt.

Attributes:

Name Type Description
exception ExcT

The handler's exception. For an ExceptionGroup that holds a single exception, e.g. from a task group in which one task failed, it is that exception.

record ConsumerRecord

The input record.

attempt int

The number of handler calls of the record so far, including the failed one.

ExceptionMiddleware

ExceptionMiddleware(*rules: Rule[Any])

Bases: BaseMiddleware

Retries or recovers from a handler's exceptions by asking rules in order, like except clauses.

With potik.build_runtime(), pass the rules as exception_rules instead of creating this middleware. Build rules with on(), retry(), retry_forever(), skip(), reraise(), and dead_letter().

The first rule that returns something other than None decides. RetryAfter, e.g. from retry(), calls the handler again after its delay. Records, e.g. from skip() or dead_letter(), commit as the record's outputs in place of the failure. Reraise, from reraise(), lets the exception propagate. An exception that no rule handles propagates too, and stops the partition, or fails the run (see Config.stop_failed_partitions).

Rules see the single exception of an ExceptionGroup that holds only one, so on(TimeoutError, ...) matches a TimeoutError from a task group in the handler. A group of several exceptions stays a group, which on(ExceptionGroup, ...) or a function matches. Either way, an exception that no rule handles propagates as the handler raised it.

A partition handles one record at a time, so a retry holds up its partition until the record succeeds or a later rule handles it. With key concurrency (see Runtime), a retry holds up only its key, though the records after it commit only once it succeeds or a rule handles it. A retry() rule that has used up its attempts returns None, so a later rule handles the exception. So does a retry() rule for a PermanentError, e.g. a DecodeError, which would fail the same way again.

The middleware's output_topics collects its rules' (see DeclaresOutputs), e.g. the dead letter topics of dead_letter() rules, so that the topic check covers them.

With the otel extra, the middleware records each exception that it retries or handles as an exception event on the current span. Inside TracingMiddleware, that span is the record's process span.

Parameters:

Name Type Description Default
*rules Rule[Any]

Sync or async functions of an ExceptionContext that return a Decision, asked in order.

()

Raises:

Type Description
TypeError

If a rule is a type or is not callable.

Example
ExceptionMiddleware(
    on(is_transient, retry(max_attempts=5)),
    on(Exception, dead_letter(dead_letters)),
)

ExponentialBackoff dataclass

ExponentialBackoff(*, initial_seconds: float = 0.1, max_seconds: float | None = 30.0, multiplier: float = 2.0, jitter: bool = False)

A backoff whose delay grows by a factor after each attempt, up to a cap.

Parameters:

Name Type Description Default
initial_seconds float

The delay after the first attempt.

0.1
max_seconds float | None

The longest delay, or None for no cap. The default cap keeps retry_forever() retrying at least every 30 seconds during a long outage.

30.0
multiplier float

The factor that multiplies the delay after each attempt.

2.0
jitter bool

Whether each wait is a random time between zero and the computed delay, so that partitions do not retry in step.

False

Raises:

Type Description
ValueError

If a delay is negative, or multiplier is less than 1.

Reraise dataclass

Reraise()

A decision to ask no more rules and let the exception propagate, like a bare raise in an except clause.

The exception then stops the partition, or fails the run (see Config.stop_failed_partitions).

RetryAfter dataclass

RetryAfter(seconds: float)

A decision to call the handler again after seconds.

DeclaresOutputs

Bases: Protocol

A middleware or exception rule with output_topics, the topics that it writes records to.

dead_letter() rules declare their dead letter topic this way, for example. potik.build_runtime() adds the declared topics of its middleware and exception rules to its topic check. They do not restrict where records go.

output_topics property

output_topics: Sequence[str]

The names of the topics that it writes records to.

NamedTopic

Bases: Protocol

Any object with a topic's name, e.g. a Topic. APIs that take topics take one in place of a name.

name property

name: str

The topic's name.

TopicCheckError

TopicCheckError(missing: tuple[str, ...], unauthorized: tuple[str, ...])

Bases: PotikError

Raised by TopicCheckMiddleware when topics are missing, or the client is not authorized to describe them.

Attributes:

Name Type Description
missing

The missing topics.

unauthorized

The topics that the client is not authorized to describe.

TopicCheckMiddleware

TopicCheckMiddleware(topics: Iterable[NamedTopic | str], *, timeout: float = 10.0)

Bases: BaseMiddleware

Checks that topics exist before a run or a Producer starts, and fails the start otherwise.

Put it first, so that a missing topic fails the start before other lifespans open resources. potik.build_runtime() puts one first, with its handled topics and every output topic that its handlers, routers, middleware, and exception rules declare, unless check_topics=False.

The client needs permission to describe each topic. A missing or denied topic fails the start with TopicCheckError, and other broker errors, e.g. a timeout, with KafkaException. Cancelling the start, e.g. on SIGTERM, stops the check. It never creates a missing topic.

Parameters:

Name Type Description Default
topics Iterable[NamedTopic | str]

The topics to check, as Topics or names, e.g. the topics that a run subscribes to and returns outputs to, or that a Producer sends to.

required
timeout float

How long the metadata request may take, including finding a broker, in seconds.

10.0

Raises:

Type Description
ValueError

If a topic name is invalid.

dead_letter

dead_letter(topic: NamedTopic | str, *, headers: Callable[[ExceptionContext[Exception]], Iterable[tuple[str, str | bytes | None]]] = dead_letter_headers) -> Rule[Exception]

Write a copy of the failed record to topic, as the record's outputs.

The copy commits in the same transaction as the failed record's offset. It keeps the record's key, value, and headers as bytes, even if they do not decode, and adds the headers that headers returns. The producer's partitioner picks its partition from the key, because the dead letter topic may have fewer partitions than the input's.

The rule declares topic as its output topic, so the topic check covers it. To write a different dead letter, write a rule of your own and declare its topic, e.g. in Router(output_topics=...).

Parameters:

Name Type Description Default
topic NamedTopic | str

The dead letter topic, as a Topic or a name.

required
headers Callable[[ExceptionContext[Exception]], Iterable[tuple[str, str | bytes | None]]]

Returns the headers to add, e.g. names that your dead letter consumers expect, or dead_letter_headers without the exception message. Defaults to dead_letter_headers.

dead_letter_headers

dead_letter_headers

dead_letter_headers(context: ExceptionContext[Exception]) -> list[tuple[str, str | None]]

Return the headers that dead_letter() adds by default.

  • potik.dlq.source.topic, potik.dlq.source.partition, and potik.dlq.source.offset: where the record came from.
  • potik.dlq.exception.type: the exception's qualified class name.
  • potik.dlq.exception.message: str() of the exception, which may contain data from the record. A message longer than 1000 characters is cut, and ends with ....
  • potik.dlq.attempts: the number of handler calls.

Parameters:

Name Type Description Default
context ExceptionContext[Exception]

The failed record and its exception.

required

on

on(match: Match[ExcT], rule: Rule[ExcT], *, topics: Iterable[NamedTopic | str] | None = None) -> Rule[Exception]

Apply rule only to matching exceptions, and optionally only to some topics.

For other exceptions and topics, the returned rule returns None, so the next rule decides.

Parameters:

Name Type Description Default
match Match[ExcT]

An exception type, a tuple of them as in an except clause, or a function that takes the exception and returns whether it matches.

required
rule Rule[ExcT]

The rule to apply.

required
topics Iterable[NamedTopic | str] | None

The topics whose records the rule applies to, as Topics or names, or None for every topic.

None

Raises:

Type Description
TypeError

If match names something other than Exception subclasses.

ValueError

If a topic name is invalid.

Example
on(TimeoutError, retry(max_attempts=5), topics=[orders])

reraise

reraise() -> Rule[Exception]

Ask no more rules and let the exception propagate, so that it stops the partition.

Put it before a rule that handles every exception, e.g. on(InvalidOutputError, reraise()) before a dead_letter() rule. With Config(stop_failed_partitions=False), the exception fails the run instead.

retry

retry(*, max_attempts: int, backoff: Backoff | None = None) -> Rule[Exception]

Retry until the handler has run max_attempts times, then pass the exception to the next rule.

A PermanentError, e.g. a DecodeError, goes straight to the next rule, because retrying cannot fix it.

Parameters:

Name Type Description Default
max_attempts int

The maximum number of handler calls of a record, including the first.

required
backoff Backoff | None

Returns the seconds to wait before each retry. Defaults to ExponentialBackoff().

None

Raises:

Type Description
ValueError

If max_attempts is not a positive integer.

retry_forever

retry_forever(*, backoff: Backoff | None = None) -> Rule[Exception]

Retry until the handler succeeds.

Like retry(), it passes a PermanentError to the next rule. Its partition makes no progress until the record succeeds.

Parameters:

Name Type Description Default
backoff Backoff | None

Returns the seconds to wait before each retry. Defaults to ExponentialBackoff().

None

skip

skip() -> Rule[Exception]

Drop the failed record: commit its offset with no outputs.

potik.middleware.tracing

Trace records with OpenTelemetry, from their handling to the transaction that commits their outputs, and sends through a Producer.

Needs the otel extra.

TracingMiddleware

TracingMiddleware(*, optional_attributes: Collection[str] = DEFAULT_OPTIONAL_ATTRIBUTES, key_text: Callable[[bytes], str | None] = _utf8, transaction_sample_ratio: float = 1.0, tracer_provider: TracerProvider | None = None)

Bases: BaseMiddleware

Traces each record's handling, its outputs to the transaction that commits them, and each Producer send.

It creates these spans:

  • process {topic} (consumer), one per handler call. It continues the trace in the record's headers. A record that runs again, e.g. after a crash, gets a second process span in the same trace.
  • create {topic} (producer), one per output, a child of its record's process span. The output's headers carry its context. It ends at once, because backends count producer spans as dependency latency, and the transaction's span covers the produce.
  • send {topic} (client), one per transaction, as the root of its own trace. A transaction without outputs is named commit {topic}, and the topic is left out of the name when there are several. It links to the sampled create span of each output, up to the SDK's link limit. These links connect a record's trace to the transaction that commits its outputs. Only a committed handling's outputs reach downstream consumers, so when a record runs again, its committed process span is the one whose create spans have a link and downstream children. A record without outputs has no link to its transaction.
  • send {topic} (producer), one per Producer.send(), a child of the current span, e.g. a handler's process span or an API's request span. The record's headers carry its context. It lasts until the broker acknowledges the record.

Each span has the attributes that the OpenTelemetry messaging semantic conventions require for it, e.g. messaging.consumer.group.name on a run's spans. optional_attributes picks the optional ones. To add attributes of your own, or to rename a span, use trace.get_current_span() in a handler or in middleware after this one. The current span there is the process span, the transaction's span, or the send span.

Use one instance for each runtime and each producer. Put it first in the runtime's middleware, so that its spans cover the other middleware. potik.build_runtime() runs the exception rules inside its middleware, so one process span covers a record's retries and the outputs of exception rules, such as dead letters. In a core Runtime, put it before ExceptionMiddleware for the same span. A cancellation, e.g. by a revocation, ends the process span with a cancelled event instead of an error.

A transaction's span is the root of its own trace, so the tracer provider's sampler decides on it apart from the records in it. To sample transactions, use transaction_sample_ratio with a sampler that keeps root spans.

Parameters:

Name Type Description Default
optional_attributes Collection[str]

The optional attributes to add, where they apply. The default, TracingMiddleware.DEFAULT_OPTIONAL_ATTRIBUTES, is messaging.destination.partition.id, messaging.kafka.offset, and messaging.client.id if the client's client.id is set. messaging.kafka.message.key is off by default, because keys often identify people, and so is messaging.message.body.size, which the conventions make opt-in.

DEFAULT_OPTIONAL_ATTRIBUTES
key_text Callable[[bytes], str | None]

Returns the text of a key for messaging.kafka.message.key, or None to leave the key out. The default decodes UTF-8 and leaves out keys that are not valid UTF-8.

_utf8
transaction_sample_ratio float

The share of transactions that get a span, from 0 to 1. A failed transaction always gets one, but if it was not picked, its span is not the current span while the transaction runs. The tracer provider's sampler still decides whether to keep each span.

1.0
tracer_provider TracerProvider | None

Defaults to the global tracer provider.

None

Raises:

Type Description
ValueError

If optional_attributes names an attribute that this middleware does not add, or transaction_sample_ratio is not between 0 and 1.

DEFAULT_OPTIONAL_ATTRIBUTES class-attribute

DEFAULT_OPTIONAL_ATTRIBUTES: frozenset[str] = frozenset({MESSAGING_DESTINATION_PARTITION_ID, MESSAGING_KAFKA_OFFSET, MESSAGING_CLIENT_ID})

The optional attributes that it adds by default.

potik.middleware.metrics

Measure record handling, transactions, and sends through a Producer with OpenTelemetry metrics.

Needs the otel extra.

MetricsMiddleware

MetricsMiddleware(*, optional_attributes: Collection[str] = DEFAULT_OPTIONAL_ATTRIBUTES, meter_provider: MeterProvider | None = None)

Bases: BaseMiddleware

Measures handler calls, transactions, and Producer sends with the OpenTelemetry messaging metrics.

  • messaging.client.consumed.messages counts the records that the run passes to its handler, per topic. Its messaging.operation.name is process, the operation that delivers the records to the application.
  • messaging.process.duration measures each handler call, per topic, with error.type if it raised.
  • messaging.client.operation.duration measures each transaction, including its retries after a rebalance, with error.type if it failed. Its operation is send, or commit if it has no outputs.
  • messaging.client.sent.messages counts each transaction's outputs, per topic, with error.type if the transaction failed.
  • For each Producer.send(), messaging.client.operation.duration measures the whole call, including any wait for queue room, and messaging.client.sent.messages adds one, per topic.
  • potik.partition.stopped counts the stopped partitions, per topic (see Config.stop_failed_partitions). The probes ignore them (see potik.core.health), so alert when it stays above zero. The semantic conventions do not define this metric.

Each metric has the attributes that the conventions require for it, e.g. messaging.consumer.group.name on a run's durations. optional_attributes picks the optional ones. To rename a metric, drop an attribute, or change the buckets, configure a View in the SDK's MeterProvider.

Use one instance for each runtime and each producer. A handler call's duration includes its retries, because potik.build_runtime() runs the exception rules inside its middleware. In a core Runtime, put it before ExceptionMiddleware for the same duration. Cancelled handler calls, transactions, and sends record no duration and no sent count, but a cancelled handler call still counts as consumed.

Parameters:

Name Type Description Default
optional_attributes Collection[str]

The optional attributes to add, where they apply. The default is MetricsMiddleware.DEFAULT_OPTIONAL_ATTRIBUTES: messaging.destination.partition.id.

DEFAULT_OPTIONAL_ATTRIBUTES
meter_provider MeterProvider | None

Defaults to the global meter provider.

None

Raises:

Type Description
ValueError

If optional_attributes names an attribute that this middleware does not add.

DEFAULT_OPTIONAL_ATTRIBUTES class-attribute

DEFAULT_OPTIONAL_ATTRIBUTES: frozenset[str] = frozenset({MESSAGING_DESTINATION_PARTITION_ID})

The optional attributes that it adds by default.

Typed API

potik

Typed handlers for Kafka topics.

Declare each Topic with the types and serdes of its keys and values, register handlers on a Router, and build a core Runtime from the routers with build_runtime(). The package also re-exports the core names that services use.

HandlerResult module-attribute

HandlerResult = TypeAliasType('HandlerResult', ProducerRecord[_V] | Sequence[ProducerRecord[_V]] | None, type_params=(_V,))

What a typed handler returns: None, a ProducerRecord, or a list or tuple of them.

potik commits the records with the input record's offset, in one transaction.

Router

Router(*, output_topics: Iterable[OutputTopic[Any] | str] = ())

A group of topic handlers, which potik.build_runtime() builds a runtime from.

Register handlers with @router.handler(topic). A router needs no runtime, so a module can define one at import, and tests can call its handlers directly. A router has no middleware of its own. To apply an exception rule to some topics only, use on(..., topics=...), and in middleware, check record.topic().

Parameters:

Name Type Description Default
output_topics Iterable[OutputTopic[Any] | str]

Topics that records go to beyond the handlers' own output_topics, e.g. the topic of your own exception rule. potik.build_runtime() checks that they exist.

()

Raises:

Type Description
RegistrationError

If an output is not a valid topic name.

handler

handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None) -> Callable[[H], H]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[V]]) -> Callable[[Callable[P, Awaitable[HandlerResult[V]]]], Callable[P, Awaitable[HandlerResult[V]]]]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[V] | str]) -> Callable[[Callable[P, Awaitable[HandlerResult[V | bytes]]]], Callable[P, Awaitable[HandlerResult[V | bytes]]]]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[Any] | str] = ()) -> Callable[[_UserHandler], _UserHandler]

Register an async function as the handler of topic, and return the function unchanged.

Each of the handler's parameters needs a type annotation, and takes one part of the input record:

  • Annotated[T, Key()] takes the key, as the topic's key type, or as bytes for the raw key.
  • Annotated[T, Header(name)] takes a header's last value.
  • Annotated[T, Context()] takes the run's context.
  • Annotated[T, Depends(dependency)] takes what a dependency returns or yields (see Depends).
  • A parameter annotated ConsumerRecord takes the record itself.
  • The one parameter left, if any, takes the value, as the topic's value type, or as bytes for the raw value.

The handler returns None, a ProducerRecord, or a list or tuple of them. Anything else raises InvalidOutputError from the handler's call.

Parameters:

Name Type Description Default
topic str | Topic[Any, Any]

The handled topic, as a Topic, or the name of a topic of bytes values and str keys.

required
max_concurrency int | None

The maximum number of calls of this handler that run at once, across the topic's partitions, or None for no limit.

None
key_concurrency int | None

The maximum number of the topic's keys in progress at once, across its partitions, or None to handle each partition's records one at a time. Records with the same key still run one at a time, in offset order. A key stays in progress during its retries' backoff, so set max_concurrency too, to limit the handler calls, e.g. key_concurrency=64, max_concurrency=20 for a pool of 20 connections. See the key_concurrency of potik.core.Runtime.

None
output_topics Iterable[OutputTopic[Any] | str]

The topics that the handler returns records to, as Topics or names of bytes topics. potik.build_runtime() checks that they exist before the run starts. Type checkers then require the handler to return records of their value types, e.g. ProducerRecord[Accepted]. The handler may still return records to other topics, which the check does not cover.

()

Raises:

Type Description
RegistrationError

If the router already handles topic, a limit is not a positive integer or None, the handler is not an async function, a parameter of the handler or of a dependency does not fit the topic, e.g. its type differs from the topic's value type, or a dependency depends on itself or is decorated as a context manager.

Example
@router.handler(orders, output_topics=[accepted])
async def accept(order: Order, customer: Annotated[str, Key()]) -> ProducerRecord[Accepted]:
    return accepted(Accepted(order.id), key=customer)

RegistrationError

Bases: PotikError

Raised when a router cannot register a handler, or potik.build_runtime() cannot build a runtime from routers.

Topic

Topic(name: str, value_type: TypeForm[_V], *, serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)
Topic(name: str, value_type: TypeForm[_V], *, key_type: TypeForm[_K], serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)
Topic(name: str, value_type: Any, *, key_type: Any = str, serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)

Bases: Generic[_V, _K]

A Kafka topic's name, and the types and serdes of its values and keys.

Declare each topic once, e.g. in a module that its producers and consumers share, so that they all use the same wire format. Pass it to Router.handler() to handle it, and call it to make a ProducerRecord to it. Type checkers infer its types from the given ones, e.g. Topic("events", A | B, serde=json) is a Topic[A | B]. A topic is immutable.

The declaration gets its decoders and encoders from the serdes, so a serde that does not support a type fails here, not on the first record.

A tombstone, a record with a null value, has no value to decode. A handler's value parameter takes its default for it, else None if its type allows None, e.g. Order | None for a Topic[Order]. Otherwise the handler call raises DecodeError.

Parameters:

Name Type Description Default
name str

The topic's name.

required
value_type Any

The type of the topic's values.

required
key_type Any

The type of the topic's keys. Defaults to str.

str
serde Serde | AsyncSerde

The Serde or AsyncSerde of the values. Defaults to Text().

_TEXT
key_serde Serde | AsyncSerde

The Serde or AsyncSerde of the keys. Defaults to Text().

_TEXT

Raises:

Type Description
TypeError

If a serde is neither a Serde nor an AsyncSerde, or both, or does not support its type.

ValueError

If name is not a valid topic name.

Example
json = PydanticJson()
orders = Topic("orders", Order, serde=json)  # Topic[Order], str keys
by_number = Topic("orders.by_number", Order, key_type=int, serde=json)

decode async

decode(data: bytes) -> _V

Decode a value of this topic, e.g. in middleware.

A failure raises the serde's own exception, not DecodeError.

decode_key async

decode_key(data: bytes) -> _K

Decode a key of this topic, e.g. in middleware.

A failure raises the serde's own exception, not DecodeError.

OutputTopic

Bases: NamedTopic, Protocol

The type of the topics in a handler's output_topics, e.g. a Topic[V].

You rarely name it yourself. Unlike Topic, it is covariant in V, so type checkers accept output_topics=[a, b], with a: Topic[A] and b: Topic[B], as outputs of A | B.

Key dataclass

Key()

Marks a parameter, Annotated[T, Key()], that takes the record's key.

T is the topic's key type, or bytes for the raw key. A null key takes the parameter's default if it has one, else None if T allows None, else the call fails with DecodeError.

Header dataclass

Header(name: str, decoder: Decoder | None = None)

Marks a parameter, Annotated[T, Header(name)], that takes a header's value.

If the record has the header more than once, the parameter takes the last value. A missing or null header takes the parameter's default if it has one, else None if T allows None, else the call fails with DecodeError.

Parameters:

Name Type Description Default
name str

The header's name.

required
decoder Decoder | None

Decodes the header's value. Defaults to Text's decoder of T, and bytes takes the value as it is.

None

Context dataclass

Context()

Marks a parameter, Annotated[T, Context()], that takes the run's context.

The context is the object passed as potik.build_runtime(context=...), e.g. a dataclass of the service's clients. potik does not check it against T. potik.build_runtime() raises RegistrationError if a handler or one of its dependencies takes it and no context is given.

Depends dataclass

Depends(dependency: Callable[..., object])

Marks a parameter, Annotated[T, Depends(dependency)], that takes what dependency returns.

A dependency is a function, sync or async, a class, or a callable object, which must be hashable. Its parameters take parts of the record, the run's context, or other dependencies, as a handler's parameters do, and registration checks them against the topic in the same way. A call decodes each part of the record once, e.g. the value, for every parameter that takes it with the same decoder, so they all get the same object. Each one still takes its own default for a null part. A sync dependency runs on the event loop, so it must not block. potik does not check what a dependency returns against T.

A dependency that yields, sync or async, gives the parameter the value that it yields, and runs the rest of its body after the handler returns or raises, e.g. to commit or roll back a transaction. Do not decorate it with @contextmanager or @asynccontextmanager: registration rejects such a function, because potik enters dependencies that yield itself.

Each handler call resolves the handler's dependencies anew, in the call's task, one at a time, in parameter order, and each one after its own dependencies. A dependency runs once per call, even if several parameters take it, in the handler and in its dependencies, or take dependencies equal to it, e.g. the same bound method. A retry of the call by an exception rule resolves them again.

After the call, potik resumes the dependencies that yield, last entered first. Each one resumes even if one resumed before it failed:

  • If the handler, or a dependency, raised, yield raises that exception in the dependency. A dependency cannot suppress it: potik raises it even if the dependency catches it, so exception rules still handle it. The one exception is a cancellation by the dependency's own cancel scope around yield: with anyio.move_on_after(), a call that runs out of time returns no outputs, so its record commits, unless the call raised another exception, which it then raises. With anyio.fail_after(), it raises TimeoutError. The scope also covers the code after yield of the dependencies that potik entered after it. A dependency that swallows an outer cancellation, e.g. a revocation's, suppresses nothing: the call still raises it.
  • If a dependency raises after yield, the call fails with that exception. Exception rules then handle it as a handler failure, and the handler's outputs are not committed.
  • If a revocation or shutdown cancels the call, the code after yield runs in the cancelled scope, where each await raises the cancellation again. Shield the awaits that must finish, with a timeout, e.g. with anyio.move_on_after(5, shield=True):.

The code after yield runs before the transaction commits the handler's outputs. So a write that it commits, e.g. to a database, happens at least once, not exactly once.

Parameters:

Name Type Description Default
dependency Callable[..., object]

The function or class to call.

required

Raises:

Type Description
TypeError

If dependency is not callable.

Example
async def transaction(services: Annotated[Services, Context()]) -> AsyncIterator[Transaction]:
    tx = await services.db.begin()
    try:
        yield tx
    except BaseException:
        with anyio.move_on_after(5, shield=True):
            await tx.rollback()
        raise
    await tx.commit()


@router.handler(orders)
async def save(order: Order, tx: Annotated[Transaction, Depends(transaction)]) -> None:
    await tx.execute("INSERT INTO orders VALUES ($1)", order.sku)

DecodeError

DecodeError(message: str, *, parameter: str)

Bases: PotikError, PermanentError

Raised from a handler call when a parameter, of the handler or of a dependency, cannot take its part of the record.

The part fails to decode, or it is missing or null and the parameter has no default and does not allow None. It is a PermanentError, so retry() rules do not retry it.

Attributes:

Name Type Description
parameter

The parameter's name.

EncodeError

EncodeError(message: str, *, topic: str)

Bases: PotikError, PermanentError

Raised when a topic cannot encode a key or value.

The key or value has the wrong type, or the topic's serde fails on it. It is a PermanentError, so retry() rules do not retry it.

Attributes:

Name Type Description
topic

The topic's name.

build_runtime

build_runtime(config: Config, routers: Iterable[Router], *, context: object = _NO_CONTEXT, middleware: Iterable[BaseMiddleware] = (), exception_rules: Iterable[Rule[Exception]] = (), max_concurrency: int | None = None, check_topics: bool = True) -> Runtime

Build a Runtime that runs the routers' handlers.

Open the resources that handlers need first, e.g. a database pool or a Producer, then build the runtime with them and run it. A runtime runs once, so another run needs a new runtime with new middleware instances.

Each record passes through middleware, then the exception rules, then the concurrency limits, then its handler.

Parameters:

Name Type Description Default
config Config

The runtime limits and client options.

required
routers Iterable[Router]

The routers whose handlers to run. The runtime copies their handlers, so later registrations have no effect.

required
context object

What Context() parameters take, of handlers and of their dependencies, e.g. a dataclass of the service's clients.

_NO_CONTEXT
middleware Iterable[BaseMiddleware]

Middleware instances, outermost first (see BaseMiddleware). Pass exception rules as exception_rules, not as an ExceptionMiddleware.

()
exception_rules Iterable[Rule[Exception]]

Rules that retry or recover from handler exceptions, tried in order (see ExceptionMiddleware). They run inside middleware, so e.g. tracing and metrics see one call per record, including its retries. An exception that no rule handles stops the record's partition, or fails the run (see Config.stop_failed_partitions). To apply a rule to some topics only, use on(..., topics=...).

()
max_concurrency int | None

The maximum number of handler calls that run at once, across all partitions and topics, e.g. the size of a database pool, or None for no limit. Like each handler's own max_concurrency, it limits only handler calls, so retry backoff holds no slot.

None
check_topics bool

Whether run() first checks that the topics exist, with a TopicCheckMiddleware before middleware. It checks the handled topics and the output topics of the handlers, routers, middleware, and exception rules. A missing topic fails the run with TopicCheckError. Turn it off e.g. to start before the topics are created.

True

Returns:

Type Description
Runtime

The runtime. Call its run() to run it.

Raises:

Type Description
RegistrationError

If the routers have no handlers, two routers handle the same topic, a handler or one of its dependencies takes Context() and there is no context, or middleware includes an ExceptionMiddleware.

TypeError

If a router is not a Router, a middleware is not a BaseMiddleware instance, or an exception rule is a type or is not callable.

ValueError

If max_concurrency is not a positive integer or None.

Example
async def main() -> None:
    async with create_pool() as db:
        await potik.build_runtime(
            config,
            [router],
            context=db,
            middleware=[TracingMiddleware()],
            exception_rules=[
                on(TimeoutError, retry(max_attempts=5)),
                on(Exception, dead_letter(dead_letters)),
            ],
        ).run()

potik.serde

Serdes, which convert a topic's keys and values between bytes and Python types.

A Serde returns a decoder and an encoder for each type it supports, and None for the others. A SerdeContext tells it which topic's keys or values they serve. An AsyncSerde returns a resolver in place of a decoder, for formats whose decoding needs I/O, e.g. a schema registry's. Type checkers infer these functions' types from the requested type. A Topic names the serdes of its values and keys, so the wire format never depends on how a handler annotates a type.

Decoders validate the data against the type, as far as the serde's library does. Encoders do not validate.

Decoder module-attribute

Decoder = TypeAliasType('Decoder', Callable[[bytes], _T], type_params=(_T,))

Decodes bytes to a T, e.g. a key, a value, or a header's value.

Encoder module-attribute

Encoder = TypeAliasType('Encoder', Callable[[_T], bytes], type_params=(_T,))

Encodes a T to bytes.

Resolver module-attribute

Resolver = TypeAliasType('Resolver', Callable[[bytes], Awaitable[Decoder[_T]]], type_params=(_T,))

Does the I/O that decoding one record needs, e.g. fetches its writer schema, and returns the record's Decoder.

Serde

Bases: Protocol

Makes the decoders and encoders of the types it supports.

A Topic calls decoder() and encoder() once for its keys and once for its values, when it is declared, usually at import, so they must not do I/O. The functions they return run synchronously on the event loop for every record, so they must not do I/O either. When potik decodes a handler parameter, a decoder's exception becomes a DecodeError. When a Topic call encodes a record, an encoder's exception becomes an EncodeError. Both are permanent, so retry() rules do not retry them. For a format whose decoding needs I/O, write an AsyncSerde.

decoder

decoder(tp: TypeForm[T], context: SerdeContext) -> Decoder[T] | None

Return a function that decodes bytes to a tp of context, or None if this serde does not support tp.

encoder

encoder(tp: TypeForm[T], context: SerdeContext) -> Encoder[T] | None

Return a function that encodes a tp of context to bytes, or None if this serde does not support tp.

AsyncSerde

Bases: Protocol

Makes the resolvers and encoders of the types it supports, for formats whose decoding needs I/O.

A resolver does the I/O that decoding one record needs, e.g. fetching the writer schema that the record names from a registry, and returns the decoder of that record. A resolver's exceptions reach the exception rules as they are, so retry() rules retry e.g. a timeout. Raise a PermanentError for a failure that would repeat, e.g. an unknown schema. The decoder that the resolver returns, and the encoders, work as a Serde's do: they are synchronous, must not do I/O, and their failures are permanent.

A Topic calls resolver() and encoder() once for its keys and once for its values, when it is declared, so they must not do I/O. An encoder whose format needs I/O, e.g. the registry ID of its writer schema, gets it before the run. The serde learns every topic's keys and values when they are declared, so a method of its own, which the service's main awaits before the run, can register their schemas. potik does not open or close a client that the serde uses, e.g. a registry's. Open and close it around the run, e.g. in the service's main.

resolver

resolver(tp: TypeForm[T], context: SerdeContext) -> Resolver[T] | None

Return an async function that returns a record's decoder to a tp of context, or None if this serde does not support tp.

encoder

encoder(tp: TypeForm[T], context: SerdeContext) -> Encoder[T] | None

Return a function that encodes a tp of context to bytes, or None if this serde does not support tp.

SerdeContext dataclass

SerdeContext(topic: str, part: Literal['key', 'value'])

The keys or values of one topic, which a serde makes a decoder and an encoder for.

A serde that depends on the topic reads it, e.g. one that names a schema registry subject after the topic, like Confluent's TopicNameStrategy (orders-value). Other serdes ignore it.

Attributes:

Name Type Description
topic str

The topic's name.

part Literal['key', 'value']

"key" for the topic's keys, "value" for its values.

Text

Converts str as UTF-8, int and UUID as their text, and bytes as they are.

It is the default serde of topics' keys and values, and the default decoder of Header parameters. A subclass counts as its base, e.g. a StrEnum as its value, and a NewType counts as its supertype. It does not support bool, or Annotated types such as Annotated[str, Field(min_length=3)], rather than ignore their constraints.

An int decodes only from ASCII digits with an optional leading -, the form that it encodes to, so different keys never decode to the same int.

PydanticJson

PydanticJson(**dump_options: Any)

JSON through pydantic, for any type that pydantic's TypeAdapter supports.

Decoding validates the data against the type. Encoding serializes without validating, but fails on a value that pydantic would warn about, e.g. an int in a str field. Pass warnings=... to change that. Install the pydantic extra to use it.

Parameters:

Name Type Description Default
**dump_options Any

Options of TypeAdapter.dump_json, e.g. by_alias=True.

{}

Raises:

Type Description
ImportError

If pydantic is not installed.

MsgspecJson

MsgspecJson()

JSON through msgspec, for any type that msgspec supports without dec_hook or enc_hook.

Decoding validates the data against the type. Encoding serializes by the value's own type. Install the msgspec extra to use it.

Raises:

Type Description
ImportError

If msgspec is not installed.

potik.testing

Run a whole app in tests, on an in-process Kafka cluster instead of a broker.

MockCluster

MockCluster()

An in-process Kafka cluster that runs a whole app in a test, without a broker.

The app's own clients connect to it, so its runtime, middleware, and topic check run as in production. Point the app at the cluster with config(), create its topics, run the app, and wait for its outputs with wait_records().

The cluster has one broker. It creates no topic on its own, so a topic that the test did not create fails the app's start. It commits transactions and hides uncommitted outputs from readers. It does not store the offsets that a transaction commits, and it does not fence stale group members. So a restarted app handles its input again, and only a real broker can test what a run committed.

Enter it with with. Exit it only after the app's run and every other client of the test have closed, because a client that outlives the cluster keeps trying to reconnect. It runs librdkafka's mock cluster, on Linux and macOS only.

Raises:

Type Description
RuntimeError

On other platforms, or if the installed confluent-kafka does not export librdkafka's mock cluster API.

Example
async def test_accept() -> None:
    with MockCluster() as cluster:
        cluster.create_topic(orders)
        cluster.create_topic(accepted)
        async with Producer(cluster.bootstrap_servers) as producer:
            await producer.send(orders(Order(sku="A"), key="c1"))
        runtime = build(cluster.config(CONFIG))  # your service's potik.build_runtime(...)
        async with anyio.create_task_group() as tasks:
            await tasks.start(partial(runtime.run, handle_signals=False))
            (record,) = await cluster.wait_records(accepted, 1)
            tasks.cancel_scope.cancel()  # shuts the run down gracefully

bootstrap_servers property

bootstrap_servers: str

The cluster's bootstrap servers, for the app's Config and any other client.

config

config(config: Config) -> Config

Return a copy of config that connects to this cluster, for an app under test.

The copy starts each partition at its first record (resume_or_earliest), so the app also handles the records that the test sent before the partition was assigned. It also sets fetch.wait.max.ms to 10ms, because the mock cluster holds a fetch without records for that whole wait, even when a record arrives.

create_topic

create_topic(topic: NamedTopic | str, partitions: int = 1) -> None

Create topic, e.g. a Topic or a name, with partitions partitions.

Raises:

Type Description
ValueError

If the name is invalid, or partitions is less than 1.

KafkaException

If the cluster refuses the topic, e.g. one that exists.

wait_records async

wait_records(topic: NamedTopic | str, count: int, *, timeout: float = 10.0) -> list[ConsumerRecord]

Read topic until it has at least count committed records and every partition has reached its end.

It returns every record that it read, which may be more than count. A topic that keeps receiving records may time out. A record counts once a read_committed consumer can read it. A Producer send counts at once, and a handler's output once its transaction commits. Each partition's records come in offset order.

Raises:

Type Description
TimeoutError

If the topic has fewer than count committed records, or a partition has not reached its end, after timeout seconds.

KafkaException

If the topic does not exist.