API reference¶
potik also re-exports the core names that services use, such as Config, Producer, Runtime, and the record types. Each name is documented once, under the layer that defines it.
Runtime¶
potik.core ¶
The core runtime, a minimal wrapper around confluent-kafka.
It works without the typed layer: a handler takes a raw ConsumerRecord and returns a list or tuple of ProducerRecords.
OffsetPolicy ¶
OffsetPolicy = Callable[[OffsetContext], int]
A function that returns where a newly assigned partition starts, e.g. resume_or_earliest.
It returns the committed offset or an offset within the log, else the run fails with InvalidOffsetError. If it returns another offset than the committed one, the runtime commits it before handling, so a restart resumes from it. An offset below the committed one handles records again, whose outputs may already be committed, and an offset above it skips records. It runs on the consumer's thread during the rebalance, so it must be synchronous and return quickly. rebalance_timeout_ms bounds the whole assignment.
ConsumerRecord ¶
ConsumerRecord = Message
The native confluent_kafka.Message, not a copy. Its fields are methods, e.g. record.key() and record.offset().
Headers ¶
Headers = list[tuple[str, str | bytes | None]] | dict[str, str | bytes | None]
A record's headers: a list of (name, value) pairs, which keeps order and allows repeated names, or a dict.
Values are str, bytes, or None.
RecordHandler ¶
RecordHandler = Callable[[ConsumerRecord], Awaitable[Sequence[ProducerRecord]]]
A core handler. It takes one record and returns its outputs as a list or tuple, possibly empty.
Returning does not commit. The runtime commits the outputs and the record's offset later, in one transaction with other handled records.
RecordSender ¶
RecordSender = Callable[[ProducerRecord], Awaitable[Delivery]]
The call_next of BaseMiddleware.send. It sends one record and returns once the broker acknowledges it.
Config
dataclass
¶
Config(*, bootstrap_servers: str, group_id: str, offset_policy: OffsetPolicy = resume_or_latest, max_buffered_bytes: int = 16 * 1024 * 1024, stop_failed_partitions: bool = True, commit_batch_size: int = 1000, commit_interval_ms: int = 100, transaction_timeout_ms: int = 10000, rebalance_timeout_ms: int = 60000, revocation_grace_ms: int = 0, shutdown_grace_ms: int = 0, max_poll_interval_ms: int = 300000, transactional_id_prefix: str = 'potik', options: Mapping[str, Any] = dict(), consumer_options: Mapping[str, Any] = dict(), producer_options: Mapping[str, Any] = dict())
Runtime limits and librdkafka client options.
The runtime owns the client settings that its guarantees depend on, e.g. isolation.level and enable.auto.commit, and the options cannot override them. Callbacks in the options, e.g. error_cb, run on the runtime's client threads, so they must be thread-safe functions, not coroutines.
The producer's transactional.id is {transactional_id_prefix}-{random}, new each run. With static membership, i.e. a group.instance.id in the options, it derives from the group and instance IDs instead. A restarted instance then aborts the transaction that its crashed run left open, so read_committed readers do not wait transaction_timeout_ms for it. A second live process with the same instance ID fences the first one's producer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bootstrap_servers
|
str
|
The cluster's bootstrap servers. |
required |
group_id
|
str
|
The consumer group. |
required |
offset_policy
|
OffsetPolicy
|
Picks the start offset of each newly assigned partition (see |
resume_or_latest
|
max_buffered_bytes
|
int
|
The memory of the fetched records that have not gone to a transaction yet, across all partitions, i.e. those that wait for their handler, run, or wait for an earlier record of their partition (see the |
16 * 1024 * 1024
|
stop_failed_partitions
|
bool
|
Whether a failure of one partition stops it, instead of failing the run. A stopped partition handles nothing past the failed record, which stays uncommitted, while other partitions keep running. The record runs again once the partition is assigned again, e.g. after a restart. A failure is a handler exception that no exception rule handles, or a fetch error, e.g. a committed offset that retention deleted. |
True
|
commit_batch_size
|
int
|
The maximum number of handled records per transaction. |
1000
|
commit_interval_ms
|
int
|
The minimum time between the starts of two transactions, unless the batch fills first. A transaction also waits for the previous one to finish, so batches grow with throughput. |
100
|
transaction_timeout_ms
|
int
|
The producer's |
10000
|
rebalance_timeout_ms
|
int
|
The longest that the runtime may take to handle an assignment or a revocation, including the revoked partitions' grace period and final commit. Exceeding it fails the run. Must be less than |
60000
|
revocation_grace_ms
|
int
|
How long in-flight handlers of revoked partitions may run before the runtime cancels them. Their records run again on the partitions' next owner. The default, 0, cancels them at once. Must be less than |
0
|
shutdown_grace_ms
|
int
|
How long in-flight handlers may run on shutdown before the runtime cancels them. The default, 0, cancels them at once. Plus |
0
|
max_poll_interval_ms
|
int
|
The consumer's |
300000
|
transactional_id_prefix
|
str
|
The start of the producer's |
'potik'
|
options
|
Mapping[str, Any]
|
librdkafka options of both clients, e.g. for the connection, TLS, SASL, OAuth, |
dict()
|
consumer_options
|
Mapping[str, Any]
|
librdkafka options of the consumer, e.g. |
dict()
|
producer_options
|
Mapping[str, Any]
|
librdkafka options of the transactional producer, e.g. |
dict()
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a limit is out of range, or options override a setting that the runtime owns. |
client_config ¶
client_config() -> dict[str, Any]
Return librdkafka options to connect a client of your own to the cluster, e.g. an AdminClient.
InvalidOffsetError ¶
Bases: PotikError
Raised when an offset policy refuses a partition, e.g. resume_or_fail, or returns an invalid offset.
A valid offset is the committed one, or within the partition's log. The error fails the run.
InvalidOutputError ¶
Bases: PotikError, PermanentError
Raised when a handler returns something other than a list or tuple of ProducerRecords.
A core handler that returns a single record raises it, for example. It comes from the handler call, so exception rules see it like any other handler exception. To stop the partition instead of letting a catch-all rule handle it, put on(InvalidOutputError, reraise()) before that rule.
PartitionsLostError ¶
Bases: PotikError
Raised when the consumer loses partitions without a revocation, so another member may already own them.
It happens e.g. when the consumer misses its heartbeats. The error fails the run, which commits nothing more.
PermanentError ¶
Bases: Exception
Marks a failure that would repeat if the handler ran again, e.g. a value that does not decode.
retry() and retry_forever() rules pass it to the next rule instead of retrying. Subclass it for such failures of your own. It does not derive from PotikError, so except PotikError does not catch your subclasses.
PotikError ¶
Bases: Exception
The base of the errors that potik raises.
Errors of the Kafka client stay KafkaExceptions, and invalid arguments raise ValueError or TypeError.
BaseMiddleware ¶
The base class of middleware, which wraps the stages of a Runtime or a Producer.
Subclass it and override any of these stages:
lifespansurrounds a run, or the time aProduceris open.handlewraps each handler call of a run.commitwraps each transaction of a run.sendwraps eachProducer.send().
Each stage except lifespan receives call_next, the rest of the chain. A middleware can run code around it, observe its outcome, and scope state to it, e.g. a span. One class can override every stage, so that e.g. one tracing middleware serves runtimes and producers. A runtime or producer runs its middleware outermost first.
An instance serves one run or one Producer, once, so its state lasts one run. Using it a second time raises RuntimeError. Create the instances where you create the runtime or producer, and pass resources, e.g. a connection pool, to __init__.
lifespan ¶
lifespan(scope: RunScope | ProducerScope) -> AbstractAsyncContextManager[None]
Return an async context manager that surrounds one run, or the time a Producer is open.
Write it with @asynccontextmanager. scope is a RunScope for a run, and a ProducerScope for a producer. Code before yield runs before the clients start, and a run is ready only once every lifespan has entered. Code after yield runs once the clients have closed, in reverse order. yield raises the run's failure, or the cancellation that stopped it, and the lifespan cannot suppress it. The exit is shielded from cancellation, so it may await, e.g. to close a connection pool. A stuck exit holds up shutdown, so cancel any background tasks after yield.
handle
async
¶
handle(record: ConsumerRecord, call_next: RecordHandler) -> Sequence[ProducerRecord]
Wrap one handler call, and return its outputs.
It is the only stage that may recover from a failure, e.g. by calling call_next again, or by returning outputs instead of raising. It may also change the outputs.
commit
async
¶
commit(transaction: Transaction, call_next: Callable[[Transaction], Awaitable[None]]) -> None
Wrap one transaction, including the retries when a rebalance makes the broker reject it.
Call call_next exactly once. Pass it the transaction, or a copy with other outputs made with dataclasses.replace(), which keeps the offsets. Code after call_next runs once the transaction has committed, before the next transaction starts, and before a revocation or shutdown that waits for it completes. Slow code there slows commits, and so the handlers. Raising fails the run. If code after call_next raises, the transaction stays committed. The stage cannot suppress an exception from call_next. A run failure cancels the stage, so the code after call_next may not run even when the transaction committed.
send
async
¶
send(record: ProducerRecord, call_next: RecordSender) -> Delivery
Wrap one send through a Producer, from producing record until the broker acknowledges it.
Outputs never pass through it. They go through handle and commit.
ProducerScope
dataclass
¶
ProducerScope(*, client_options: Mapping[str, Any])
Describes a Producer to a middleware's lifespan.
Attributes:
| Name | Type | Description |
|---|---|---|
client_options |
Mapping[str, Any]
|
librdkafka options to connect a client of the middleware's own to the producer's cluster. |
RunScope
dataclass
¶
RunScope(*, config: Config, status: Callable[[], RuntimeStatus], client_options: Mapping[str, Any])
Describes a run to a middleware's lifespan.
Attributes:
| Name | Type | Description |
|---|---|---|
config |
Config
|
The run's config. |
status |
Callable[[], RuntimeStatus]
|
Returns a snapshot of the run. Any thread may call it, e.g. a metrics exporter's. Before the lifespan yields, the run is not running yet. |
client_options |
Mapping[str, Any]
|
librdkafka options to connect a client of the middleware's own to the run's cluster, e.g. an |
OffsetContext ¶
OffsetContext(topic: str, partition: int, committed: int | None, *, watermarks: Callable[[], tuple[int, int]])
What an offset policy decides on: a newly assigned partition, its committed offset, and its log's bounds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The partition's topic. |
required |
partition
|
int
|
The partition's number. |
required |
committed
|
int | None
|
The committed offset, or None if there is none. |
required |
watermarks
|
Callable[[], tuple[int, int]]
|
Returns the log's first and end offsets. The context calls it only once a policy reads |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
The partition's topic. |
|
partition |
The partition's number. |
|
committed |
The committed offset, or None if there is none. |
Producer ¶
Producer(bootstrap_servers: str, *, options: Mapping[str, Any] | None = None, middleware: Iterable[BaseMiddleware] = (), close_timeout: float = _CLOSE_TIMEOUT)
A Kafka producer that sends records at once, outside any transaction.
Use it where there is no input record to commit with, e.g. in an API service, or from a handler. Open it once with async with, e.g. in an API service's lifespan, or around a run, and pass it to handlers through potik.build_runtime(context=...). Exiting waits up to close_timeout for every send to be delivered or fail, then fails the rest. On a normal exit, it then raises the first failure of a send that nobody awaited, e.g. a cancelled one. A producer opens only once.
A send from a handler is at-least-once, unlike the handler's outputs. A retried or replayed handler sends again, and readers see the record before the input record commits.
The producer is idempotent and waits for all in-sync replicas (acks=all). After a fatal error, e.g. a broken idempotent sequence, every later send fails. A callback in options that raises does the same, and also fails the sends that wait for a delivery report. Check failure in a liveness probe to restart the process then.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bootstrap_servers
|
str
|
The cluster's bootstrap servers. |
required |
options
|
Mapping[str, Any] | None
|
librdkafka options, e.g. TLS, SASL, |
None
|
middleware
|
Iterable[BaseMiddleware]
|
Middleware instances, outermost first. A producer runs their |
()
|
close_timeout
|
float
|
How many seconds exiting waits for sends to be delivered or fail. Then it fails the rest with a |
_CLOSE_TIMEOUT
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
failure
property
¶
failure: KafkaException | None
The fatal error that stopped this producer, if any.
Only a new producer, e.g. in a restarted process, can send again.
from_config
classmethod
¶
from_config(config: Config, *, middleware: Iterable[BaseMiddleware] = (), close_timeout: float = _CLOSE_TIMEOUT) -> Self
Make a producer for a runtime's cluster, with the config's options and producer_options.
send
async
¶
send(record: ProducerRecord) -> Delivery
Send record at once, and return where the broker appended it.
It returns once the broker acknowledges the record. If librdkafka's queue is full, it first waits for room. A cancelled send may still be delivered.
Raises:
| Type | Description |
|---|---|
KafkaException
|
If the record was not delivered, or the producer has failed. |
RuntimeError
|
If the producer is not entered. |
PendingReply ¶
PendingReply(correlation_id: str)
The reply that a ReplyListener.expect() block awaits.
wait
async
¶
wait() -> ConsumerRecord
Return the first reply with this correlation ID, which may have arrived before the call.
Bound the wait with e.g. anyio.fail_after().
Raises:
| Type | Description |
|---|---|
KafkaException
|
If the listener failed. |
RuntimeError
|
If the listener closed before the reply arrived. |
ReplyListener ¶
ReplyListener(reply_topic: str, *, bootstrap_servers: str, correlate: Callable[[ConsumerRecord], str | None], options: Mapping[str, Any] | None = None)
Hands each reply from a reply topic to the request in this process that awaits it.
Every process that sends requests, e.g. each replica of an API, reads every partition of the reply topic itself. The consumer belongs to no group and commits nothing. It reads from the end of each partition, as it is when the listener opens, so it never reads a reply to a request from an earlier process. It reads with read_committed, so a transactional responder's reply arrives once its request's offset commits.
The listener does not send requests, and potik sets no convention for correlation IDs. The caller puts each request's ID wherever its responder reads it, e.g. a header or a field of the value, and the responder copies it to its reply, where correlate reads it.
A reply that arrives while nothing awaits it is dropped, e.g. after its request timed out, or one to another process. A request that times out may still be handled. A responder that runs it again sends another reply, and only the first one counts. A reply whose correlation ID correlate cannot read is logged and dropped.
Open it once with async with, e.g. in an API's lifespan, before the API takes requests. Once it opens, it reads every reply to a request that the process sends. Exiting fails the requests that still await a reply. After a fatal consumer error, every awaited and later request fails. Check failure in a liveness probe to restart the process then.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reply_topic
|
str
|
The topic that the responders reply to. |
required |
bootstrap_servers
|
str
|
The cluster's bootstrap servers. |
required |
correlate
|
Callable[[ConsumerRecord], str | None]
|
Returns a reply's correlation ID, or None if it has none. It runs on the listener's thread for every record of the reply topic, so it must be thread-safe and fast. |
required |
options
|
Mapping[str, Any] | None
|
librdkafka consumer options, e.g. TLS or SASL. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
failure
property
¶
failure: KafkaException | None
The error that stopped reading replies, if any. Check it in a liveness probe.
expect ¶
expect(correlation_id: str | None = None) -> Generator[PendingReply]
Await the reply with correlation_id from entering the block until leaving it.
Enter the block before sending the request, so that a fast reply cannot arrive first. Send PendingReply.correlation_id with the request in any way the responder reads it, e.g. a header, a field of its value, or an HTTP call that makes the responder reply to Kafka.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
correlation_id
|
str | None
|
The request's ID, unique among this process's requests. Defaults to a new UUID. |
None
|
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the listener is not open. |
KafkaException
|
If the listener failed. |
ValueError
|
If another block in this process awaits the same ID. |
Runtime ¶
Runtime(config: Config, topics: Iterable[str], handler: RecordHandler, *, middleware: Iterable[BaseMiddleware] = (), key_concurrency: Mapping[str, int] | None = None)
Consumes topics, calls a handler for each record, and commits its outputs with the record's offset.
Each transaction produces a batch of outputs and commits their input records' offsets atomically, so each record's outputs appear exactly once to read_committed readers. A run uses one consumer, with the consumer group protocol of Kafka 4.0+, and one transactional producer. Each assigned partition handles its records one at a time, in offset order, unless its topic runs by key (see key_concurrency), and partitions run concurrently. The consumer keeps polling while handlers run, so a slow handler does not make it leave the group.
On a revocation or shutdown, the runtime lets in-flight handlers finish within a grace period, then cancels them, and commits the handled records before it gives up the partitions. A failure of one partition stops it (see Config.stop_failed_partitions). Any other failure, e.g. a failed transaction or lost partitions, stops the run without committing more, and the uncommitted records run again on the next run.
Cancelling run() requests a graceful shutdown and waits for it. If the shutdown fails, run() raises its error instead of the cancellation. The runtime never abandons a blocking client call, so the process supervisor must enforce a hard deadline.
A runtime runs once. Make a new one, with new middleware, for another run. potik.build_runtime() builds one from routers of typed handlers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
Config
|
The runtime limits and client options. |
required |
topics
|
Iterable[str]
|
The names of the topics to subscribe to. |
required |
handler
|
RecordHandler
|
Returns the outputs of one record of any of |
required |
middleware
|
Iterable[BaseMiddleware]
|
Middleware instances, outermost first (see |
()
|
key_concurrency
|
Mapping[str, int] | None
|
The topics whose records run concurrently by key, each with the maximum number of its keys whose records run at once, across its partitions. While keys wait for their turn, a key with more records takes turns with them, so that every partition makes progress. Records with the same key run one at a time, in offset order, and records with other keys run concurrently. All records with a null key count as one key. A key is in progress while its records run, including their retries' backoff, so the limit is not a pool of handler calls: limit those with a |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
TypeError
|
If |
status ¶
status() -> RuntimeStatus
Return a snapshot of the run.
Any thread may call it, e.g. a metrics exporter's. Off the event loop, the fields may come from slightly different instants.
liveness ¶
liveness(*, slack: float = 30.0) -> str | None
Check for failures that a restart fixes, with potik.core.health.liveness.
Serve it as the liveness probe. Any thread may call it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
slack
|
float
|
Seconds on top of each limit. |
30.0
|
Returns:
| Type | Description |
|---|---|
str | None
|
Why the probe fails, or None if it passes. |
readiness ¶
readiness() -> str | None
Check that the run is running, with potik.core.health.readiness.
Serve it as the readiness probe. Any thread may call it.
Returns:
| Type | Description |
|---|---|
str | None
|
Why the probe fails, or None if it passes. |
run
async
¶
run(*, task_status: TaskStatus[None] = anyio.TASK_STATUS_IGNORED, handle_signals: bool = True) -> None
Run until stopped or failed, and raise the run's failure, if any.
Cancelling it shuts the run down gracefully.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_status
|
TaskStatus[None]
|
Lets |
TASK_STATUS_IGNORED
|
handle_signals
|
bool
|
Whether the first SIGTERM or SIGINT shuts the run down gracefully, after which |
True
|
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the runtime has already run, or it handles signals outside the main thread or while another run in the process handles them. |
Delivery ¶
Bases: NamedTuple
Where the broker appended a sent record.
PartitionStatus
dataclass
¶
PartitionStatus(*, topic: str, partition: int, buffered_records: int, running_handlers: int, paused: bool, handler_started_at: float | None, failure: Exception | None)
A snapshot of one assigned partition.
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
str
|
The partition's topic. |
partition |
int
|
The partition's number. |
buffered_records |
int
|
The number of fetched records that have not started, including those that wait for an earlier record of their key. |
running_handlers |
int
|
The number of handler calls that run. It is at most 1, unless the topic runs by key (see |
paused |
bool
|
Whether fetching is paused, to limit the buffer or because the partition is stopped. |
handler_started_at |
float | None
|
The |
failure |
Exception | None
|
The exception that stopped the partition (see |
ProducerRecord ¶
Bases: NamedTuple, Generic[_V_co]
A record to produce.
A handler returns it as an output, which commits in one transaction with the input record's offset. Producer.send() sends it at once instead.
ProducerRecord[V] marks a record that a Topic[V] made, for type checkers only. A record made directly is a ProducerRecord[Any].
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
str
|
The topic's name. |
key |
bytes | None
|
The encoded key, or None for a null key. |
value |
bytes | None
|
The encoded value, or None for a tombstone. |
headers |
Headers | None
|
The record's headers. |
partition |
int
|
The record's partition, or -1 to let the producer's partitioner pick one from the key. librdkafka's default partitioner hashes keys differently from the Java client's. To match Java producers, set the |
RuntimeStatus
dataclass
¶
RuntimeStatus(*, running: bool = False, polled_at: float | None = None, oldest_uncommitted_at: float | None = None, failure: Exception | None = None, partitions: tuple[PartitionStatus, ...] = ())
A snapshot of a run, from Runtime.status().
Times are time.monotonic() values. potik.core.health checks a snapshot for liveness and readiness probes.
Attributes:
| Name | Type | Description |
|---|---|---|
running |
bool
|
Whether the run has started and is neither stopping nor stopped. |
polled_at |
float | None
|
When the consumer's last poll returned, or when polling started, or None before the run starts. The runtime polls at least every 200ms, except during a rebalance or a shutdown. |
oldest_uncommitted_at |
float | None
|
When the oldest batch that waits for its transaction got its first handled record or policy-selected offset, or None if no batch waits. |
failure |
Exception | None
|
The error that stopped the run, or None. An error of a middleware's lifespan is not recorded here, but |
partitions |
tuple[PartitionStatus, ...]
|
The assigned partitions. |
Transaction
dataclass
¶
Transaction(offsets: Mapping[tuple[str, int], int], outputs: Sequence[ProducerRecord])
The outputs and offsets of a batch of handled records, which commit atomically.
Middleware sees it in the commit stage (see BaseMiddleware.commit).
Attributes:
| Name | Type | Description |
|---|---|---|
offsets |
Mapping[tuple[str, int], int]
|
The offset to resume from of each |
outputs |
Sequence[ProducerRecord]
|
The records to produce. |
resume_or_earliest ¶
resume_or_earliest(context: OffsetContext) -> int
Resume the committed offset, or start at the beginning of the log.
resume_or_fail ¶
resume_or_fail(context: OffsetContext) -> int
Resume the committed offset, or fail the run with InvalidOffsetError.
resume_or_latest ¶
resume_or_latest(context: OffsetContext) -> int
Resume the committed offset, or start at the end of the log.
potik.core.health ¶
Liveness and readiness checks of a RuntimeStatus, to serve over any probe transport, e.g. HTTP.
Runtime.liveness() and Runtime.readiness() call them. Each check returns why the probe fails, or None if it passes. A reason names at most an exception's type, never its message, so it is safe to expose.
liveness ¶
liveness(status: RuntimeStatus, config: Config, *, now: float | None = None, slack: float = 30.0) -> str | None
Fail the probe when a restart would help, i.e. the run failed, or it stopped polling or committing.
Polling counts as stuck when no poll has returned for rebalance_timeout_ms plus slack. Committing counts as stuck when a handled record stays uncommitted for commit_interval_ms plus twice transaction_timeout_ms, plus slack.
A broker outage alone does not fail the check, because a restart cannot fix it. The check fails only if the outage fails the run, e.g. by timing out a transaction. The check also passes while a run starts or stops, unless the run failed. A handler may run for any time, so the check ignores how long it runs. It ignores stopped partitions too, because a restart would run the failed record again, which usually fails again, and stop every other partition meanwhile. Alert on them instead, e.g. with MetricsMiddleware.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
status
|
RuntimeStatus
|
The runtime's status, e.g. from |
required |
config
|
Config
|
The run's config. |
required |
now
|
float | None
|
The |
None
|
slack
|
float
|
Seconds added to each limit, to allow for a slow event loop and slow |
30.0
|
Returns:
| Type | Description |
|---|---|
str | None
|
Why the probe fails, or None if it passes. |
readiness ¶
readiness(status: RuntimeStatus) -> str | None
Pass the probe while the run is running, whether or not it has assigned or stopped partitions.
A consumer serves no requests, so readiness only paces rollouts and disruption budgets. It ignores partitions, because an instance may get none, e.g. with more replicas than partitions.
Returns:
| Type | Description |
|---|---|
str | None
|
Why the probe fails, or None if it passes. |
Middleware¶
potik.middleware ¶
The built-in middleware: exception rules, the topic check, and concurrency limits.
Tracing and metrics are in potik.middleware.tracing and potik.middleware.metrics, and need the otel extra.
Backoff ¶
Backoff = Callable[[ExceptionContext[Any]], float]
Returns the seconds to wait before the next attempt, e.g. from a Retry-After value that the exception carries.
Decision ¶
Decision = RetryAfter | Sequence[ProducerRecord] | Reraise | None
What a rule returns.
It returns RetryAfter to call the handler again, a list or tuple of records to commit as the record's outputs, Reraise to let the exception propagate, or None to ask the next rule.
Match ¶
Match = type[ExcT] | tuple[type[ExcT], ...] | Callable[[Exception], bool]
The exceptions that on() applies a rule to: a type, a tuple of them as in an except clause, or a predicate.
Rule ¶
Rule = Callable[[ExceptionContext[ExcT]], Decision | Awaitable[Decision]]
A sync or async function of an ExceptionContext that returns a Decision.
ConcurrencyMiddleware ¶
ConcurrencyMiddleware(*, max_concurrency: int | None = None, per_topic: Mapping[str, int] | None = None)
Bases: BaseMiddleware
Limits how many handler calls run at once across partitions, in total and per topic.
A call takes a slot of its topic first, then a shared one, so a call that waits for its topic holds no shared slot. Only the middleware after it runs with a slot, so put it last. Exception rules before it then do not hold a slot during retry backoff. potik.build_runtime() adds one from its own max_concurrency and its handlers', after the exception rules.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_concurrency
|
int | None
|
The maximum number of handler calls that run at once across all topics, e.g. the size of a database pool, or None for no limit. |
None
|
per_topic
|
Mapping[str, int] | None
|
The maximum number of handler calls that run at once for each topic. Topics missing from it have no limit of their own. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a limit is not a positive integer, or a topic name is invalid. |
ExceptionContext
dataclass
¶
ExceptionContext(*, exception: ExcT, record: ConsumerRecord, attempt: int)
What an exception rule decides on: the exception, the record, and the attempt.
Attributes:
| Name | Type | Description |
|---|---|---|
exception |
ExcT
|
The handler's exception. For an |
record |
ConsumerRecord
|
The input record. |
attempt |
int
|
The number of handler calls of the record so far, including the failed one. |
ExceptionMiddleware ¶
ExceptionMiddleware(*rules: Rule[Any])
Bases: BaseMiddleware
Retries or recovers from a handler's exceptions by asking rules in order, like except clauses.
With potik.build_runtime(), pass the rules as exception_rules instead of creating this middleware. Build rules with on(), retry(), retry_forever(), skip(), reraise(), and dead_letter().
The first rule that returns something other than None decides. RetryAfter, e.g. from retry(), calls the handler again after its delay. Records, e.g. from skip() or dead_letter(), commit as the record's outputs in place of the failure. Reraise, from reraise(), lets the exception propagate. An exception that no rule handles propagates too, and stops the partition, or fails the run (see Config.stop_failed_partitions).
Rules see the single exception of an ExceptionGroup that holds only one, so on(TimeoutError, ...) matches a TimeoutError from a task group in the handler. A group of several exceptions stays a group, which on(ExceptionGroup, ...) or a function matches. Either way, an exception that no rule handles propagates as the handler raised it.
A partition handles one record at a time, so a retry holds up its partition until the record succeeds or a later rule handles it. With key concurrency (see Runtime), a retry holds up only its key, though the records after it commit only once it succeeds or a rule handles it. A retry() rule that has used up its attempts returns None, so a later rule handles the exception. So does a retry() rule for a PermanentError, e.g. a DecodeError, which would fail the same way again.
The middleware's output_topics collects its rules' (see DeclaresOutputs), e.g. the dead letter topics of dead_letter() rules, so that the topic check covers them.
With the otel extra, the middleware records each exception that it retries or handles as an exception event on the current span. Inside TracingMiddleware, that span is the record's process span.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*rules
|
Rule[Any]
|
Sync or async functions of an |
()
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If a rule is a type or is not callable. |
Example
ExceptionMiddleware(
on(is_transient, retry(max_attempts=5)),
on(Exception, dead_letter(dead_letters)),
)
ExponentialBackoff
dataclass
¶
ExponentialBackoff(*, initial_seconds: float = 0.1, max_seconds: float | None = 30.0, multiplier: float = 2.0, jitter: bool = False)
A backoff whose delay grows by a factor after each attempt, up to a cap.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
initial_seconds
|
float
|
The delay after the first attempt. |
0.1
|
max_seconds
|
float | None
|
The longest delay, or None for no cap. The default cap keeps |
30.0
|
multiplier
|
float
|
The factor that multiplies the delay after each attempt. |
2.0
|
jitter
|
bool
|
Whether each wait is a random time between zero and the computed delay, so that partitions do not retry in step. |
False
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a delay is negative, or |
Reraise
dataclass
¶
Reraise()
A decision to ask no more rules and let the exception propagate, like a bare raise in an except clause.
The exception then stops the partition, or fails the run (see Config.stop_failed_partitions).
RetryAfter
dataclass
¶
RetryAfter(seconds: float)
A decision to call the handler again after seconds.
DeclaresOutputs ¶
Bases: Protocol
A middleware or exception rule with output_topics, the topics that it writes records to.
dead_letter() rules declare their dead letter topic this way, for example. potik.build_runtime() adds the declared topics of its middleware and exception rules to its topic check. They do not restrict where records go.
output_topics
property
¶
output_topics: Sequence[str]
The names of the topics that it writes records to.
NamedTopic ¶
Bases: Protocol
Any object with a topic's name, e.g. a Topic. APIs that take topics take one in place of a name.
TopicCheckError ¶
TopicCheckError(missing: tuple[str, ...], unauthorized: tuple[str, ...])
Bases: PotikError
Raised by TopicCheckMiddleware when topics are missing, or the client is not authorized to describe them.
Attributes:
| Name | Type | Description |
|---|---|---|
missing |
The missing topics. |
|
unauthorized |
The topics that the client is not authorized to describe. |
TopicCheckMiddleware ¶
TopicCheckMiddleware(topics: Iterable[NamedTopic | str], *, timeout: float = 10.0)
Bases: BaseMiddleware
Checks that topics exist before a run or a Producer starts, and fails the start otherwise.
Put it first, so that a missing topic fails the start before other lifespans open resources. potik.build_runtime() puts one first, with its handled topics and every output topic that its handlers, routers, middleware, and exception rules declare, unless check_topics=False.
The client needs permission to describe each topic. A missing or denied topic fails the start with TopicCheckError, and other broker errors, e.g. a timeout, with KafkaException. Cancelling the start, e.g. on SIGTERM, stops the check. It never creates a missing topic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topics
|
Iterable[NamedTopic | str]
|
The topics to check, as |
required |
timeout
|
float
|
How long the metadata request may take, including finding a broker, in seconds. |
10.0
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a topic name is invalid. |
dead_letter ¶
dead_letter(topic: NamedTopic | str, *, headers: Callable[[ExceptionContext[Exception]], Iterable[tuple[str, str | bytes | None]]] = dead_letter_headers) -> Rule[Exception]
Write a copy of the failed record to topic, as the record's outputs.
The copy commits in the same transaction as the failed record's offset. It keeps the record's key, value, and headers as bytes, even if they do not decode, and adds the headers that headers returns. The producer's partitioner picks its partition from the key, because the dead letter topic may have fewer partitions than the input's.
The rule declares topic as its output topic, so the topic check covers it. To write a different dead letter, write a rule of your own and declare its topic, e.g. in Router(output_topics=...).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
NamedTopic | str
|
The dead letter topic, as a |
required |
headers
|
Callable[[ExceptionContext[Exception]], Iterable[tuple[str, str | bytes | None]]]
|
Returns the headers to add, e.g. names that your dead letter consumers expect, or |
dead_letter_headers
|
dead_letter_headers ¶
dead_letter_headers(context: ExceptionContext[Exception]) -> list[tuple[str, str | None]]
Return the headers that dead_letter() adds by default.
potik.dlq.source.topic,potik.dlq.source.partition, andpotik.dlq.source.offset: where the record came from.potik.dlq.exception.type: the exception's qualified class name.potik.dlq.exception.message:str()of the exception, which may contain data from the record. A message longer than 1000 characters is cut, and ends with....potik.dlq.attempts: the number of handler calls.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
ExceptionContext[Exception]
|
The failed record and its exception. |
required |
on ¶
on(match: Match[ExcT], rule: Rule[ExcT], *, topics: Iterable[NamedTopic | str] | None = None) -> Rule[Exception]
Apply rule only to matching exceptions, and optionally only to some topics.
For other exceptions and topics, the returned rule returns None, so the next rule decides.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
match
|
Match[ExcT]
|
An exception type, a tuple of them as in an |
required |
rule
|
Rule[ExcT]
|
The rule to apply. |
required |
topics
|
Iterable[NamedTopic | str] | None
|
The topics whose records the rule applies to, as |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If a topic name is invalid. |
Example
on(TimeoutError, retry(max_attempts=5), topics=[orders])
reraise ¶
reraise() -> Rule[Exception]
Ask no more rules and let the exception propagate, so that it stops the partition.
Put it before a rule that handles every exception, e.g. on(InvalidOutputError, reraise()) before a dead_letter() rule. With Config(stop_failed_partitions=False), the exception fails the run instead.
retry ¶
retry(*, max_attempts: int, backoff: Backoff | None = None) -> Rule[Exception]
Retry until the handler has run max_attempts times, then pass the exception to the next rule.
A PermanentError, e.g. a DecodeError, goes straight to the next rule, because retrying cannot fix it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_attempts
|
int
|
The maximum number of handler calls of a record, including the first. |
required |
backoff
|
Backoff | None
|
Returns the seconds to wait before each retry. Defaults to |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
retry_forever ¶
retry_forever(*, backoff: Backoff | None = None) -> Rule[Exception]
Retry until the handler succeeds.
Like retry(), it passes a PermanentError to the next rule. Its partition makes no progress until the record succeeds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
backoff
|
Backoff | None
|
Returns the seconds to wait before each retry. Defaults to |
None
|
potik.middleware.tracing ¶
Trace records with OpenTelemetry, from their handling to the transaction that commits their outputs, and sends through a Producer.
Needs the otel extra.
TracingMiddleware ¶
TracingMiddleware(*, optional_attributes: Collection[str] = DEFAULT_OPTIONAL_ATTRIBUTES, key_text: Callable[[bytes], str | None] = _utf8, transaction_sample_ratio: float = 1.0, tracer_provider: TracerProvider | None = None)
Bases: BaseMiddleware
Traces each record's handling, its outputs to the transaction that commits them, and each Producer send.
It creates these spans:
process {topic}(consumer), one per handler call. It continues the trace in the record's headers. A record that runs again, e.g. after a crash, gets a secondprocessspan in the same trace.create {topic}(producer), one per output, a child of its record'sprocessspan. The output's headers carry its context. It ends at once, because backends count producer spans as dependency latency, and the transaction's span covers the produce.send {topic}(client), one per transaction, as the root of its own trace. A transaction without outputs is namedcommit {topic}, and the topic is left out of the name when there are several. It links to the sampledcreatespan of each output, up to the SDK's link limit. These links connect a record's trace to the transaction that commits its outputs. Only a committed handling's outputs reach downstream consumers, so when a record runs again, its committedprocessspan is the one whosecreatespans have a link and downstream children. A record without outputs has no link to its transaction.send {topic}(producer), one perProducer.send(), a child of the current span, e.g. a handler'sprocessspan or an API's request span. The record's headers carry its context. It lasts until the broker acknowledges the record.
Each span has the attributes that the OpenTelemetry messaging semantic conventions require for it, e.g. messaging.consumer.group.name on a run's spans. optional_attributes picks the optional ones. To add attributes of your own, or to rename a span, use trace.get_current_span() in a handler or in middleware after this one. The current span there is the process span, the transaction's span, or the send span.
Use one instance for each runtime and each producer. Put it first in the runtime's middleware, so that its spans cover the other middleware. potik.build_runtime() runs the exception rules inside its middleware, so one process span covers a record's retries and the outputs of exception rules, such as dead letters. In a core Runtime, put it before ExceptionMiddleware for the same span. A cancellation, e.g. by a revocation, ends the process span with a cancelled event instead of an error.
A transaction's span is the root of its own trace, so the tracer provider's sampler decides on it apart from the records in it. To sample transactions, use transaction_sample_ratio with a sampler that keeps root spans.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
optional_attributes
|
Collection[str]
|
The optional attributes to add, where they apply. The default, |
DEFAULT_OPTIONAL_ATTRIBUTES
|
key_text
|
Callable[[bytes], str | None]
|
Returns the text of a key for |
_utf8
|
transaction_sample_ratio
|
float
|
The share of transactions that get a span, from 0 to 1. A failed transaction always gets one, but if it was not picked, its span is not the current span while the transaction runs. The tracer provider's sampler still decides whether to keep each span. |
1.0
|
tracer_provider
|
TracerProvider | None
|
Defaults to the global tracer provider. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
DEFAULT_OPTIONAL_ATTRIBUTES
class-attribute
¶
DEFAULT_OPTIONAL_ATTRIBUTES: frozenset[str] = frozenset({MESSAGING_DESTINATION_PARTITION_ID, MESSAGING_KAFKA_OFFSET, MESSAGING_CLIENT_ID})
The optional attributes that it adds by default.
potik.middleware.metrics ¶
Measure record handling, transactions, and sends through a Producer with OpenTelemetry metrics.
Needs the otel extra.
MetricsMiddleware ¶
MetricsMiddleware(*, optional_attributes: Collection[str] = DEFAULT_OPTIONAL_ATTRIBUTES, meter_provider: MeterProvider | None = None)
Bases: BaseMiddleware
Measures handler calls, transactions, and Producer sends with the OpenTelemetry messaging metrics.
messaging.client.consumed.messagescounts the records that the run passes to its handler, per topic. Itsmessaging.operation.nameisprocess, the operation that delivers the records to the application.messaging.process.durationmeasures each handler call, per topic, witherror.typeif it raised.messaging.client.operation.durationmeasures each transaction, including its retries after a rebalance, witherror.typeif it failed. Its operation issend, orcommitif it has no outputs.messaging.client.sent.messagescounts each transaction's outputs, per topic, witherror.typeif the transaction failed.- For each
Producer.send(),messaging.client.operation.durationmeasures the whole call, including any wait for queue room, andmessaging.client.sent.messagesadds one, per topic. potik.partition.stoppedcounts the stopped partitions, per topic (seeConfig.stop_failed_partitions). The probes ignore them (seepotik.core.health), so alert when it stays above zero. The semantic conventions do not define this metric.
Each metric has the attributes that the conventions require for it, e.g. messaging.consumer.group.name on a run's durations. optional_attributes picks the optional ones. To rename a metric, drop an attribute, or change the buckets, configure a View in the SDK's MeterProvider.
Use one instance for each runtime and each producer. A handler call's duration includes its retries, because potik.build_runtime() runs the exception rules inside its middleware. In a core Runtime, put it before ExceptionMiddleware for the same duration. Cancelled handler calls, transactions, and sends record no duration and no sent count, but a cancelled handler call still counts as consumed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
optional_attributes
|
Collection[str]
|
The optional attributes to add, where they apply. The default is |
DEFAULT_OPTIONAL_ATTRIBUTES
|
meter_provider
|
MeterProvider | None
|
Defaults to the global meter provider. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
DEFAULT_OPTIONAL_ATTRIBUTES
class-attribute
¶
DEFAULT_OPTIONAL_ATTRIBUTES: frozenset[str] = frozenset({MESSAGING_DESTINATION_PARTITION_ID})
The optional attributes that it adds by default.
Typed API¶
potik ¶
Typed handlers for Kafka topics.
Declare each Topic with the types and serdes of its keys and values, register handlers on a Router, and build a core Runtime from the routers with build_runtime(). The package also re-exports the core names that services use.
HandlerResult
module-attribute
¶
HandlerResult = TypeAliasType('HandlerResult', ProducerRecord[_V] | Sequence[ProducerRecord[_V]] | None, type_params=(_V,))
What a typed handler returns: None, a ProducerRecord, or a list or tuple of them.
potik commits the records with the input record's offset, in one transaction.
Router ¶
Router(*, output_topics: Iterable[OutputTopic[Any] | str] = ())
A group of topic handlers, which potik.build_runtime() builds a runtime from.
Register handlers with @router.handler(topic). A router needs no runtime, so a module can define one at import, and tests can call its handlers directly. A router has no middleware of its own. To apply an exception rule to some topics only, use on(..., topics=...), and in middleware, check record.topic().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_topics
|
Iterable[OutputTopic[Any] | str]
|
Topics that records go to beyond the handlers' own |
()
|
Raises:
| Type | Description |
|---|---|
RegistrationError
|
If an output is not a valid topic name. |
handler ¶
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None) -> Callable[[H], H]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[V]]) -> Callable[[Callable[P, Awaitable[HandlerResult[V]]]], Callable[P, Awaitable[HandlerResult[V]]]]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[V] | str]) -> Callable[[Callable[P, Awaitable[HandlerResult[V | bytes]]]], Callable[P, Awaitable[HandlerResult[V | bytes]]]]
handler(topic: str | Topic[Any, Any], *, max_concurrency: int | None = None, key_concurrency: int | None = None, output_topics: Iterable[OutputTopic[Any] | str] = ()) -> Callable[[_UserHandler], _UserHandler]
Register an async function as the handler of topic, and return the function unchanged.
Each of the handler's parameters needs a type annotation, and takes one part of the input record:
Annotated[T, Key()]takes the key, as the topic's key type, or asbytesfor the raw key.Annotated[T, Header(name)]takes a header's last value.Annotated[T, Context()]takes the run's context.Annotated[T, Depends(dependency)]takes what a dependency returns or yields (seeDepends).- A parameter annotated
ConsumerRecordtakes the record itself. - The one parameter left, if any, takes the value, as the topic's value type, or as
bytesfor the raw value.
The handler returns None, a ProducerRecord, or a list or tuple of them. Anything else raises InvalidOutputError from the handler's call.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str | Topic[Any, Any]
|
The handled topic, as a |
required |
max_concurrency
|
int | None
|
The maximum number of calls of this handler that run at once, across the topic's partitions, or None for no limit. |
None
|
key_concurrency
|
int | None
|
The maximum number of the topic's keys in progress at once, across its partitions, or None to handle each partition's records one at a time. Records with the same key still run one at a time, in offset order. A key stays in progress during its retries' backoff, so set |
None
|
output_topics
|
Iterable[OutputTopic[Any] | str]
|
The topics that the handler returns records to, as |
()
|
Raises:
| Type | Description |
|---|---|
RegistrationError
|
If the router already handles |
Example
@router.handler(orders, output_topics=[accepted])
async def accept(order: Order, customer: Annotated[str, Key()]) -> ProducerRecord[Accepted]:
return accepted(Accepted(order.id), key=customer)
RegistrationError ¶
Bases: PotikError
Raised when a router cannot register a handler, or potik.build_runtime() cannot build a runtime from routers.
Topic ¶
Topic(name: str, value_type: TypeForm[_V], *, serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)
Topic(name: str, value_type: TypeForm[_V], *, key_type: TypeForm[_K], serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)
Topic(name: str, value_type: Any, *, key_type: Any = str, serde: Serde | AsyncSerde = _TEXT, key_serde: Serde | AsyncSerde = _TEXT)
Bases: Generic[_V, _K]
A Kafka topic's name, and the types and serdes of its values and keys.
Declare each topic once, e.g. in a module that its producers and consumers share, so that they all use the same wire format. Pass it to Router.handler() to handle it, and call it to make a ProducerRecord to it. Type checkers infer its types from the given ones, e.g. Topic("events", A | B, serde=json) is a Topic[A | B]. A topic is immutable.
The declaration gets its decoders and encoders from the serdes, so a serde that does not support a type fails here, not on the first record.
A tombstone, a record with a null value, has no value to decode. A handler's value parameter takes its default for it, else None if its type allows None, e.g. Order | None for a Topic[Order]. Otherwise the handler call raises DecodeError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The topic's name. |
required |
value_type
|
Any
|
The type of the topic's values. |
required |
key_type
|
Any
|
The type of the topic's keys. Defaults to |
str
|
serde
|
Serde | AsyncSerde
|
The |
_TEXT
|
key_serde
|
Serde | AsyncSerde
|
The |
_TEXT
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If a serde is neither a |
ValueError
|
If |
Example
json = PydanticJson()
orders = Topic("orders", Order, serde=json) # Topic[Order], str keys
by_number = Topic("orders.by_number", Order, key_type=int, serde=json)
decode
async
¶
decode(data: bytes) -> _V
Decode a value of this topic, e.g. in middleware.
A failure raises the serde's own exception, not DecodeError.
decode_key
async
¶
decode_key(data: bytes) -> _K
Decode a key of this topic, e.g. in middleware.
A failure raises the serde's own exception, not DecodeError.
OutputTopic ¶
Bases: NamedTopic, Protocol
The type of the topics in a handler's output_topics, e.g. a Topic[V].
You rarely name it yourself. Unlike Topic, it is covariant in V, so type checkers accept output_topics=[a, b], with a: Topic[A] and b: Topic[B], as outputs of A | B.
Key
dataclass
¶
Key()
Marks a parameter, Annotated[T, Key()], that takes the record's key.
T is the topic's key type, or bytes for the raw key. A null key takes the parameter's default if it has one, else None if T allows None, else the call fails with DecodeError.
Header
dataclass
¶
Header(name: str, decoder: Decoder | None = None)
Marks a parameter, Annotated[T, Header(name)], that takes a header's value.
If the record has the header more than once, the parameter takes the last value. A missing or null header takes the parameter's default if it has one, else None if T allows None, else the call fails with DecodeError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The header's name. |
required |
decoder
|
Decoder | None
|
Decodes the header's value. Defaults to |
None
|
Context
dataclass
¶
Context()
Marks a parameter, Annotated[T, Context()], that takes the run's context.
The context is the object passed as potik.build_runtime(context=...), e.g. a dataclass of the service's clients. potik does not check it against T. potik.build_runtime() raises RegistrationError if a handler or one of its dependencies takes it and no context is given.
Depends
dataclass
¶
Depends(dependency: Callable[..., object])
Marks a parameter, Annotated[T, Depends(dependency)], that takes what dependency returns.
A dependency is a function, sync or async, a class, or a callable object, which must be hashable. Its parameters take parts of the record, the run's context, or other dependencies, as a handler's parameters do, and registration checks them against the topic in the same way. A call decodes each part of the record once, e.g. the value, for every parameter that takes it with the same decoder, so they all get the same object. Each one still takes its own default for a null part. A sync dependency runs on the event loop, so it must not block. potik does not check what a dependency returns against T.
A dependency that yields, sync or async, gives the parameter the value that it yields, and runs the rest of its body after the handler returns or raises, e.g. to commit or roll back a transaction. Do not decorate it with @contextmanager or @asynccontextmanager: registration rejects such a function, because potik enters dependencies that yield itself.
Each handler call resolves the handler's dependencies anew, in the call's task, one at a time, in parameter order, and each one after its own dependencies. A dependency runs once per call, even if several parameters take it, in the handler and in its dependencies, or take dependencies equal to it, e.g. the same bound method. A retry of the call by an exception rule resolves them again.
After the call, potik resumes the dependencies that yield, last entered first. Each one resumes even if one resumed before it failed:
- If the handler, or a dependency, raised,
yieldraises that exception in the dependency. A dependency cannot suppress it: potik raises it even if the dependency catches it, so exception rules still handle it. The one exception is a cancellation by the dependency's own cancel scope aroundyield: withanyio.move_on_after(), a call that runs out of time returns no outputs, so its record commits, unless the call raised another exception, which it then raises. Withanyio.fail_after(), it raisesTimeoutError. The scope also covers the code afteryieldof the dependencies that potik entered after it. A dependency that swallows an outer cancellation, e.g. a revocation's, suppresses nothing: the call still raises it. - If a dependency raises after
yield, the call fails with that exception. Exception rules then handle it as a handler failure, and the handler's outputs are not committed. - If a revocation or shutdown cancels the call, the code after
yieldruns in the cancelled scope, where each await raises the cancellation again. Shield the awaits that must finish, with a timeout, e.g.with anyio.move_on_after(5, shield=True):.
The code after yield runs before the transaction commits the handler's outputs. So a write that it commits, e.g. to a database, happens at least once, not exactly once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dependency
|
Callable[..., object]
|
The function or class to call. |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
Example
async def transaction(services: Annotated[Services, Context()]) -> AsyncIterator[Transaction]:
tx = await services.db.begin()
try:
yield tx
except BaseException:
with anyio.move_on_after(5, shield=True):
await tx.rollback()
raise
await tx.commit()
@router.handler(orders)
async def save(order: Order, tx: Annotated[Transaction, Depends(transaction)]) -> None:
await tx.execute("INSERT INTO orders VALUES ($1)", order.sku)
DecodeError ¶
DecodeError(message: str, *, parameter: str)
Bases: PotikError, PermanentError
Raised from a handler call when a parameter, of the handler or of a dependency, cannot take its part of the record.
The part fails to decode, or it is missing or null and the parameter has no default and does not allow None. It is a PermanentError, so retry() rules do not retry it.
Attributes:
| Name | Type | Description |
|---|---|---|
parameter |
The parameter's name. |
EncodeError ¶
EncodeError(message: str, *, topic: str)
Bases: PotikError, PermanentError
Raised when a topic cannot encode a key or value.
The key or value has the wrong type, or the topic's serde fails on it. It is a PermanentError, so retry() rules do not retry it.
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
The topic's name. |
build_runtime ¶
build_runtime(config: Config, routers: Iterable[Router], *, context: object = _NO_CONTEXT, middleware: Iterable[BaseMiddleware] = (), exception_rules: Iterable[Rule[Exception]] = (), max_concurrency: int | None = None, check_topics: bool = True) -> Runtime
Build a Runtime that runs the routers' handlers.
Open the resources that handlers need first, e.g. a database pool or a Producer, then build the runtime with them and run it. A runtime runs once, so another run needs a new runtime with new middleware instances.
Each record passes through middleware, then the exception rules, then the concurrency limits, then its handler.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
Config
|
The runtime limits and client options. |
required |
routers
|
Iterable[Router]
|
The routers whose handlers to run. The runtime copies their handlers, so later registrations have no effect. |
required |
context
|
object
|
What |
_NO_CONTEXT
|
middleware
|
Iterable[BaseMiddleware]
|
Middleware instances, outermost first (see |
()
|
exception_rules
|
Iterable[Rule[Exception]]
|
Rules that retry or recover from handler exceptions, tried in order (see |
()
|
max_concurrency
|
int | None
|
The maximum number of handler calls that run at once, across all partitions and topics, e.g. the size of a database pool, or None for no limit. Like each handler's own |
None
|
check_topics
|
bool
|
Whether |
True
|
Returns:
| Type | Description |
|---|---|
Runtime
|
The runtime. Call its |
Raises:
| Type | Description |
|---|---|
RegistrationError
|
If the routers have no handlers, two routers handle the same topic, a handler or one of its dependencies takes |
TypeError
|
If a router is not a |
ValueError
|
If |
Example
async def main() -> None:
async with create_pool() as db:
await potik.build_runtime(
config,
[router],
context=db,
middleware=[TracingMiddleware()],
exception_rules=[
on(TimeoutError, retry(max_attempts=5)),
on(Exception, dead_letter(dead_letters)),
],
).run()
potik.serde ¶
Serdes, which convert a topic's keys and values between bytes and Python types.
A Serde returns a decoder and an encoder for each type it supports, and None for the others. A SerdeContext tells it which topic's keys or values they serve. An AsyncSerde returns a resolver in place of a decoder, for formats whose decoding needs I/O, e.g. a schema registry's. Type checkers infer these functions' types from the requested type. A Topic names the serdes of its values and keys, so the wire format never depends on how a handler annotates a type.
Decoders validate the data against the type, as far as the serde's library does. Encoders do not validate.
Decoder
module-attribute
¶
Decoder = TypeAliasType('Decoder', Callable[[bytes], _T], type_params=(_T,))
Decodes bytes to a T, e.g. a key, a value, or a header's value.
Encoder
module-attribute
¶
Encoder = TypeAliasType('Encoder', Callable[[_T], bytes], type_params=(_T,))
Encodes a T to bytes.
Resolver
module-attribute
¶
Resolver = TypeAliasType('Resolver', Callable[[bytes], Awaitable[Decoder[_T]]], type_params=(_T,))
Does the I/O that decoding one record needs, e.g. fetches its writer schema, and returns the record's Decoder.
Serde ¶
Bases: Protocol
Makes the decoders and encoders of the types it supports.
A Topic calls decoder() and encoder() once for its keys and once for its values, when it is declared, usually at import, so they must not do I/O. The functions they return run synchronously on the event loop for every record, so they must not do I/O either. When potik decodes a handler parameter, a decoder's exception becomes a DecodeError. When a Topic call encodes a record, an encoder's exception becomes an EncodeError. Both are permanent, so retry() rules do not retry them. For a format whose decoding needs I/O, write an AsyncSerde.
decoder ¶
decoder(tp: TypeForm[T], context: SerdeContext) -> Decoder[T] | None
Return a function that decodes bytes to a tp of context, or None if this serde does not support tp.
encoder ¶
encoder(tp: TypeForm[T], context: SerdeContext) -> Encoder[T] | None
Return a function that encodes a tp of context to bytes, or None if this serde does not support tp.
AsyncSerde ¶
Bases: Protocol
Makes the resolvers and encoders of the types it supports, for formats whose decoding needs I/O.
A resolver does the I/O that decoding one record needs, e.g. fetching the writer schema that the record names from a registry, and returns the decoder of that record. A resolver's exceptions reach the exception rules as they are, so retry() rules retry e.g. a timeout. Raise a PermanentError for a failure that would repeat, e.g. an unknown schema. The decoder that the resolver returns, and the encoders, work as a Serde's do: they are synchronous, must not do I/O, and their failures are permanent.
A Topic calls resolver() and encoder() once for its keys and once for its values, when it is declared, so they must not do I/O. An encoder whose format needs I/O, e.g. the registry ID of its writer schema, gets it before the run. The serde learns every topic's keys and values when they are declared, so a method of its own, which the service's main awaits before the run, can register their schemas. potik does not open or close a client that the serde uses, e.g. a registry's. Open and close it around the run, e.g. in the service's main.
resolver ¶
resolver(tp: TypeForm[T], context: SerdeContext) -> Resolver[T] | None
Return an async function that returns a record's decoder to a tp of context, or None if this serde does not support tp.
encoder ¶
encoder(tp: TypeForm[T], context: SerdeContext) -> Encoder[T] | None
Return a function that encodes a tp of context to bytes, or None if this serde does not support tp.
SerdeContext
dataclass
¶
SerdeContext(topic: str, part: Literal['key', 'value'])
The keys or values of one topic, which a serde makes a decoder and an encoder for.
A serde that depends on the topic reads it, e.g. one that names a schema registry subject after the topic, like Confluent's TopicNameStrategy (orders-value). Other serdes ignore it.
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
str
|
The topic's name. |
part |
Literal['key', 'value']
|
|
Text ¶
Converts str as UTF-8, int and UUID as their text, and bytes as they are.
It is the default serde of topics' keys and values, and the default decoder of Header parameters. A subclass counts as its base, e.g. a StrEnum as its value, and a NewType counts as its supertype. It does not support bool, or Annotated types such as Annotated[str, Field(min_length=3)], rather than ignore their constraints.
An int decodes only from ASCII digits with an optional leading -, the form that it encodes to, so different keys never decode to the same int.
PydanticJson ¶
PydanticJson(**dump_options: Any)
JSON through pydantic, for any type that pydantic's TypeAdapter supports.
Decoding validates the data against the type. Encoding serializes without validating, but fails on a value that pydantic would warn about, e.g. an int in a str field. Pass warnings=... to change that. Install the pydantic extra to use it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**dump_options
|
Any
|
Options of |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If pydantic is not installed. |
MsgspecJson ¶
MsgspecJson()
JSON through msgspec, for any type that msgspec supports without dec_hook or enc_hook.
Decoding validates the data against the type. Encoding serializes by the value's own type. Install the msgspec extra to use it.
Raises:
| Type | Description |
|---|---|
ImportError
|
If msgspec is not installed. |
potik.testing ¶
Run a whole app in tests, on an in-process Kafka cluster instead of a broker.
MockCluster ¶
MockCluster()
An in-process Kafka cluster that runs a whole app in a test, without a broker.
The app's own clients connect to it, so its runtime, middleware, and topic check run as in production. Point the app at the cluster with config(), create its topics, run the app, and wait for its outputs with wait_records().
The cluster has one broker. It creates no topic on its own, so a topic that the test did not create fails the app's start. It commits transactions and hides uncommitted outputs from readers. It does not store the offsets that a transaction commits, and it does not fence stale group members. So a restarted app handles its input again, and only a real broker can test what a run committed.
Enter it with with. Exit it only after the app's run and every other client of the test have closed, because a client that outlives the cluster keeps trying to reconnect. It runs librdkafka's mock cluster, on Linux and macOS only.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
On other platforms, or if the installed confluent-kafka does not export librdkafka's mock cluster API. |
Example
async def test_accept() -> None:
with MockCluster() as cluster:
cluster.create_topic(orders)
cluster.create_topic(accepted)
async with Producer(cluster.bootstrap_servers) as producer:
await producer.send(orders(Order(sku="A"), key="c1"))
runtime = build(cluster.config(CONFIG)) # your service's potik.build_runtime(...)
async with anyio.create_task_group() as tasks:
await tasks.start(partial(runtime.run, handle_signals=False))
(record,) = await cluster.wait_records(accepted, 1)
tasks.cancel_scope.cancel() # shuts the run down gracefully
bootstrap_servers
property
¶
bootstrap_servers: str
The cluster's bootstrap servers, for the app's Config and any other client.
config ¶
config(config: Config) -> Config
Return a copy of config that connects to this cluster, for an app under test.
The copy starts each partition at its first record (resume_or_earliest), so the app also handles the records that the test sent before the partition was assigned. It also sets fetch.wait.max.ms to 10ms, because the mock cluster holds a fetch without records for that whole wait, even when a record arrives.
create_topic ¶
create_topic(topic: NamedTopic | str, partitions: int = 1) -> None
Create topic, e.g. a Topic or a name, with partitions partitions.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the name is invalid, or |
KafkaException
|
If the cluster refuses the topic, e.g. one that exists. |
wait_records
async
¶
wait_records(topic: NamedTopic | str, count: int, *, timeout: float = 10.0) -> list[ConsumerRecord]
Read topic until it has at least count committed records and every partition has reached its end.
It returns every record that it read, which may be more than count. A topic that keeps receiving records may time out. A record counts once a read_committed consumer can read it. A Producer send counts at once, and a handler's output once its transaction commits. Each partition's records come in offset order.
Raises:
| Type | Description |
|---|---|
TimeoutError
|
If the topic has fewer than |
KafkaException
|
If the topic does not exist. |