description	OpenTelemetry is a set of APIs, libraries, agents, and instrumentation to provide observability to your applications. Fedify supports OpenTelemetry for tracing and metrics. This document explains how to use OpenTelemetry with Fedify.

OpenTelemetry

This API is available since Fedify 1.3.0.

OpenTelemetry is a standardized set of APIs, libraries, agents, and instrumentation to provide observability to your applications. Fedify supports OpenTelemetry for tracing and metrics. This document explains how to use OpenTelemetry with Fedify.

Setting up OpenTelemetry

Tip

If you are using Deno 2.2 or later, you can use Deno's built-in OpenTelemetry support. See the Using Deno's built-in OpenTelemetry support section for more details.

To trace your Fedify application and collect metrics with OpenTelemetry, you need to set up the OpenTelemetry SDK. First of all, you need to install the OpenTelemetry SDK and the exporter you want to use. For example, if you want to use the trace exporter for OTLP (http/protobuf), you should install the following packages:

::: code-group

deno add npm:@opentelemetry/sdk-node npm:@opentelemetry/exporter-trace-otlp-proto

npm add @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto

bun add @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto

:::

Then you can set up the OpenTelemetry SDK in your Fedify application. Here is an example code snippet to set up the OpenTelemetry SDK with the OTLP trace exporter:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";

const sdk = new NodeSDK({
  serviceName: "my-fedify-app",
  traceExporter: new OTLPTraceExporter({
    url: "http://localhost:4317",
    headers: { "x-some-header": "some-value" }
  }),
});

sdk.start();

Caution

The above code which sets up the OpenTelemetry SDK needs to be executed before the Fedify server starts. Otherwise, the tracing may not work as expected.

Using Deno's built-in OpenTelemetry support

Since Deno 2.2, Deno has built-in support for OpenTelemetry. This means you can use OpenTelemetry with your Fedify application on Deno without manually setting up the OpenTelemetry SDK.

To enable the OpenTelemetry integration in Deno, you need to:

Run your Deno script with the --unstable-otel flag
Set the environment variable OTEL_DENO=true

For example:

OTEL_DENO=true deno run --unstable-otel your_fedify_app.ts

This will automatically collect and export runtime observability data to an OpenTelemetry endpoint at localhost:4318 using Protobuf over HTTP (http/protobuf).

You can customize the endpoint and protocol using environment variables like OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL. For authentication, you can use the OTEL_EXPORTER_OTLP_HEADERS environment variable.

Explicit `TracerProvider` configuration

The createFederation() function accepts the tracerProvider option to explicitly configure the TracerProvider for the OpenTelemetry SDK. Note that if it's omitted, Fedify will use the global default TracerProvider provided by the OpenTelemetry SDK.

For example, if you want to use Sentry as the trace exporter, you can set up the Sentry SDK and pass the TracerProvider provided by the Sentry SDK to the createFederation() function:

// @noErrors: 2339
import type { KvStore } from "@fedify/fedify";
// ---cut-before---
import { createFederation } from "@fedify/fedify";
import { getClient } from "@sentry/node";

const federation = createFederation<void>({
// ---cut-start---
  kv: null as unknown as KvStore,
// ---cut-end---
  // Omitted for brevity; see the related section for details.
  tracerProvider: getClient()?.traceProvider,
});

Caution

The Sentry SDK's OpenTelemetry integration is available since @sentry/node 8.0.0, and it's not available yet in @sentry/deno or @sentry/bun as of November 2024.

For more information about the Sentry SDK's OpenTelemetry integration, please refer to the OpenTelemetry Support section in the Sentry SDK docs.

Explicit `MeterProvider` configuration

This API is available since Fedify 2.3.0.

The createFederation() function also accepts the meterProvider option to explicitly configure the MeterProvider for OpenTelemetry metrics. If it is omitted, Fedify uses the global default MeterProvider provided by the OpenTelemetry SDK.

import type { KvStore } from "@fedify/fedify";
// ---cut-before---
import { createFederation } from "@fedify/fedify";
import { metrics } from "@opentelemetry/api";

const federation = createFederation<void>({
// ---cut-start---
  kv: null as unknown as KvStore,
// ---cut-end---
  // Omitted for brevity; see the related section for details.
  meterProvider: metrics.getMeterProvider(),
});

Instrumented spans

Fedify automatically instruments the following operations with OpenTelemetry spans:

Span name	Span kind	Description
`{method} {template}`	Server	Serves the incoming HTTP request.
`activitypub.dispatch_actor`	Server	Dispatches the ActivityPub actor.
`activitypub.dispatch_actor_key_pairs`	Server	Dispatches the ActivityPub actor key pairs.
`activitypub.dispatch_collection {collection}`	Server	Dispatches the ActivityPub collection.
`activitypub.dispatch_collection_page {collection}`	Server	Dispatches the ActivityPub collection page.
`activitypub.dispatch_inbox_listener {type}`	Internal	Dispatches the ActivityPub inbox listener.
`activitypub.dispatch_object`	Server	Dispatches the Activity Streams object.
`activitypub.fanout`	Consumer	Dequeues the ActivityPub activity to fan out.
`activitypub.fanout`	Producer	Enqueues the ActivityPub activity to fan out.
`activitypub.fetch_key`	Client	Fetches the public keys for the actor.
`activitypub.get_actor_handle`	Client	Resolves the actor handle.
`activitypub.inbox`	Consumer	Dequeues the ActivityPub activity to receive.
`activitypub.inbox`	Internal	Manually routes the ActivityPub activity.
`activitypub.inbox`	Producer	Enqueues the ActivityPub activity to receive.
`activitypub.inbox`	Server	Receives the ActivityPub activity.
`activitypub.lookup_object`	Client	Looks up the Activity Streams object.
`activitypub.outbox`	Client	Sends the ActivityPub activity.
`activitypub.outbox`	Consumer	Dequeues the ActivityPub activity to send.
`activitypub.outbox`	Producer	Enqueues the ActivityPub activity to send.
`activitypub.parse_object`	Internal	Parses the Activity Streams object.
`activitypub.fetch_document`	Client	Fetches a remote JSON-LD document.
`activitypub.send_activity`	Client	Sends the ActivityPub activity.
`activitypub.verify_key_ownership`	Internal	Verifies actor ownership of a key.
`http_signatures.sign`	Internal	Signs the HTTP request.
`http_signatures.verify`	Internal	Verifies the HTTP request signature.
`ld_signatures.sign`	Internal	Makes the Linked Data signature.
`ld_signatures.verify`	Internal	Verifies the Linked Data signature.
`object_integrity_proofs.sign`	Internal	Makes the object integrity proof.
`object_integrity_proofs.verify`	Internal	Verifies the object integrity proof.
`webfinger.handle`	Server	Handles the WebFinger request.
`webfinger.lookup`	Client	Looks up the WebFinger resource.

More operations will be instrumented in the future releases.

Span events

In addition to spans, Fedify also records span events to capture rich, structured data about key operations. Span events allow recording complex data that wouldn't fit in span attributes (which are limited to primitive values).

The following span events are recorded:

Event name	Recorded on span	Description
`activitypub.activity.received`	`activitypub.inbox`	Records full activity JSON and verification status when an activity is received.
`activitypub.activity.sent`	`activitypub.send_activity`	Records delivery details when an activity is sent.
`activitypub.delivery.failed`	`activitypub.outbox`	Records queued outbox delivery failure details before retry or abandonment.
`activitypub.object.fetched`	`activitypub.lookup_object`	Records full object JSON when successfully fetched.

Event attributes

Each span event includes attributes with detailed information:

activitypub.activity.received event attributes:

activitypub.activity.json: The complete activity JSON
activitypub.activity.verified: Whether the activity was verified (true/false)
ld_signatures.verified: Whether Linked Data Signatures were verified (true/false)
http_signatures.verified: Whether HTTP Signatures were verified (true/false)
http_signatures.key_id: The key ID used for HTTP signature verification
http_signatures.failure_reason (optional): Why HTTP signature verification failed (noSignature, invalidSignature, or keyFetchError)
http_signatures.key_fetch_status (optional): The HTTP status code when fetching the signing key failed with an HTTP response
http_signatures.key_fetch_error (optional): The error type when fetching the signing key failed without an HTTP response

activitypub.activity.sent event attributes:

activitypub.inbox.url: The inbox URL where the activity was delivered
activitypub.activity.id: The activity ID
activitypub.activity.type (optional): The qualified activity type URI
activitypub.actor.id (optional): The sender actor ID

The activitypub.activity.sent event records delivery metadata and lightweight activity identifiers only. It does not include the full activitypub.activity.json payload; if you need the full outbound activity for auditing, store it in your application before delivery and correlate it with activitypub.activity.id.

activitypub.delivery.failed event attributes:

activitypub.remote.host: The remote inbox host
activitypub.delivery.attempt: The zero-based queue delivery attempt
activitypub.delivery.permanent_failure: Whether Fedify will abandon the delivery instead of retrying
http.response.status_code (optional): The HTTP response status code returned by the remote inbox

activitypub.object.fetched event attributes:

activitypub.object.type: The type URI of the fetched object
activitypub.object.json: The complete object JSON

Instrumented metrics

This API is available since Fedify 2.3.0.

Fedify records the following OpenTelemetry metrics:

Metric name	Instrument	Unit	Description
`activitypub.delivery.sent`	Counter	`{attempt}`	Counts outgoing ActivityPub delivery attempts.
`activitypub.delivery.permanent_failure`	Counter	`{failure}`	Counts outgoing deliveries abandoned as permanent failures.
`activitypub.delivery.duration`	Histogram	`ms`	Measures outgoing ActivityPub delivery attempt duration.
`activitypub.inbox.processing_duration`	Histogram	`ms`	Measures inbox listener processing duration.
`activitypub.signature.verification_failure`	Counter	`{failure}`	Counts failed signature verification for inbox requests.
`fedify.http.server.request.count`	Counter	`{request}`	Counts inbound HTTP requests handled by `Federation.fetch()`.
`fedify.http.server.request.duration`	Histogram	`ms`	Measures inbound HTTP request duration in `Federation.fetch()`.
`fedify.queue.task.enqueued`	Counter	`{task}`	Counts inbox, outbox, and fanout tasks Fedify enqueued.
`fedify.queue.task.started`	Counter	`{task}`	Counts queue tasks Fedify began processing as a worker.
`fedify.queue.task.completed`	Counter	`{task}`	Counts queue tasks Fedify finished processing without throwing.
`fedify.queue.task.failed`	Counter	`{task}`	Counts queue tasks Fedify abandoned because processing threw.
`fedify.queue.task.duration`	Histogram	`ms`	Measures queue task processing duration in Fedify workers.
`fedify.queue.task.in_flight`	UpDownCounter	`{task}`	Tracks queue tasks currently in flight in this Fedify process.

Metric attributes

activitypub.delivery.sent : activitypub.remote.host, activitypub.delivery.success, and activitypub.activity.type when Fedify knows the activity type.

activitypub.delivery.permanent_failure : activitypub.remote.host and http.response.status_code.

activitypub.delivery.duration : activitypub.remote.host, activitypub.delivery.success, and activitypub.activity.type when Fedify knows the activity type.

activitypub.inbox.processing_duration : activitypub.activity.type.

activitypub.signature.verification_failure : activitypub.verification.failure_reason, plus activitypub.remote.host when the failed signature includes a key ID.

fedify.http.server.request.count and fedify.http.server.request.duration : http.request.method and fedify.endpoint are always present. http.request.method is normalized to one of the standard HTTP methods (CONNECT, DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT, QUERY, TRACE) or _OTHER for any other value, so that an arbitrary client cannot inflate metric cardinality by sending custom methods. http.response.status_code is recorded when a Response is produced (success and non-2xx alike) and omitted when the request threw an exception before a response could be returned. fedify.route.template is recorded when a route matched, and contains the URI Template parameter names (for example /users/{identifier}) rather than the matched parameter values.

fedify.queue.task.enqueued, fedify.queue.task.started, fedify.queue.task.completed, fedify.queue.task.failed, and fedify.queue.task.duration : fedify.queue.role (inbox, outbox, or fanout) is always present. fedify.queue.backend is the queue implementation's constructor name (for example RedisMessageQueue) when available; it is omitted for queues whose constructor is the plain Object (for example, MessageQueue instances built from an object literal). fedify.queue.native_retrial reflects the queue backend's nativeRetrial flag when set on the queue. activitypub.activity.type is recorded whenever Fedify knows the activity type for the queued message; for inbox tasks the type only becomes available after the activity is parsed, so the started counter for inbox tasks may be recorded without it. fedify.queue.task.enqueued additionally carries a zero-based fedify.queue.task.attempt so that retry re-enqueues are distinguishable from initial enqueues. fedify.queue.task.completed, fedify.queue.task.failed, and fedify.queue.task.duration carry fedify.queue.task.result, which is completed when processing returned without throwing, failed when the worker re-threw a non-abort error, and aborted when the worker re-threw an AbortError (for example, because a graceful-shutdown AbortSignal interrupted processing). When the queue backend does not declare nativeRetrial, Fedify catches inbox listener and outbox delivery errors itself; if its retry policy still allows another attempt, it schedules a retry by re-enqueuing the message and returns from the worker without re-throwing, so the worker boundary records result=completed. When the retry policy gives up, the worker also returns normally (result=completed) without scheduling a retry. Outbox-side activity failures remain observable through the activitypub.delivery.* metrics and the activitypub.delivery.failed span event, and any retry attempt — inbox or outbox — appears as a fedify.queue.task.enqueued measurement with a non-zero fedify.queue.task.attempt. Inbox listener errors that the retry policy abandons are visible through error logs and the inbox span's error status, but not through a dedicated metric.

fedify.queue.task.in_flight : fedify.queue.role and fedify.queue.backend (when available), plus fedify.queue.native_retrial when set on the queue. Per-message attributes such as activitypub.activity.type, fedify.queue.task.attempt, and fedify.queue.task.result are deliberately omitted so that increment and decrement operations always pair up cleanly per attribute series. This UpDownCounter is process-local: it tracks tasks currently being processed in this Fedify process, not cross-process totals. Aggregate it across replicas in your metrics backend.

The fedify.queue.task.* metrics describe what Fedify's workers do with queued messages. They complement the backend-side MessageQueue.getDepth() API, which reports how many messages are currently waiting in the queue backend. Reading both signals together — task throughput plus backlog depth — makes it possible to distinguish a small, slow queue from a large, fast one and to set alerting thresholds for delivery latency under load.

Fedify records activitypub.remote.host as the URL hostname only; ports, paths, and query strings are deliberately excluded to keep metric cardinality bounded. Activity types use the same qualified URI form as Fedify's trace attributes, for example https://www.w3.org/ns/activitystreams#Create.

The HTTP server request metrics deliberately exclude high-cardinality fields such as the full URL, raw path, query string, actor identifier, and inbox URL. Use the request span's url.full attribute when you need the exact URL for a sampled trace; the metrics expose the stable endpoint category and route template so that aggregate request rate, latency, and status-code error rate remain meaningful even when traces are sampled.

The fedify.endpoint attribute is drawn from a fixed enumeration: webfinger, nodeinfo, actor, inbox, shared_inbox, outbox, object, following, followers, liked, featured, featured_tags, collection, not_found, not_acceptable, and error. When a request throws an exception after Fedify has already classified its endpoint, the metric retains the matched endpoint (for example actor) so that fault-attribution stays per endpoint; error is only used when classification itself failed.

Semantic attributes for ActivityPub

The OpenTelemetry Semantic Conventions currently do not have a specification for ActivityPub as of November 2024. However, Fedify provides a set of semantic attributes for ActivityPub. The following table shows the semantic attributes for ActivityPub:

Attribute	Type	Description	Example
`activitypub.activity.id`	string	The URI of the activity object.	`"https://example.com/activity/1"`
`activitypub.activity.type`	string[]	The qualified URI(s) of the activity type(s).	`["https://www.w3.org/ns/activitystreams#Create"]`
`activitypub.activity.to`	string[]	The URI(s) of the recipient collections/actors of the activity.	`["https://example.com/1/followers/2"]`
`activitypub.activity.cc`	string[]	The URI(s) of the carbon-copied recipient collections/actors of the activity.	`["https://www.w3.org/ns/activitystreams#Public"]`
`activitypub.activity.bto`	string[]	The URI(s) of the blind recipient collections/actors of the activity.	`["https://example.com/1/followers/2"]`
`activitypub.activity.bcc`	string[]	The URI(s) of the blind carbon-copied recipient collections/actors of the activity.	`["https://www.w3.org/ns/activitystreams#Public"]`
`activitypub.activity.retries`	int	The ordinal number of activity resending attempt (if and only if it's retried).	`3`
`activitypub.delivery.attempt`	int	The zero-based delivery attempt number for a queued outgoing activity.	`0`
`activitypub.delivery.permanent_failure`	boolean	Whether an outgoing delivery failure will be abandoned instead of retried.	`true`
`activitypub.actor.id`	string	The URI of the actor object.	`"https://example.com/actor/1"`
`activitypub.actor.key.cached`	boolean	Whether the actor's public keys are cached.	`true`
`activitypub.actor.type`	string[]	The qualified URI(s) of the actor type(s).	`["https://www.w3.org/ns/activitystreams#Person"]`
`activitypub.key.id`	string	The URI of the cryptographic key being verified.	`"https://example.com/actor/1#main-key"`
`activitypub.key_ownership.method`	string	The method used to verify key ownership (`owner_id` or `actor_fetch`).	`"actor_fetch"`
`activitypub.key_ownership.verified`	boolean	Whether the key ownership was successfully verified.	`true`
`activitypub.collection.id`	string	The URI of the collection object.	`"https://example.com/collection/1"`
`activitypub.collection.type`	string[]	The qualified URI(s) of the collection type(s).	`["https://www.w3.org/ns/activitystreams#OrderedCollection"]`
`activitypub.collection.total_items`	int	The total number of items in the collection.	`42`
`activitypub.object.id`	string	The URI of the object or the object enclosed by the activity.	`"https://example.com/object/1"`
`activitypub.object.type`	string[]	The qualified URI(s) of the object type(s).	`["https://www.w3.org/ns/activitystreams#Note"]`
`activitypub.object.in_reply_to`	string[]	The URI(s) of the original object to which the object reply.	`["https://example.com/object/1"]`
`activitypub.inboxes`	int	The number of inboxes the activity is sent to.	`12`
`activitypub.remote.host`	string	The hostname of the remote ActivityPub server.	`"example.com"`
`activitypub.shared_inbox`	boolean	Whether the activity is sent to the shared inbox.	`true`
`docloader.context_url`	string	The URL of the JSON-LD context document (if provided via Link header).	`"https://www.w3.org/ns/activitystreams"`
`docloader.document_url`	string	The final URL of the fetched document (after following redirects).	`"https://example.com/object/1"`
`fedify.actor.identifier`	string	The identifier of the actor.	`"1"`
`fedify.endpoint`	string	The bounded endpoint category that classified an inbound HTTP request handled by `Federation.fetch()`.	`"actor"`
`fedify.route.template`	string	The matched URI Template, with parameter names (not values).	`"/users/{identifier}"`
`fedify.inbox.recipient`	string	The identifier of the inbox recipient.	`"1"`
`fedify.object.type`	string	The URI of the object type.	`"https://www.w3.org/ns/activitystreams#Note"`
`fedify.object.values.{parameter}`	string[]	The argument values of the object dispatcher.	`["1", "2"]`
`fedify.collection.cursor`	string	The cursor of the collection.	`"eyJpZCI6IjEiLCJ0eXBlIjoiT3JkZXJlZENvbGxlY3Rpb24ifQ=="`
`fedify.collection.items`	number	The number of items in the collection page. It can be less than the total items.	`10`
`fedify.queue.role`	string	The Fedify queue role for the task: `inbox`, `outbox`, or `fanout`.	`"outbox"`
`fedify.queue.backend`	string	The queue implementation's constructor name (best-effort backend identifier).	`"RedisMessageQueue"`
`fedify.queue.native_retrial`	boolean	Whether the queue backend declares `nativeRetrial`, meaning Fedify defers retry handling to the backend.	`true`
`fedify.queue.task.attempt`	int	The zero-based attempt number recorded on `fedify.queue.task.enqueued`; non-zero for retry re-enqueues.	`1`
`fedify.queue.task.result`	string	The terminal outcome of queue task processing: `completed`, `failed`, or `aborted`.	`"failed"`
`http.redirect.url`	string	The redirect URL when a document fetch results in a redirect.	`"https://example.com/new-location"`
`http.response.status_code`	int	The HTTP response status code.	`200`
`http_signatures.signature`	string	The signature of the HTTP request in hexadecimal.	`"73a74c990beabe6e59cc68f9c6db7811b59cbb22fd12dcffb3565b651540efe9"`
`http_signatures.algorithm`	string	The algorithm of the HTTP request signature.	`"rsa-sha256"`
`http_signatures.key_id`	string	The public key ID of the HTTP request signature.	`"https://example.com/actor/1#main-key"`
`http_signatures.verified`	boolean	Whether the HTTP request signature was verified successfully.	`false`
`http_signatures.failure_reason`	string	Why HTTP signature verification failed (`noSignature`, `invalidSignature`, or `keyFetchError`).	`"keyFetchError"`
`http_signatures.key_fetch_status`	int	The HTTP status code from a failed signing-key fetch, when available.	`410`
`http_signatures.key_fetch_error`	string	The error type from a non-HTTP signing-key fetch failure, when available.	`"TypeError"`
`http_signatures.digest.{algorithm}`	string	The digest of the HTTP request body in hexadecimal. The `{algorithm}` is the digest algorithm (e.g., `sha`, `sha-256`).	`"d41d8cd98f00b204e9800998ecf8427e"`
`ld_signatures.key_id`	string	The public key ID of the Linked Data signature.	`"https://example.com/actor/1#main-key"`
`ld_signatures.signature`	string	The signature of the Linked Data in hexadecimal.	`"73a74c990beabe6e59cc68f9c6db7811b59cbb22fd12dcffb3565b651540efe9"`
`ld_signatures.type`	string	The algorithm of the Linked Data signature.	`"RsaSignature2017"`
`object_integrity_proofs.cryptosuite`	string	The cryptographic suite of the object integrity proof.	`"eddsa-jcs-2022"`
`object_integrity_proofs.key_id`	string	The public key ID of the object integrity proof.	`"https://example.com/actor/1#main-key"`
`object_integrity_proofs.signature`	string	The integrity proof of the object in hexadecimal.	`"73a74c990beabe6e59cc68f9c6db7811b59cbb22fd12dcffb3565b651540efe9"`
`url.full`	string	The full URL being fetched by the document loader.	`"https://example.com/actor/1"`
`webfinger.resource`	string	The queried resource URI.	`"acct:fedify@hollo.social"`
`webfinger.resource.scheme`	string	The scheme of the queried resource URI.	`"acct"`

Building observability tools with OpenTelemetry

The OpenTelemetry instrumentation in Fedify provides a powerful foundation for building custom observability tools. By implementing a custom SpanExporter, you can capture and process all the telemetry data generated by Fedify to build tools like debug dashboards, activity monitors, or analytics systems.

Example: ActivityPub debug dashboard

Here's an example of how you might implement a custom SpanExporter to capture ActivityPub activities for a debug dashboard:

import type { SpanExporter, ReadableSpan } from "@opentelemetry/sdk-trace-base";
import { ExportResultCode } from "@opentelemetry/core";

interface InboundActivityRecord {
  direction: "inbound";
  activity: unknown;
  timestamp: Date;
  verified?: boolean;
}

interface OutboundActivityRecord {
  direction: "outbound";
  activityId?: string;
  inboxUrl?: string;
  timestamp: Date;
}

type ActivityRecord = InboundActivityRecord | OutboundActivityRecord;

export class FedifyDebugExporter implements SpanExporter {
  private activities: ActivityRecord[] = [];

  export(spans: ReadableSpan[], resultCallback: (result: { code: ExportResultCode }) => void): void {
    for (const span of spans) {
      // Capture inbound activities
      if (span.name === "activitypub.inbox") {
        const event = span.events.find(
          (e) => e.name === "activitypub.activity.received"
        );
        if (event && event.attributes) {
          this.activities.push({
            direction: "inbound",
            activity: JSON.parse(
              event.attributes["activitypub.activity.json"] as string
            ),
            timestamp: new Date(span.startTime[0] * 1000),
            verified: event.attributes["activitypub.activity.verified"] as boolean,
          });
        }
      }

      // Capture outbound activities
      if (span.name === "activitypub.send_activity") {
        const event = span.events.find(
          (e) => e.name === "activitypub.activity.sent"
        );
        if (event && event.attributes) {
          const activityId = event.attributes[
            "activitypub.activity.id"
          ] as string | undefined;
          const inboxUrl = event.attributes[
            "activitypub.inbox.url"
          ] as string | undefined;
          const activityType = event.attributes[
            "activitypub.activity.type"
          ] as string | undefined;
          const actorId = event.attributes[
            "activitypub.actor.id"
          ] as string | undefined;
          this.activities.push({
            direction: "outbound",
            activityId,
            activityType,
            actorId,
            inboxUrl,
            timestamp: new Date(span.startTime[0] * 1000),
          });
        }
      }
    }
    resultCallback({ code: ExportResultCode.SUCCESS });
  }

  async forceFlush(): Promise<void> {
    // Flush any pending data
  }

  async shutdown(): Promise<void> {
    // Clean up resources
  }

  getActivities(): ActivityRecord[] {
    return this.activities;
  }
}

Integrating the custom exporter

To use the custom exporter, add it to your OpenTelemetry SDK configuration:

import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { createFederation } from "@fedify/fedify";

const debugExporter = new FedifyDebugExporter();
const tracerProvider = new NodeTracerProvider({
  spanProcessors: [new SimpleSpanProcessor(debugExporter)],
});

const federation = createFederation({
  kv: /* your KV store */,
  tracerProvider,
});

Now the debugExporter will receive all telemetry data from Fedify, and you can use debugExporter.getActivities() to access the captured activities for your debug dashboard or other observability tools.

Distributed trace storage with `FedifySpanExporter`

This API is available since Fedify 1.10.0.

The example FedifyDebugExporter shown above stores activities in memory, which works well for single-process applications. However, Fedify applications often run in distributed environments where:

The web server handling HTTP requests runs on different nodes than the background workers processing the message queue.
Multiple worker nodes may process queued messages in parallel.
The debug dashboard itself may run on yet another node.

In such environments, an in-memory exporter cannot aggregate traces across nodes. Each node would only see its own spans, making it impossible to view the complete picture of a distributed trace.

Fedify provides FedifySpanExporter which persists trace data to a KvStore, enabling distributed tracing across multiple nodes. All nodes can write to the same storage, and your debug dashboard can query this shared storage to display complete traces.

Setting up `FedifySpanExporter`

To use FedifySpanExporter, import it from the @fedify/fedify/otel module and configure it with a KvStore:

::: code-group

import type { KvStore, MessageQueue } from "@fedify/fedify";
// ---cut-before---
import { createFederation } from "@fedify/fedify";
import { RedisKvStore } from "@fedify/redis";
import { FedifySpanExporter } from "@fedify/fedify/otel";
import {
  BasicTracerProvider,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import Redis from "ioredis";

const redis = new Redis();
const kv = new RedisKvStore(redis);

// Create the exporter that writes to KvStore
const fedifyExporter = new FedifySpanExporter(kv, {
  ttl: Temporal.Duration.from({ hours: 1 }),
});

const tracerProvider = new BasicTracerProvider({
  spanProcessors: [new SimpleSpanProcessor(fedifyExporter)],
});

const federation = createFederation<void>({
  kv,
  tracerProvider,
// ---cut-start---
  queue: null as unknown as MessageQueue,
// ---cut-end---
  // Omitted for brevity; see the related section for details.
});

import { createFederation } from "@fedify/fedify";
import { RedisKvStore } from "@fedify/redis";
import { FedifySpanExporter } from "@fedify/fedify/otel";
import { NodeTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";
import Redis from "ioredis";

const redis = new Redis();
const kv = new RedisKvStore(redis);

// Create the exporter that writes to KvStore
const fedifyExporter = new FedifySpanExporter(kv, {
  ttl: Temporal.Duration.from({ hours: 1 }),
});

const tracerProvider = new NodeTracerProvider({
  spanProcessors: [new SimpleSpanProcessor(fedifyExporter)],
});

const federation = createFederation({
  kv,
  tracerProvider,
  // Omitted for brevity; see the related section for details.
});

:::

Querying stored traces

The FedifySpanExporter provides methods to query stored trace data:

import { MemoryKvStore } from "@fedify/fedify";
import { FedifySpanExporter } from "@fedify/fedify/otel";
const kv = new MemoryKvStore();
const fedifyExporter = new FedifySpanExporter(kv);
const traceId = "";
// ---cut-before---
// Get all activities for a specific trace
const activities = await fedifyExporter.getActivitiesByTraceId(traceId);

// Get recent traces (with optional limit)
const recentTraces = await fedifyExporter.getRecentTraces({ limit: 100 });

Note

The ~FedifySpanExporter.getRecentTraces() method requires a KvStore implementation that supports the list() method. When using a store that only provides cas() without list() support, this method will return an empty array.

Each TraceActivityRecord contains:

traceId: The OpenTelemetry trace ID
spanId: The OpenTelemetry span ID
parentSpanId: The parent span ID (if any)
direction: "inbound" or "outbound"
activityType: The ActivityPub activity type (e.g., "Create", "Follow")
activityId: The activity's ID URL
actorId: The actor ID URL (sender of the activity)
activityJson: The complete activity JSON
verified: Whether the activity was verified (for inbound activities)
signatureDetails: Detailed signature verification information (for inbound activities), containing:
- httpSignaturesVerified: Whether HTTP Signatures were verified
- httpSignaturesKeyId (optional): The key ID used for HTTP signature verification, if available
- httpSignaturesFailureReason (optional): Why HTTP signature verification failed, if available
- httpSignaturesKeyFetchStatus (optional): The HTTP status code from a failed key fetch, if available
- httpSignaturesKeyFetchError (optional): The error type from a non-HTTP key fetch failure, if available
- ldSignaturesVerified: Whether Linked Data Signatures were verified
timestamp: ISO 8601 timestamp
inboxUrl: The target inbox URL (for outbound activities)

Configuration options

The FedifySpanExporter constructor accepts the following options:

ttl : The time-to-live for stored trace data. If not specified, data will be stored indefinitely (or until manually deleted). This is useful for automatically cleaning up old trace data:

~~~~ typescript twoslash
import { MemoryKvStore } from "@fedify/fedify";
import { FedifySpanExporter } from "@fedify/fedify/otel";
const kv = new MemoryKvStore();
// ---cut-before---
const exporter = new FedifySpanExporter(kv, {
  ttl: Temporal.Duration.from({ hours: 24 }),
});
~~~~

keyPrefix : The key prefix for storing trace data in the KvStore. Defaults to ["fedify", "traces"]. You can customize this to avoid conflicts with other data in the same KvStore:

~~~~ typescript twoslash
import { MemoryKvStore } from "@fedify/fedify";
import { FedifySpanExporter } from "@fedify/fedify/otel";
const kv = new MemoryKvStore();
// ---cut-before---
const exporter = new FedifySpanExporter(kv, {
  keyPrefix: ["myapp", "otel", "traces"],
});
~~~~

`KvStore` requirements

The FedifySpanExporter requires a KvStore that supports either the list() method (preferred) or the cas() method:

When list() is available, the exporter stores each activity record under its own unique key, enabling efficient prefix scans without concurrency issues.
When only cas() is available, the exporter uses compare-and-swap operations to append records to a list, which works but may experience contention under high load.
If neither method is available, the constructor throws an error.

The following KvStore implementations support the required operations:

MemoryKvStore (supports both list() and cas())
RedisKvStore from @fedify/redis (supports both list() and cas())
PostgresKvStore from @fedify/postgres (supports list())
SqliteKvStore from @fedify/sqlite (supports list())
DenoKvStore from @fedify/denokv (supports both list() and cas())
WorkersKvStore from @fedify/cfworkers (supports list())

FilesExpand file tree

opentelemetry.md

Latest commit

History