Sending spans

Telmoni takes spans over OpenTelemetry's own protocol, OTLP/HTTP, at one endpoint:

POST https://api.telmoni.com/otlp/v1/traces

(On a self-hosted instance, replace https://api.telmoni.com with your deployment's own machine host, or with its console's host where the lanes are routed by path: Self-hosting → Production.)

The wire is OTLP's, unchanged: an ExportTraceServiceRequest, as Content-Type: application/json in OTLP's JSON encoding or application/x-protobuf, gzipped (Content-Encoding: gzip) or not. What is Telmoni's is which attributes are read, and what is done with each, and the Telmoni-Sent-At header the SDKs add, by which a clock set wrong is read (what is kept).

The keyCopy Anchor Link

Send a project API key holding the ingest scope as a bearer token:

Authorization: Bearer telmoni_...

Every span of the export is the key's project's. No key, or a revoked, expired or unknown one, answers 401 /errors/auth/invalid-token; a live key without the ingest scope answers 403 /errors/authz/forbidden.

What a span must carryCopy Anchor Link

Spans are kept inside a run: one execution of an agent. Every span of a run carries the run's id, telmoni.run_id, a UUID minted as the run starts and the same on each of its spans, and the run's agent, telmoni.agent, up to 128 characters. A span without both is refused, one by one, and named in the answer, so a stock OpenTelemetry exporter sends nothing that is kept unless the application sets both on every span itself.

What a span is, its kind, follows gen_ai.operation.name: invoke_agent or invoke_workflow is the run itself; chat, generate_content, text_completion or embeddings a model call; execute_tool a tool call; any other span opened under the telmoni instrumentation scope a step, and one opened elsewhere a plain span. A run is named by gen_ai.agent.name (or gen_ai.workflow.name), a tool call by gen_ai.tool.name, and every other span by its own name, 1 to 145 characters: room for a model call's <operation> <model>, the longest operation and a model at its cap.

What is keptCopy Anchor Link

Each field below is a column, read from the attribute named, and the names are the GenAI semantic conventions' — the ones they replaced are read as the current ones (gen_ai.system, gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.usage.cache_creation.input_tokens):

A field whose value breaks its rule refuses the span whole; an attribute past its cap is dropped and the span kept.

A span's start and end are kept as sent, to the microsecond, unless the clock that timed them is a time zone or a date out. The SDKs send their clock with every export, as Telmoni-Sent-At, in nanoseconds since the Unix epoch; where it stands more than 55 minutes from the server's as the export arrives, ahead or behind, every span of the export is moved by the difference, each keeping its length, so every export of a run moves alike and its parts keep their places. A recorded run's times are read as that clock's too. A span is sent once it has ended, so an export holding one that ends more than 55 minutes after the clock it names was not timed by that clock, or was mistyped; it is read as though it named none. Where none is named, as from another exporter, the spans one resource sends of a trace in an export are moved back together when they end more than 55 minutes after the export arrived, until the last ends as it arrived, so a run sent over several exports is moved export by export, its parts perhaps landing seconds apart, or further where an export arrives late. A clock drifting by less is left as it is.

The field inventory names, on a span, gen_ai.response.id, gen_ai.response.finish_reasons, gen_ai.response.status, gen_ai.request.reasoning.level, gen_ai.agent.version, gen_ai.tool.call.id, gen_ai.tool.type, gen_ai.output.type, gen_ai.prompt.name, gen_ai.prompt.version, gen_ai.data_source.id and server.address, and on the resource service.name and every telemetry.sdk.* and telemetry.distro.*; the resource's deployment.environment.name and service.version are the environment and release columns above, never attributes.

What is dropped at the doorCopy Anchor Link

Content — gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.retrieval.query.text, gen_ai.retrieval.documents, gen_ai.memory.query.text, gen_ai.memory.records, gen_ai.prompt.variable, and the user.email, user.full_name and user.name that name a person — is dropped on a project whose content mode is off, which is every project today, and never reaches a store. The span is kept, and the answer names the field.

Any other text, or array, the field inventory does not name is dropped in every mode as unlisted: content arrives under names no convention gave it. A span's events and its status message are never kept.

The answerCopy Anchor Link

200 with OTLP's ExportTraceServiceResponse in the request's encoding, once the rows are stored, and the header Telmoni-Content-Mode naming the project's mode. partialSuccess is null in JSON, and absent in protobuf, when no span was refused and no attribute dropped; otherwise rejectedSpans counts the spans refused whole, and errorMessage names each span refused and each attribute dropped, with why — its first 4 KiB, then a count of the rest:

{
  "partialSuccess": {
    "rejectedSpans": 1,
    "errorMessage": "span 0202020202020202 refused: no telmoni.run_id and telmoni.agent; a span outside a Telmoni run is refused\nspan 00f067aa0ba902b7 dropped gen_ai.input.messages: content; the project's content mode is off"
  }
}

An exporter retries with the same bytes when an answer was lost. Where the server reaches its Redis, an export sent again within an hour, byte for byte, gets its first answer again and is not written twice, and a copy that arrives while the first is still being written is answered 503 with Retry-After: 1; without Redis, every copy is written.

RefusalsCopy Anchor Link

A refusal is OTLP's Status, in the request's encoding, its message opening with the error type and its code the gRPC code the status maps to:

{ "code": 3, "message": "/errors/telemetry/invalid-batch: the export holds no span" }

429 and 503 are the two an exporter retries; 400, 401, 402, 403, 404, 405, 413 and 500 it does not, though a 500 for the database is worth sending again later.

Every refusal of an export waits for its body, up to 60 seconds, so an exporter still sending one reads the refusal rather than a broken connection. A body past 1 MiB is not read past that MiB, its connection closed with the refusal; one sent with Expect: 100-continue is refused without being asked for.