What do we need?

• A a clear way on how to structure tracing spans and events to discriminate properly what is happening and where is it happening.
• Being able to enable/disable tracing, both in compile time and in runtime
• Offer different types of log output formats (Ex: opentelemetry, gelf, …)
• Aggregate logs from different sources (nodes) for our internal debugging

Node span

For debugging purposes the node will span a global span, the span must include a debug_idfor the internal testnet in where we will be aggregating the trace logs we need to discriminate nodes. It can be auto-generated and should not link to any other identification (network, consensus, etc).

Services spans

Services information should be easy to discriminate and easy to track and understand.

• Each service need to create its own global Span

• Each subsequent span (per service) need to be properly discriminated by a level (Trace, Debug, Info, Warn, Error)

  • Just use trace and debug for anything that helps us as devs to add information about what is going around. Ex: track methods calls.
  • info for any information that may be of use for operators (only)
  • warn and error critical failures

• Each core service will spawn a root span from where sub spans inherit.

  • Each root span must be named after the SERVICE_ID

    async run(...) {
    	let service_root_span = debug_span!(<Self as ServiceData>::SERVICE_ID)
    	async {
    	...
    	}.instrument(service_root_span)
    

• Services spans/logs should not share any sensitive information about the node itself like:

  • Ids
  • Keys (public nor private)
  • Stake
  • IP

• Events records:

  • Should not include duplicated information
  • If possible incorporate the most generic and important records as close to the parent as possible so they can be re-utilized.

Spans structure visualization

Nomos core services