Acuity Index

Kusama Treasury Proposal: omni-indexer

Track: Small Spender
Value: 256 KSM (~4000 USD)

Project background

Acuity Index was originally called Hybrid and was funded by two (1, 2) Web3 Foundation grants and a Kusama Treasury referenda.

It solves a critical issue in the Polkadot ecosystem - it enables dapps with rich functionality to be fully decentralized.

Currently a fully decentralized Polkadot dapp can connect to any RPC for the chain and query the state. Results can even be verified by light client. This is how the Polkadot Developer Interface works. However, maintaining this level of decentralization is currently very limiting. For a rich UI it is necessary to query what has happened in the past and this is not currently possible to do in a decentralized way. For example, even a simple wallet dapp cannot display account transaction history without querying some centralized database.

Dapps tend to build their own centralized backends so the user interface can be fast and rich. This, of course, undermines some of the most important properties of a dapp: permissionless and trustless access to the system.

Solving this issue is key to the mission of Polkadot.

A record of everything that happens on a Polkadot chain is recorded in blocks in the form of events, but these are not indexed. For example, it is not possible to query for all the events related to a specific AccountId.

An Acuity Index node provides a standardized RPC interface for event querying in the same manner that a regular full node provides a standardized RPC interface for state queries. Therefore a dapp can be written in such a way that it is not tied to a single centralized backend.

Previous Kusama Treasury Referendum

The deliverables from the previous referenda were largely completed and new version (blog post) of acuity-index-substrate and acuity-index-polkadot released.

It was anticipated that this release would include returning light client proofs with index results, but this proved more complicated than anticipated and will be addressed in a later version.

Don't decode events into specific Rust types

Currently Acuity Index works by decoding each event into a Rust type that has been generated automatically by the subxt macro by calling EventDetails::as_root_event()

This has a number of problems. Currently event types are only derived for the latest runtime for each chain. If events have been modified or removed during chain upgrades the latest version of the indexer may not be able to index certain events from older blocks.

One solution to this would be to derive event types and maintain indexing macros for all runtimes that a chain has had. The burden of maintaining the indexer for each chain would be large and complex.

During build the subxt macro is extremely slow and requires a considerable amount of RAM. Running it for every individual runtime would increase the build requirements considerably making maintenance even harder.

Converting the event into dedicated types also increases the CPU time required during indexing.

Consumers of the index (dapps) typically will not want to decode events into runtime-specific types. This would add to the complexity and maintenance burden of the dapp.

It is unusual for the schema of an individual event to change over a runtime upgrade. Typically events are added or removed.

A much better approach is to call EventDetails::field_values()

This has a number of advantages:

  • separate indexers do not need to be built for each chain
  • macros do not need to be written for custom pallets
  • chain indexing can be specified in a simple TOML file
  • a single specification can handle indexing all of a chain's runtime versions
  • decoded parameters for each individual event can be stored and returned in query results

A chain index specification will include the following config items:

  • name
  • genesis hash
  • list of block numbers where index updates have occured - re-indexing will occur as necessary upon upgrade of Acuity Index or index specification
  • default full node url
  • Polkadot SDK version - Acuity Index must be updated to support it
  • for each current or historic pallet either specify:
    • default for Polkadot SDK built-in pallets
    • custom for each event variant specify which parameters should be indexed by which key

Deliverables

  • Update to subxt v0.43.0
  • combine acuity-index-substrate and acuity-index-polkadot into acuity-index
  • replace decoding events into Rust types with schema-less event parameters
  • replace building chain-specific indexers with a configurable omni-indexer
  • replace storing and serving block event bytes with schema-less events

Point of Contact

  • name: Jonathan Brown
  • email: jbrown@acuity.network
  • Telegram: ethernomad
  • GitHub: ethernomad
  • X: bluedroplet

Timeline

The deliverables will be completed by the end of August 2025.

The Problem →