Mapping

... 2022-8-20 About 12 min

# Mapping

Mapping functions define how chain data is transformed into the optimised GraphQL entities that we have previously defined in the schema.graphql file.

  • Mappings are defined in the src/mappings directory and are exported as a function.
  • These mappings are also exported in src/index.ts.
  • The mappings files are reference in project.yaml under the mapping handlers.

There are different classes of mappings functions depending on the network you are indexing; Block handlers, Event Handlers, Call Handlers, Log Handlers, Transaction Handlers, and Message Handlers.

# Block Handler

You can use block handlers to capture information each time a new block is attached to the chain, e.g. block number. To achieve this, a defined BlockHandler will be called once for every block.

A SubstrateBlock is an extended interface type of signedBlock (opens new window), but also includes the specVersion and timestamp.

A CosmosBlock is a TODO

A TerraBlock is an extended interface type of Terra.js (opens new window) BlockInfo, but also encapsulates the BlockInfo and TxInfo of all transactions in the block.

An AvalancheBlock encapsulates all transactions and events in the block.

# Event Handler

You can use event handlers to capture information when certain events are included on a new block. The events that are part of the default runtime and a block may contain multiple events.

During the processing, the event handler will receive an event as an argument with the event's typed inputs and outputs. Any type of event will trigger the mapping, allowing activity with the data source to be captured. You should use Mapping Filters in your manifest to filter events to reduce the time it takes to index data and improve mapping performance.

A SubstrateEvent is an extended interface type of the EventRecord (opens new window). Besides the event data, it also includes an id (the block to which this event belongs) and the extrinsic inside of this block.

An AvalancheEvent encapsulates Event data and Transaction/Block information corresponding to the event.

A CosmosEvent/TerraEvent encapsulates Event data and TxLog corresponding to the event. It also contains CosmosMessage/TerraMessage data of the message connected to the event. Also, it includes the CosmosBlock/TerraBlock and CosmosTransaction/TerraTransaction data of the block and transaction from which the event was emitted.

Note

From @subql/types version X.X.X onwards SubstrateEvent is now generic. This can provide you with higher type safety when developing your project.

async function handleEvmLog(event: SubstrateEvent<[EvmLog]>): Promise<void> {
  // `eventData` will be of type `EvmLog` before it would have been `Codec`
  const [eventData] = original.event.data;
}
1
2
3
4

# Call Handler

Call handlers (Substrate/Polkadot only) are used when you want to capture information on certain substrate extrinsics. You should use Mapping Filters in your manifest to filter calls to reduce the time it takes to index data and improve mapping performance.

export async function handleCall(extrinsic: SubstrateExtrinsic): Promise<void> {
    const record = new CallEntity(extrinsic.block.block.header.hash.toString());
    record.field4 = extrinsic.block.timestamp;
    await record.save();
}
1
2
3
4
5

The SubstrateExtrinsic (opens new window) extends GenericExtrinsic (opens new window). It is assigned an id (the block to which this extrinsic belongs) and provides an extrinsic property that extends the events among this block. Additionally, it records the success status of this extrinsic.

Note

From @subql/types version X.X.X onwards SubstrateExtrinsic is now generic. This can provide you with higher type safety when developing your project.

async function handleEvmCall(call: SubstrateExtrinsic<[TransactionV2 | EthTransaction]>): Promise<void> {
  // `tx` will be of type `TransactionV2 | EthTransaction` before it would have been `Codec`
  const [tx] = original.extrinsic.method.args;
}
1
2
3
4

# Log Handler

You can use log handlers (Avalanche only) to capture information when certain logs are included on transactions. During the processing, the log handler will receive a log as an argument with the log's typed inputs and outputs. Any type of event will trigger the mapping, allowing activity with the data source to be captured. You should use Mapping Filters in your manifest to filter events to reduce the time it takes to index data and improve mapping performance.

import { AvalancheLog } from "@subql/types-avalanche";

export async function handleLog(event: AvalancheLog): Promise<void> {
  const record = new EventEntity(
    `${event.blockHash}-${event.logIndex}`
  );
  record.blockHeight = BigInt(event.blockNumber);
  record.topics = event.topics; // Array of strings
  record.data = event.data;
  await record.save();
}
1
2
3
4
5
6
7
8
9
10
11

# Transaction Handler

You can use transaction handlers (Avalanche and Terra only) to capture information about each of the transactions in a block. To achieve this, a defined TransactionHandler will be called once for every transaction. You should use Mapping Filters in your manifest to filter transactions to reduce the time it takes to index data and improve mapping performance.

The CosmosTransaction/TerraTransaction encapsulates TxInfo and the corresponding CosmosBlock/TerraBlock in which the transaction occured.

The AvalancheTransaction encapsulates TxInfo and the corresponding block information in which the transaction occured.

# Message Handler

You can use message handlers to capture information from each message in a transaction. To achieve this, a defined MessageHandler will be called once for every message. You should use Mapping Filters in your manifest to filter messages to reduce the time it takes to index data and improve mapping performance.

CosmosMessage/TerraMessage encapsulates the msg object containing the message data, the CosmosTransaction/TerraTransaction in which the message occured in and also the CosmosBlock/TerraBlock in which the transaction occured in.

# The Sandbox

SubQuery is deterministic by design, that means that each SubQuery project is guranteed to index the same data set. This is a critical factor that is required to decentralise SubQuery in the SubQuery Network. This limitation means that the indexer is by default run in a strict virtual machine, with access to a strict number of third party libraries.

You can bypass this limitation, allowing you to index and retrieve information from third party data sources like HTTP endpoints, non historical RPC calls, and more. In order to do to, you must run your project in unsafe-mode, you can read more about this in the references. An easy way to do this while developing (and running in Docker) is to add the following line to your docker-compose.yml:

subquery-node:
  image: onfinality/subql-node:latest
  ...
  command:
    - -f=/app
    - --db-schema=app
    - --unsafe
  ...
1
2
3
4
5
6
7
8

By default, the VM2 (opens new window) sandbox only allows the folling:

  • only some certain built-in modules, e.g. assert, buffer, crypto,util and path
  • third-party libraries written by CommonJS.
  • hybrid libraries like @polkadot/* that uses ESM as default. However, if any other libraries depend on any modules in ESM format, the virtual machine will NOT compile and return an error.
  • Historical/safe queries, see RPC Calls.
  • Note HTTP and WebSocket connections are forbidden

# Modules and Libraries

To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the sandbox, and have allowed users to call third-party libraries.

Please note this is an experimental feature and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in GitHub (opens new window).

# Built-in modules

Currently, we allow the following NodeJS modules: assert, buffer, crypto, util, and path.

Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.

import {hashMessage} from "ethers/lib/utils"; //Good way
import {utils} from "ethers" //Bad way

export async function handleCall(extrinsic: SubstrateExtrinsic): Promise<void> {
    const record = new CallEntity(extrinsic.block.block.header.hash.toString());
    record.field1 = hashMessage('Hello');
    await record.save();
}
1
2
3
4
5
6
7
8

# Query States

Our goal is to cover all data sources for users for mapping handlers (more than just the three interface event types above). Therefore, we have exposed some of the @polkadot/api interfaces to increase capabilities.

These are the interfaces we currently support:

These are the interfaces we do NOT support currently:

  • api.tx.*
  • api.derive.*
  • api.query.<module>.<method>.at
  • api.query.<module>.<method>.entriesAt
  • api.query.<module>.<method>.entriesPaged
  • api.query.<module>.<method>.hash
  • api.query.<module>.<method>.keysAt
  • api.query.<module>.<method>.keysPaged
  • api.query.<module>.<method>.range
  • api.query.<module>.<method>.sizeAt

See an example of using this API in our validator-threshold (opens new window) example use case.

# RPC calls

We also support some API RPC methods that are remote calls that allow the mapping function to interact with the actual node, query, and submission.

Documents in JSON-RPC (opens new window) provide some methods that take BlockHash as an input parameter (e.g. at?: BlockHash), which are now permitted. We have also modified these methods to take the current indexing block hash by default.

// Let's say we are currently indexing a block with this hash number
const blockhash = `0x844047c4cf1719ba6d54891e92c071a41e3dfe789d064871148e9d41ef086f6a`;

// Original method has an optional input is block hash
const b1 = await api.rpc.chain.getBlock(blockhash);

// It will use the current block has by default like so
const b2 = await api.rpc.chain.getBlock();
1
2
3
4
5
6
7
8

# Chain Type Registries

Some decoded message data from Cosmos Chains has nested message types that don't get decoded.

We inject the registry (opens new window) globally into the sandbox so that users can decode more messages as they need.

  
import {MsgUpdateClient} from "cosmjs-types/ibc/core/client/v1/tx";
  
registry.register("/ibc.core.client.v1.MsgUpdateClient", MsgUpdateClient)
1
2
3
4

# Custom Substrate Chains

SubQuery can be used on any Substrate-based chain, not just Polkadot or Kusama.

You can use a custom Substrate-based chain and we provide tools to import types, interfaces, and additional methods automatically using @polkadot/typegen (opens new window).

In the following sections, we use our kitty example (opens new window) to explain the integration process.

# Preparation

Create a new directory api-interfaces under the project src folder to store all required and generated files. We also create an api-interfaces/kitties directory as we want to add decoration in the API from the kitties module.

Metadata

We need metadata to generate the actual API endpoints. In the kitty example, we use an endpoint from a local testnet, and it provides additional types. Follow the steps in PolkadotJS metadata setup (opens new window) to retrieve a node's metadata from its HTTP endpoint.

curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933
1

or from its websocket endpoint with help from websocat (opens new window):

//Install the websocat
brew install websocat

//Get metadata
echo state_getMetadata | websocat 'ws://127.0.0.1:9944' --jsonrpc
1
2
3
4
5

Next, copy and paste the output to a JSON file. In our kitty example (opens new window), we have created api-interface/kitty.json.

Type definitions

We assume that the user knows the specific types and RPC support from the chain, and it is defined in the Manifest.

Following types setup (opens new window), we create :

  • src/api-interfaces/definitions.ts - this exports all the sub-folder definitions
export { default as kitties } from './kitties/definitions';
1
  • src/api-interfaces/kitties/definitions.ts - type definitions for the kitties module
export default {
    // custom types
    types: {
        Address: "AccountId",
        LookupSource: "AccountId",
        KittyIndex: "u32",
        Kitty: "[u8; 16]"
    },
    // custom rpc : api.rpc.kitties.getKittyPrice
    rpc: {
        getKittyPrice:{
            description: 'Get Kitty price',
            params: [
                {
                    name: 'at',
                    type: 'BlockHash',
                    isHistoric: true,
                    isOptional: false
                },
                {
                    name: 'kittyIndex',
                    type: 'KittyIndex',
                    isOptional: false
                }
            ],
            type: 'Balance'
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Packages

  • In the package.json file, make sure to add @polkadot/typegen as a development dependency and @polkadot/api as a regular dependency (ideally the same version). We also need ts-node as a development dependency to help us run the scripts.
  • We add scripts to run both types; generate:defs and metadata generate:meta generators (in that order, so metadata can use the types).

Here is a simplified version of package.json. Make sure in the scripts section the package name is correct and the directories are valid.

{
  "name": "kitty-birthinfo",
  "scripts": {
    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package kitty-birthinfo/api-interfaces --input ./src/api-interfaces",
    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package kitty-birthinfo/api-interfaces --endpoint ./src/api-interfaces/kitty.json --output ./src/api-interfaces --strict"
  },
  "dependencies": {
    "@polkadot/api": "^4.9.2"
  },
  "devDependencies": {
    "typescript": "^4.1.3",
    "@polkadot/typegen": "^4.9.2",
    "ts-node": "^8.6.2"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Type generation

Now that preparation is completed, we are ready to generate types and metadata. Run the commands below:

# Yarn to install new dependencies
yarn

# Generate types
yarn generate:defs
1
2
3
4
5

In each modules folder (eg /kitties), there should now be a generated types.ts that defines all interfaces from this modules' definitions, also a file index.ts that exports them all.

# Generate metadata
yarn generate:meta
1
2

This command will generate the metadata and a new api-augment for the APIs. As we don't want to use the built-in API, we will need to replace them by adding an explicit override in our tsconfig.json. After the updates, the paths in the config will look like this (without the comments):

{
  "compilerOptions": {
      // this is the package name we use (in the interface imports, --package for generators) */
      "kitty-birthinfo/*": ["src/*"],
      // here we replace the @polkadot/api augmentation with our own, generated from chain
      "@polkadot/api/augment": ["src/interfaces/augment-api.ts"],
      // replace the augmented types with our own, as generated from definitions
      "@polkadot/types/augment": ["src/interfaces/augment-types.ts"]
    }
}
1
2
3
4
5
6
7
8
9
10

# Usage

Now in the mapping function, we can show how the metadata and types actually decorate the API. The RPC endpoint will support the modules and methods we declared above. And to use custom rpc call, please see section Custom chain rpc calls

export async function kittyApiHandler(): Promise<void> {
    //return the KittyIndex type
    const nextKittyId = await api.query.kitties.nextKittyId();
    // return the Kitty type, input parameters types are AccountId and KittyIndex
    const allKitties  = await api.query.kitties.kitties('xxxxxxxxx',123)
    logger.info(`Next kitty id ${nextKittyId}`)
    //Custom rpc, set undefined to blockhash
    const kittyPrice = await api.rpc.kitties.getKittyPrice(undefined,nextKittyId);
}
1
2
3
4
5
6
7
8
9

If you wish to publish this project to our explorer, please include the generated files in src/api-interfaces.

# Custom chain RPC calls

To support customised chain RPC calls, we must manually inject RPC definitions for typesBundle, allowing per-spec configuration. You can define the typesBundle in the project.yml. And please remember only isHistoric type of calls are supported.

...
  types: {
    "KittyIndex": "u32",
    "Kitty": "[u8; 16]",
  }
  typesBundle: {
    spec: {
      chainname: {
        rpc: {
          kitties: {
            getKittyPrice:{
                description: string,
                params: [
                  {
                    name: 'at',
                    type: 'BlockHash',
                    isHistoric: true,
                    isOptional: false
                  },
                  {
                    name: 'kittyIndex',
                    type: 'KittyIndex',
                    isOptional: false
                  }
                ],
                type: "Balance",
            }
          }
        }
      }
    }
  }

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Last update: August 20, 2022 04:32