Stellar & Soroban Manifest File [Beta]

SubQuery TeamAbout 8 min

Stellar & Soroban Manifest File [Beta]

The Manifest project.ts file can be seen as an entry point of your project and it defines most of the details on how SubQuery will index and transform the chain data. It clearly indicates where we are indexing data from, and to what on chain events we are subscribing to.

The Manifest can be in either Typescript, Yaml, or JSON format.

With the number of new features we are adding to SubQuery, and the slight differences between each chain that mostly occur in the manifest, the project manifest is now written by default in Typescript. This means that you get a fully typed project manifest with documentation and examples provided your code editor.

Below is a standard example of a basic project.ts.

import {
  StellarDatasourceKind,
  StellarHandlerKind,
  StellarProject,
} from "@subql/types-stellar";
import { Horizon } from "stellar-sdk";

/* This is your project configuration */
const project: StellarProject = {
  specVersion: "1.0.0",
  name: "soroban-testnet-starter",
  version: "0.0.1",
  runner: {
    node: {
      name: "@subql/node-stellar",
      version: "*",
    },
    query: {
      name: "@subql/query",
      version: "*",
    },
  },
  description:
    "This project can be use as a starting point for developing your new Stellar SubQuery project (testnet)",
  repository: "https://github.com/subquery/stellar-subql-starter",
  schema: {
    file: "./schema.graphql",
  },
  network: {
    /* Stellar and Soroban uses the network passphrase as the chainId
      'Test SDF Network ; September 2015' for testnet
      'Public Global Stellar Network ; September 2015' for mainnet
      'Test SDF Future Network ; October 2022' for Future Network */
    chainId: "Test SDF Network ; September 2015",
    /* This Stellar endpoint must be a public non-pruned archive node
      We recommend providing more than one endpoint for improved reliability, performance, and uptime
      Public nodes may be rate limited, which can affect indexing speed
      When developing your project we suggest getting a private API key */
    endpoint: ["https://horizon-testnet.stellar.org"],
    /* This is a specific Soroban endpoint
      It is only required when you are using a soroban/EventHandler */
    sorobanEndpoint: "https://soroban-testnet.stellar.org",
    /* Recommended to provide the HTTP endpoint of a full chain dictionary to speed up processing
      dictionary: "https://gx.api.subquery.network/sq/subquery/eth-dictionary" */
  },
  dataSources: [
    {
      kind: StellarDatasourceKind.Runtime,
      /* Set this as a logical start block, it might be block 1 (genesis) or when your contract was deployed */
      startBlock: 1700000,
      mapping: {
        file: "./dist/index.js",
        handlers: [
          {
            handler: "handleOperation",
            kind: StellarHandlerKind.Operation,
            filter: {
              type: Horizon.OperationResponseType.payment,
            },
          },
          {
            handler: "handleCredit",
            kind: StellarHandlerKind.Effects,
            filter: {
              type: "account_credited",
            },
          },
          {
            handler: "handleDebit",
            kind: StellarHandlerKind.Effects,
            filter: {
              type: "account_debited",
            },
          },
          {
            handler: "handleEvent",
            kind: StellarHandlerKind.Event,
            filter: {
              /* You can optionally specify a smart contract address here
                contractId: "" */
              topics: [
                "transfer", // Topic signature(s) for the events, there can be up to 4
              ],
            },
          },
        ],
      },
    },
  ],
};

// Must set default to the project instance
export default project;

Below is a standard example of the legacy YAML version (project.yaml).

Legacy YAML Manifest

specVersion: "1.0.0"

name: "soroban-subql-starter"
version: "0.0.1"
runner:
  node:
    name: "@subql/node-stellar"
    version: "*"
  query:
    name: "@subql/query"
    version: "*"
description: "This project can be use as a starting point for developing your new Stellar Soroban Future Network SubQuery project"
repository: "https://github.com/subquery/stellar-subql-starter"

schema:
  file: "./schema.graphql"

network:
  # Stellar and Soroban uses the network passphrase as the chainId
  # 'Public Global Stellar Network ; September 2015' for mainnet
  # 'Test SDF Future Network ; October 2022' for Future Network
  chainId: "Test SDF Future Network ; October 2022"
  # This endpoint must be a public non-pruned archive node
  # We recommend providing more than one endpoint for improved reliability, performance, and uptime
  # Public nodes may be rate limited, which can affect indexing speed
  endpoint: ["https://horizon-futurenet.stellar.org:443"]
  sorobanEndpoint: "https://rpc-futurenet.stellar.org"
  # Recommended to provide the HTTP endpoint of a full chain dictionary to speed up processing
  # dictionary: "https://gx.api.subquery.network/sq/subquery/eth-dictionary"

dataSources:
  - kind: stellar/Runtime
    startBlock: 270000 # This is the start block from which you begin indexing
    mapping:
      file: "./dist/index.js"
      handlers:
        - handler: handleTransaction
          kind: soroban/TransactionHandler
           filter:
            account: "GAKNXHJ5PCZYFIBNBWB4RCQHH6GDEO7Z334N74BOQUQCHKOURQEPMXCH"
        - handler: handleOperation
          kind: stellar/OperationHandler
          filter:
            type: "payment"
        - handler: handleCredit
          kind: stellar/EffectHandler
          filter:
            type: "account_credited"
        - handler: handleEvent
          kind: stellar/EventHandler
          filter:
            # contractId: "" # You can optionally specify a smart contract address here
            topics:
              - "transfer" # Topic signature(s) for the events, there can be up to 4

Overview

Top Level Spec

Field	Type	Description
specVersion	String	The spec version of the manifest file
name	String	Name of your project
version	String	Version of your project
description	String	Description of your project
runner	Runner Spec	Runner specs info
repository	String	Git repository address of your project
schema	Schema Spec	The location of your GraphQL schema file
network	Network Spec	Detail of the network to be indexed
dataSources	DataSource Spec	The datasource to your project
templates	Templates Spec	Allows creating new datasources from this templates

Schema Spec

Field	Type	Description
file	String	The location of your GraphQL schema file

Network Spec

If you start your project by using the subql init command, you'll generally receive a starter project with the correct network settings. If you are changing the target chain of an existing project, you'll need to edit the Network Spec section of this manifest.

The chainId is the network identifier of the blockchain, Stellar and Soroban uses the network passphrase. Examples in Stellar are Public Global Stellar Network ; September 2015 for mainnet and Test SDF Future Network ; October 2022 for Future Network.

A unique aspect of Stellar's Soroban network is that there is an additional seperate RPC for the Soroban smart contract layer. If you are indexing using the soroban/TransactionHandler or soroban/EventHandler, you will need to provide an HTTP RPC Soroban endpoint under the sorobanEndpoint property - you will want archive nodes with high rate limits if you want to index large amounts of historical data. This only accepts a single endpoint.

You will need to update the endpoint. This defines the HTTP RPC endpoint of the blockchain to be indexed - you will want archive nodes with high rate limits if you want to index large amounts of historical data. This property can be a string or an array of strings (e.g. endpoint: ['rpc1.endpoint.com', 'rpc2.endpoint.com']). We suggest providing an array of endpoints as it has the following benefits:

Increased speed - When enabled with worker threads, RPC calls are distributed and parallelised among RPC providers. Historically, RPC latency is often the limiting factor with SubQuery.
Increased reliability - If an endpoint goes offline, SubQuery will automatically switch to other RPC providers to continue indexing without interruption.
Reduced load on RPC providers - Indexing is a computationally expensive process on RPC providers, by distributing requests among RPC providers you are lowering the chance that your project will be rate limited.

Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider.

Field	Type	Description
chainId	String	A network identifier for the blockchain, Stellar and Soroban uses the network passphrase
endpoint	String or String[]	Defines the endpoint of the blockchain to be indexed - you will want archive nodes with high rate limits if you want to index large amounts of historical datae.
sorobanEndpoint	String	Defines the soroban RPC endpoint - you will want archive nodes with high rate limits if you want to index large amounts of historical data.
dictionary	String	It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read how a SubQuery Dictionary works.
bypassBlocks	Array	Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see Bypass Blocks

Runner Spec

Field	Type	Description
node	Runner node spec	Describe the node service use for indexing
query	Runner query spec	Describe the query service

Runner Node Spec

Field	Type	Description
name	String	`@subql/node-stellar`
version	String	Version of the indexer Node service, it must follow the SEMVER rules or `latest`, you can also find available versions in subquery SDK releases
options	Runner Node Options	Runner specific options for how to run your project. These will have an impact on the data your project produces. CLI flags can be used to override these.

Runner Query Spec

Field	Type	Description
name	String	`@subql/query` and `@subql/query-subgraph`
version	String	Version of the Query service, available `@subql/query` versions and `@subql/query-subgraph` versions, it also must follow the SEMVER rules or `latest`.

Runner Node Options

Field	v1.0.0 (default)	Description
historical	Boolean (true)	Historical indexing allows you to query the state at a specific block height. e.g A users balance in the past.
unfinalizedBlocks	Boolean (false)	If enabled unfinalized blocks will be indexed, when a fork is detected the project will be reindexed from the fork. Requires historical.
unsafe	Boolean (false)	Removes all sandbox restrictions and allows access to all inbuilt node packages as well as being able to make network requests. WARNING: this can make your project non-deterministic.

Datasource Spec

Defines the data that will be filtered and extracted and the location of the mapping function handler for the data transformation to be applied.

Field	Type	Description
kind	string	stellar/Runtime
startBlock	Integer	This changes your indexing start block for this datasource, set this as high as possible to skip initial blocks with no relevant data
endBlock	Integer	This sets a end block for processing on the datasource. After this block is processed, this datasource will no longer index your data. Useful when your contracts change at a certain block height, or when you want to insert data at genesis. For example, setting both the `startBlock` and `endBlock` to 320, will mean this datasource only operates on block 320
mapping	Mapping Spec

Mapping Spec

Field	Type	Description
handlers & filters	Default handlers and filters	List all the mapping functions and their corresponding handler types, with additional mapping filters.

Data Sources and Mapping

In this section, we will talk about the default Stellar runtime and its mapping. Here is an example:

{
  ...
  dataSources: [
    {
      kind: StellarDataSourceKind.Runtime, // Indicates that this is default runtime
      startBlock: 1, // This changes your indexing start block, set this higher to skip initial blocks with less data
      mapping: {
        file: "./dist/index.js", // Entry path for this mapping
        handlers: [
          /* Enter handers here */
        ],
      }
    }
  ]
}

Mapping Handlers and Filters

The following table explains filters supported by different handlers.

Handler	Supported filter
stellar/BlockHandler	`modulo` and `timestamp`
stellar/TransactionHandler	`account` (address from)
soroban/TransactionHandler	`account` (address from)
stellar/OperationHandler	`type`, `sourceAccount`
stellar/EffectHandler	`type`, `account` (address from)
soroban/EventHandler	`contractId` and/or up to 4 `topics` filters applied as an array, and an optional `contractId`

handlers:
  - handler: handleBlock
    kind: soroban/BlockHandler
  - handler: handleTransaction
    kind: soroban/TransactionHandler
    filter:
      account: "GAKNXHJ5PCZYFIBNBWB4RCQHH6GDEO7Z334N74BOQUQCHKOURQEPMXCH"
  - handler: handleOperation
    kind: stellar/OperationHandler
    filter:
      type: "payment"
  - handler: handleEffect
    kind: stellar/EffectHandler
    filter:
      type: "account_credited"
  - handler: handleEvent
    kind: soroban/EventHandler
    filter:
      # contractId: "" # You can optionally specify a smart contract address here
      topics:
        - "transfer" # Topic signature(s) for the events, there can be up to 4

Default runtime mapping filters are an extremely useful feature to decide what event will trigger a mapping handler.

Only incoming data that satisfies the filter conditions will be processed by the mapping functions. Mapping filters are optional but are highly recommended as they significantly reduce the amount of data processed by your SubQuery project and will improve indexing performance.

The modulo filter allows handling every N blocks, which is useful if you want to group or calculate data at a set interval. The following example shows how to use this filter.

filter:
  modulo: 50 # Index every 50 blocks: 0, 50, 100, 150....

The timestamp filter is very useful when indexing block data with specific time intervals between them. It can be used in cases where you are aggregating data on a hourly/daily basis. It can be also used to set a delay between calls to blockHandler functions to reduce the computational costs of this handler.

The timestamp filter accepts a valid cron expression and runs on schedule against the timestamps of the blocks being indexed. Times are considered on UTC dates and times. The block handler will run on the first block that is after the next iteration of the cron expression.

filter:
  # This cron expression will index blocks with at least 5 minutes interval
  # between their timestamps starting at startBlock given under the datasource.
  timestamp: "*/5 * * * *"

Note

We use the cron-converter package to generate unix timestamps for iterations out of the given cron expression. So, make sure the format of the cron expression given in the timestamp filter is compatible with the package.

Some common examples

  # Every minute
  timestamp: "* * * * *"
  # Every hour on the hour (UTC)
  timestamp: "0 * * * *"
  # Every day at 1am UTC
  timestamp: "0 1 * * *"
  # Every Sunday (weekly) at 0:00 UTC
  timestamp: "0 0 * * 0"

Simplifying your Project Manifest for a large number contract addresses

If your project has the same handlers for multiple versions of the same type of contract your project manifest can get quite repetitive. e.g you want to index the transfers for many similar ERC20 contracts, there are ways to better handle a large static list of contract addresses.

Note that there is also dynamic datasources for when your list of addresses is dynamic (e.g. you use a factory contract).

Real-time indexing (Block Confirmations)

As indexers are an additional layer in your data processing pipeline, they can introduce a massive delay between when an on-chain event occurs and when the data is processed and able to be queried from the indexer.

SubQuery provides real time indexing of unconfirmed data directly from the RPC endpoint that solves this problem. SubQuery takes the most probabilistic data before it is confirmed to provide to the app. In the unlikely event that the data isn’t confirmed and a reorg occurs, SubQuery will automatically roll back and correct its mistakes quickly and efficiently - resulting in an insanely quick user experience for your customers.

To control this feature, please adjust the --block-confirmations command to fine tune your project and also ensure that historic indexing is enabled (enabled by default)

Bypass Blocks

Bypass Blocks allows you to skip the stated blocks, this is useful when there are erroneous blocks in the chain or when a chain skips a block after an outage or a hard fork. It accepts both a range or single integer entry in the array.

When declaring a range use an string in the format of "start - end". Both start and end are inclusive, e.g. a range of "100-102" will skip blocks 100, 101, and 102.

{
  network: {
    bypassBlocks: [1, 2, 3, "105-200", 290];
  }
}