[Proposal: MB11 & 30 / MR8 & 26] StakeGlmr.com and StakeMovr.com V2 - Treasury Proposal

stakebaby · August 2, 2024, 7:12am

Project Report - M2 Delivery

Code Stats Summary

Component	Language	Lines of Code
SvelteKit App	Typescript	14,740
SvelteKit App	Svelte	9,912
Chat Server	Typescript	4,346
Staking Indexer	Typescript	11,730
Deep State Indexer	Typescript	8,888

Total Typescript Lines: 39,704

Initial Schedule: 2-6 months Actual Time: 1 year and 2 months

Milestone Status

Milestone 1

SvelteKit frontend app, with mock db calls
Integration of app with wallets, chain queries (TODO: extrinsic calls)
Indexing backend and ETL services

Milestone 2

Additional microservices (pending only the collator management services)
Full frontend + backend integration (pending only the integration of the above)
Backend Testing
TODO: Frontend Testing
TODO: Deployment to production

Features planned and delivered

New dashboards with more staking info for collators
New personal account dashboards for delegators
Faster load time - V2 loads all data straight from the database on a single http call. Collator details data load on separate call, when requested.
We ended up not using Subsquid and built our own custom indexing pipeline using Temporal for cloud-level process durability on our own hardware.
Wallet Connect was used to support multiple wallets.
The new setup allows infinite resolution times, should an error occur during the ETL process. Of course, this does cause the indexer to fall behind, but there is no deadline by which data will start getting lost.
Cost-efficiency: the new setup requires one 48-core Hetzner server to house the database and workflow/activity programs ($200 per month), one vercel app ($30), and one S3-compatible data bucket. It also requires Temporal cloud services (we don’t run our own temporal system) which is a flat $50 (pro-rata, as i also run other services on Temporal) + $10 variable.

Features not planned and delivered

Deep State Substrate Indexer. We have built an indexing pipeline that goes well beyond indexing what is readily available, i.e. extrinsic, event, and block data, to indexing the values returned by pallet methods, for any valid arguments. The indexer is built around substrate and the Polkadot.js library. In a nutshell, it works by reading the chain metadata, deconstructing every method input and output values to basic types, creating new basic and complicated inputs, and feeding them back into the methods, and repeating the process (outputs create new inputs).
Chat interface for the staking data and deep state indexer. This is a ChatGPT assistant based interface that uses ~35 methods to query the database. I plan to add more methods in the future that will provide some interesting graph-based abilities to the bot.

Features planned and not delivered yet

Multiple collator staking functionality. This includes the backend logic for recommending staking allocations, and the evm calls on the frontend.
Full integration with insurance contracts. The contract code is there but i have to connect it to the frontend.
Push updates of dashboard information via websockets. This is supported out of the box by surrealdb but the functionality is still a bit buggy. Waiting for it to mature. Then, it should be as simple as rewriting the current db queries to include a LIVE keyword and opening surrealdb’s websocket port.

Features planned but will not be delivered

We will not use a two-layer indexing architecture due to the high computing cost required by the deep state indexer. Instead, we will backup the data extracted files (large jsons, initially cached in s3) and take snapshots of rocksdb.

Next Steps

Production Indexing

Catching up to the chain from block 0 will be a challenge. Using a single 48-core machine would take around 7 months to get to block 6M, and by that time the chain would be well ahead especially given the 6-second block periods. Fortunately, the process can be parallelized across multiple servers. I would like to wait for surrealdb 2 to hit the production version before starting syncing from block 1. We are currently using surrealdb 1.5

Finishing Up Pending Functionality

I will finish up all remaining tasks, but I can’t work full-time on this anymore. Indexing will take some time anyways, so I can probably finish everything over part-time weekends.

M2 Payment

We are way past the second milestone so I would like to submit a vote to the committee. If it’s ok with you, I would like to add $2K which is the minimum that I will have to pay for deep-state indexing Moonbeam form block 1. This was not part of the original proposal (neither was the deep state indexer) so I am ok with covering the cost if I have to.

Questions for the committee

Future running costs I can run the staking indexer and the apps (for both stakeglmr and stakemovr) for less than 100 EUR per month on infra costs. I think the treasury can cover this once every year or two, no problem. However, running the deep state indexer requires renting a powerful machine from Hetzner that is ~220 EUR per month for each chain. What are your thoughts on this? Should we just do one chain and see how it goes? If so, which one?