[Proposal: MB7/MR5] Web3Go Data Analytics service for Moonbeam and Moonriver of Q2 2023

[hao_ding]

Abstract - This proposal aims to continue the data analytics work for Moonbeam and Moonriver by Web3Go stream data processing system. It is important to note that Web3Go has a strong capability for real-time data analytics and has built several critical dashboards for the Moonbeam community. To facilitate the work more smoothly in the next couple of months, we have made this proposal to cover the server and manpower costs.

Motivation - Moonbeam & Moonriver is one of the leading parachains in the Polkadot ecosystem and also fully supports EVM compatibility.
The Moonbeam & Moonriver ecosystem has lacked the infrastructure of a data analytics platform, which has made it difficult for users to conveniently and quickly use and view data from the view of the big picture, and also from the perspective of each project.
Web3Go has developed a stream-based data analytics platform, which has onboarded all on-chain data from Moonbeam and Moonriver. The proven work has been highly welcomed by the community and Moonbeam&Moonriver ecosystems:

1. Overview dashboard for Moonbeam Network: web3go
2. Twitts from Moonbeam official:
   a. https://twitter.com/MoonbeamNetwork/status/1624464396826058752
   b. https://twitter.com/MoonbeamNetwork/status/1613621351427805194

Project Overview and Team Experience - Web3Go is a data platform created to provide stream analytics services to the growing Moonbeam community. Our goal is to offer a wide range of data stream infrastructure services that benefit blockchain developers worldwide. We understand that the blockchain industry is constantly evolving and that developers face new challenges daily. Therefore, we are committed to providing the best tools and insights to help developers and community members in Moonbeam.

Furthermore, we continuously explore new ways to improve our platform and expand our offerings to remain at the forefront of this exciting and rapidly growing industry. Here lists the finished work from Web3Go team which has provided lots of insights to the ecosystem:

Moonbeam Public Dashboard: web3go

Moonbeam Staking/Unstaking Website: web3go

Moonriver Staking/Unstaking Website: web3go

Our team has been developing data tools for Polkadot eco for more than 2 years now. Recently we expanded the team size of from 8 to 15 to increase the development velocity and quality of our products.

Rationale -Web3Go has become the go-to choice for most projects and users in the Polkadot and Moonbeam ecosystems when it comes to Moonbeam data review and usage. However, due to development time and cost constraints, it is currently limited in its ability to provide customized data services for Moonbeam ecological project parties and users. It is believed that Web3Go would provide a complete data service for the Moonbeam ecosystem, capable of satisfying the transparency and insight needs of Moonbeam users under the support of the treasury.

Key Terms (optional) - N/A.

Overall Cost - the total cost of the proposal is listed as follows:
Server Cost/ Month: 2000 USD equivalent GLMR or MOVR/ Month
Engineer Cost/ Month: 8,000 USD equivalent GLMR or MOVR/ Month (30 USD/ Hour, 1.5 HC / Month )
Total Cost for Q2 2023: 30,000 USD equivalent GLMR or MOVR for 3 Months (April, May, and June 2023)

Use of Treasury Funds - detailed explanation of the use of Treasury Funds including deliverables/ milestones and timeline.

1. Deliver the overview dashboard for Moonriver Network.
2. Deliver the inflation dashboard for Moonbeam and Moonriver.
3. Deliver the dApp activity dashboard for Moonbeam and Moonriver.

Specifications - The following tech stacks will be used:

1. Archive Node: archive node is a node that has full log history for Moonbeam and Moonriver, which is very useful since all data are needed during deep analytics.
2. Indexer: The indexer is used to index each block from the archive node. this is a Web3Go self-developed tool that is compatible with Substrate and EVM.
3. Database maintenance: where the data is persisted.
4. Streaming-based data processing framework: streaming technology is used here to handle the business logic and make the data processing more real-time and efficient.
5. Data visualization framework: transfer the data to visualize patterns to make it easily understandable.

Steps to Implement -

1. Archive Node Set up: This is a one-time setup work. Normally it needs 0.2 HC for regular maintenance.
2. Data Indexing: This is to index the substrate and EVM data from the node. This is a one-time setup work. Normally it needs 0.2 HC for regular maintenance.
3. Database updating: Update the structure of the databases, adding a new table if needed. it needs 0.2 HC for development.
4. Data processing: Process the data according to the business requirement. It needs 0.6 HC for development.
5. Data visualization: Visilizing the result. It needs 0.3HC for development.

Link to on-chain proposal:

Web3Go Data Analytics service for Moonbeam and Moonriver of Q2 2023

Web3Go Data Analytics service for Moonbeam and Moonriver of Q2 2023

Calculating the amount of GLMR and MOVR: MOVR and GLMR 30 days price - Google Sheets

[turrizt]

Hey, Hao! Thank you for your proposal and for your analytical service!

Delegators frequently take advantage of the ability to simulate rewards, which provides a good estimate of what rewards to expect. Additionally, I myself frequently utilize the “History of Rewards and Actions” feature, which is an incredibly valuable tool that often provides useful information.

I have a few questions regarding your proposal and service as well:

1. Could you provide a more detailed description of the engineers work plan? specifically, I would like to see a breakdown of the time required for each task, etc. In other words, i would like to see in detail how many hours an engineer will need to complete these milestones.

2. As far as I can see, the Moonbeam Public Dashboard already contains this data:

• Deliver the inflation dashboard
  

• Deliver the dApp activity dashboard
  

Could you explain what exactly you will be working on for Moonbeam?

3. Based on my experience using web3go, the website often works quite slowly and some data loads quite slowly, which actually negatively affects the user experience. here are some screenshots of what I mean:
   

   
   
   
   

4. Regarding staking data, there are often differences between your dashboard and other dashboards, for example, total bonded & APR often do not match with stakeglmr/movr & subscan.

here is an explanation from one of the devs of stakeglmr / stakemovr
web3go calculates APR based on total staked, not "total staked after undelegations" which is not accurate, although it won't make a huge difference most of the time. Also, web3go uses a constant in their formula, which is not constant over time.

4.1 Rewards history does not always work correctly and sometimes certain addresses are not displayed, although these addresses are definitely associated with staking.

5. I think that all data related to the Moonbeam / Moonriver chain should definitely be located in one place so that it is very easy to find. for example, on the screenshot, some data can be found in Insight, while other in Projects. I believe that this only creates confusion.

[hao_ding]

hi Turrizt,

Thanks for your instant reply and really appreciated your research and attention to the product of Web3Go.

Here are our answers to your questions, feel free to discuss:

I have a few questions regarding your proposal and service as well:

1. Could you provide a more detailed description of the engineer’s work plan? specifically, I would like to see a breakdown of the time required for each task, etc. In other words, I would like to see in detail how many hours an engineer will need to complete these milestones.

Please check this table:

2. Could you explain what exactly you will be working on for Moonbeam?

Inflation : Inflation logic is complex. Currently, the data is not 100% percent correct. The data from the Web3Go team still has around a %1 delta with the data from the Moonbeam team, so we are working together to fix this issue. The reason why calculating inflation is very difficult:

1. Inflation has to consider the following factors:

• burnt data (the amount of data from transaction fees) - calculation based on Treasury.Deposit event
  • treasury Rejected: not include in burnt data
  • ONLY on Moonriver (pre800 burnt data) calculation based on weight
• net inflation (on-chain total GLMR/MOVR issued - can considered as supply)
• collator minting (an eip-1559 mechanism reward only happens on certain runtime)
• inflation to staking (reward for collators and nominators)
• inflation to orbiters (post2000 runtime the orbiters reward is calculate separately from staking

1. Due to the runtime upgrade, the logic on parachain changes from time to time. So there are lots of special logic to handle it. Web3Go team and Moonbeam team are checking the data almost block by block.
2. We’re processing inflation block by block, we’re the only team can offer real-time data of inflation breakdown

dApp Activity : tracking and calculating the dApp activity is continuous work, it requires ongoing effort and attention to ensure that every dApp on Moonbeam and Moonriver has to be tracked and processed in a required time period( 1 hour, 24 hours, 1 day, etc.) and meets the desired outcomes.

With the development of the Moonbeam and Moonriver ecosystem, the old dApps might be upgraded and the new dApps are deployed. the data analytics has to make everything up to date and deliver the correct result. To make it clear, I will explain the workflow on how to track dApp activity:

1. Moonbeam team works with the Etherscan team and gets the smart contract addresses that are mapped with each project.
2. Step 1 just started last month, which means only around 30 dApps have been onboarded on the data analytics platform. The dApps will be onboarded one by one, which needs collaboration among Etherscan, the Moonbeam team, and Proejcts.
3. Once the Web3Go team got the mapping, the smart contracts will be processed by our engineers to analyze the business logic and extract related information. Finally, the result will be visualized.

You can see that this job is repeatedly carried out, is very complicated, and needs close cooperation with other teams.

3. Based on my experience using web3go, the website often works quite slowly and some data loads quite slowly, which actually negatively affects the user experience. here are some screenshots of what I mean:

The good news is that we are upgrading our website, so in the next couple of months, the community will see a new website with all Moonbeam data onboarded!

4. Regarding staking data, there are often differences between your dashboard and other dashboards, for example, total bonded & APR often do not match with stakeglmr/movr & subscan.

Answer: for the early version of APR calculation, we use the dynamic rewards and staked token amount,

like the formula

APR formula= ( reward / rounds / stake ) * roundPerYear * 100

after discussed with Moonbeam parachain team , a new formula is provided as :

0.00001388888888888889 * <total_supply>  * <avg_blocks_per_round> / <collator_counted_stake>

it’s different with other formulas used by stakeglmr/stakemovr
4.1 Rewards history does not always work correctly and sometimes certain addresses are not displayed, although these addresses are definitely associated with staking.

Answer: The bug has been fixed already.

5. I think that all data related to the Moonbeam / Moonriver chain should definitely be located in one place so that it is very easy to find. for example, on the screenshot, some data can be found in Insight, while other in Projects. I believe that this only creates confusion.

A uniform portal for data charts and dashboards will be provided when the new website is delivered.

[turrizt]

Thank you so much for your detailed response, Hao. your efforts are greatly appreciated! I am definitely looking forward to seeing and testing your new website and various analytical data, wish you the best of luck with your proposal and ongoing work!

[GAMA]

Good luck you are not asking for that much!

[dev0_sik]

Hey @hao_ding - thanks a lot for answering the questions @turrizt brought up.
Some were overlaps of what I’d have asked too

Really appreciate the in-depth answers and future implementation details.

I’ll be the accountable Treasury Council Member for this proposal so if anything comes up feel free to always reach out.
Also hereby stating my support for the proposal itself

[mtca]

I am supportive of the proposal as well

[jose.crypto]

It’s good to know that a new site is on the way, but I would like to clarify, this payment is for maintenance and delivery of new dashboards, right?

So after completing the dashboards with this payment, the cost associated with maintenance would be around $2000 per month? have I understood correctly?

I am in favor of this proposal since it is a site that I find very useful in my day to day as a moderator and it helps me to clear doubts from the members of the community, so it shows the good that it brings to the community,

I would just like to clarify those points, before continuing, thanks

[hao_ding]

hi jrafaelangarita,

for this question:

So after completing the dashboards with this payment, the cost associated with maintenance would be around $2000 per month? have I understood correctly?

The $2000 is for the hardware cost only.
Normally it needs 0.2 Engineer/Month to take care of the server and be on standby for replying to any question from the Moonbeam team and community. (actually, it happens weekly since the Web3Go team has a regular meeting with the Moonbeam weekly )

If there is no new data requirement from the Moonbeam team and community, yes, the only cost would be above.

But if there are any new requirements or improvements, a new proposal will be raised from the Web3Go team.

Hope this answers your question!

Happy Weekend!

[dev0_sik]

Thank you very much for the explanation and the outlook.

You got the majority of the councilors in favor of your proposal and could turn it into an on-chain proposal.

We would kindly ask you to split it into two proposals, one for moonriver worth 20% of the total requested amount in MOVR and one on moonbeam totalling 80% of the total in GLMR.
Please use a 30-day-average price.

If you have any questions feel free to get in touch.
TG: @dev0_sik

[jose.crypto]

Ok sir understand, thanks u very much

[hao_ding]

hi Sik,

Please check the updated proposal. I have embedded the on-chain proposals and methodology for calculating the amount of GLMR and MOVR.

Thanks for your help!

[simplysimplebrightly]

I have used web3go dashboard and definitely it has some good info.
All my concerns were pointed out already.

i would like to know was web3go a grant winner in subquery for devloping this dashboard on moonbeam?

[hao_ding]

hi, thanks for your reply!

No, we have never received any grant from Subquery.

[simplysimplebrightly]

Ok, thanks for the reply