UnitedBloc RPC service is architected to minimize failure and maximize availability. UB utilizes global server load balancing via GeoDNS services. Our gslb service monitors availability of any single server and if a server fails users are redirected to other nearby servers. We use “round robin” load balancing within Europe (4 resources) and North America (2 resources). APAC has 2 resources but in a primary/secondary setup instead of round robin. SA, with the lowest volume of usage, has a single resource, but fails over to the closest NA resource which adds only an additional 30ms of latency. Because we use multiple bare metal servers across multiple data centers across multiple providers with 24x7 monitoring and automated active response, the odds of us experiencing a significant failure event is very low.
Hey sir Can tell us on both cases, but is focused on a technical issue
Did my reply address your question adequately?
Thanks.
Yes sir thanks for your answer
Hey @jose.crypto ! Please find the answers to your questions below:
1. For paid plans whats the recovery time for a lost node and is it automatic?
We have implemented a node health-check mechanism that continuously monitors each node to ensure it is in sync with the blockchain. If a node is not syncing quickly enough or falls behind the tip of the chain by a certain number of blocks, it is automatically removed. This applies to all nodes in our blockchain network.
2. WIll the public RPC network persist without interruption if a region/origin drops?
We have health checks in place for all the proxies we use. If a proxy is not responding, it is automatically removed from load balancing to maintain uninterrupted service.
Apologies for the delayed response here.
There is no disruption to users in either of those scenarios, evidenced by 100% uptime for our Moonbeam, Moonriver, and Moonbase Alpha endpoints over the past 90 days - status.onfinality.io
We are constantly monitoring every individual node for signs of poor performance such as unresponsiveness and slow block production time. If a node is unhealthy traffic is first automatically redirected to other nodes in that region.
If an entire cluster is unable to process requests they are automatically load balanced to the next closest region.
We monitor our setup 24/7 and maintain live snapshots around the world ready to stand up new, fully synced nodes within minutes to respond to incidences or spikes in requests.
Hope this helps.
Thank you to those that took the time to put together proposals.
From a monthly cost perspective to cover all 3 networks, the proposals are as follows:
- OnFinality: 13K USD/month
- BwareLabs: 7K USD/month
- UnitedBloc: 3.6 USD/month
- Dwellir: 3K USD/month
As everyone is no doubt aware, market conditions have considerably shrunk the treasury budget in dollar terms. While having many viable options in terms of RPC providers promotes a healthy and decentralized ecosystem, this must be balanced with sustainability until more favorable conditions return.
To be sure, all four service providers are capable of providing reliable RPC services in a professional manner. While it would be wonderful if the Treasury could support all these proposals, this is just not feasible in the current climate.
As described in the original post, proposals by BwareLabs and OnFinality were approved in Q2 and Q3 for both BwareLabs and OnFinality to cover RPC services provided in the prior quarters.
The table below shows the percentage of total treasury budget allocated to RPC services for Q2, Q3 and a projection for Q4 (assuming the status quo) based on a GLMR price of $0.18 and MOVR price of $4.00.
| % of Total GLMR Treasury Budget | % of Total MOVR Treasury Budget | |
|---|---|---|
| Q2 | 24% | 37% |
| Q3 | 40% | 61% |
| Q4** | 59% | 47% |
** If proposals were approved for OnFinality and BwareLabs for RPC services provided over Q3 based on their current proposals and OnFinality were to follow the 80/20 rule for Moonbeam/Moonriver costs.
The decrease in the MOVR percentage in Q4 is only because the model shifts OnFinality to the 80/20 rule (consider costs for all 3 networks as a whole and then allocate 80% of the cost to the Moonbeam Treasury and 20% to the Moonriver Treasury).
The percentages climb even further to 80% for Moonbeam and 65% for Moonriver if all 4 proposals were to be approved.
Moreover, there are many other infrastructure public goods serving the community that may need to be funded from treasury such as block explorers, tooling, wallets and so on.
Given all of this, it is the opinion of the Treasury Council that the best way to approach this might be to restrict spending on RPC services at this time to ~35% of the Treasury Budget for each network. In dollar terms, based on the price assumptions above, this would be ~ $34K USD per quarter or a little over $11K USD per month.
In the short term, it means that some service providers may need to find ways to reduce their costs, forgo some level of funding or a combination of both. It may also mean that only a few proposals may be funded.
Longer term, assuming market conditions improve as we move into a new cycle, it means the treasury will be able to support more market participants (although the 35% level may be revisited).
In order to make an informed decision for the disbursement of funding, many questions have already been asked and answered. However, the Treasury Council has some additional feedback for each team below.
OnFinality & BwareLabs
What can be done to adjust the proposal to work within this budget while leaving room for one or more other service providers?
UnitedBloc
It would be helpful to understand if UnitedBloc could add boot nodes for Moonbeam and Moonriver alongside the RPC services under their proposal.
Dwellir
It looks like all traffic is served out of a single region. Is this expected to be the case if funding were to be granted?
While Dwellir offers the lowest cost, it is currently handling only a small portion of the traffic. How would Dwellir draw a much larger proportion of the overall traffic to its endpoints?
And if successful, how well positioned are you to handle volumes in a different order of magnitude were they to be shifted your way?
Ben from Dwellir here. Thanks for the continued inquiry @aaron.mbf . It’s important work and I understand the considerations you are making.
In a general sense, we are a premium provider that are able to deliver cost savings due to the nature of our operations. We own and operate our own bare metal and have a unique software stack that enables us to deliver high quality, low-latency services at fair prices.
To address your questions individually.
Regional Support
As per our proposal, Dwellir is prepared to scale its operations by adding Moonbeam nodes across three key regions: the United States, Europe, and North Africa. This will enhance both service redundancy and user experience.
Attracting Users
We recognise the need to attract a larger share of overall traffic. To address this, Dwellir plans to roll out a beta platform in the fourth quarter specifically for developers and teams. This new platform will make it easier for users to integrate our high-performance endpoints into their projects.
Traffic Volume and Scalability
Currently, Dwellir handles nearly 30 billion monthly requests for our customers (https://stats.dwellir.app/). Our infrastructure is designed for elasticity and is capable of scaling to meet increased traffic demands from Moonbeam.
In conclusion, Dwellir is ready and capable of expanding its regional footprint, drawing in a larger user base, and effectively managing escalated traffic volumes, should funding be granted.
Thank you for the feedback. UnitedBloc is fully equipped and willing to add boot nodes for both Moonbeam and Moonriver alongside our RPC services as proposed. We remain committed to supporting the network’s growth and infrastructure needs.
OnFinality appreciates the situation and the need to make adjustments to work within the current budget. We can make the below changes to bring down our costs and maintain quality service while meeting the needs of the community:
1. We can reduce Moonbase Alpha nodes and downsize all clusters to 2 nodes instead of 3, the price for this would be $10,000 per month. However if the community is willing to pay 6 months up-front then we can apply a significant 25% discount to bring it down to $7,500 per month.
2. We may need to slightly reduce the public rate limits to keep stability as our main priority. Our public rate limits are currently 50/s with 100 peak burst, so we will monitor and consider reducing the peak to keep a consistent 50/s - a level we have tested with common good applications such as wallets and confirmed they will still function well
3. The above does not include bootnodes. We will let the community decide whether they want us to keep our bootnodes running which are currently:
1x Moonbeam Bootnode = $500/month
2x Moonriver Bootnodes = $1000/month ($500 each)
Let me know if there are any questions here.
Hey @aaron.mbf , sorry for the late reply!
Given that we recognize the current market conditions and hold a strong belief in the future of Moonbeam networks, we are prepared to take the burden until the market improves, while making room for another provider to support the Moonbeam ecosystem.
With this in mind, we will maintain our support for the Moonbeam, Moonriver, Moonbase alpha ecosystems while revising our funding request from $7000/month to $4500/month.
thank you very much for the answers to both of you
Regarding @Mihai_BwareLabs bware, will the public rate limit remain the same, or will they be changed? If so, could you indicate the change?