UnitedBloc RPC service is architected to minimize failure and maximize availability. UB utilizes global server load balancing via GeoDNS services. Our gslb service monitors availability of any single server and if a server fails users are redirected to other nearby servers. We use “round robin” load balancing within Europe (4 resources) and North America (2 resources). APAC has 2 resources but in a primary/secondary setup instead of round robin. SA, with the lowest volume of usage, has a single resource, but fails over to the closest NA resource which adds only an additional 30ms of latency. Because we use multiple bare metal servers across multiple data centers across multiple providers with 24x7 monitoring and automated active response, the odds of us experiencing a significant failure event is very low.
Hey sir Can tell us on both cases, but is focused on a technical issue
Did my reply address your question adequately?
Thanks.
Yes sir thanks for your answer
Hey @jose.crypto ! Please find the answers to your questions below:
1. For paid plans whats the recovery time for a lost node and is it automatic?
We have implemented a node health-check mechanism that continuously monitors each node to ensure it is in sync with the blockchain. If a node is not syncing quickly enough or falls behind the tip of the chain by a certain number of blocks, it is automatically removed. This applies to all nodes in our blockchain network.
2. WIll the public RPC network persist without interruption if a region/origin drops?
We have health checks in place for all the proxies we use. If a proxy is not responding, it is automatically removed from load balancing to maintain uninterrupted service.
Apologies for the delayed response here.
There is no disruption to users in either of those scenarios, evidenced by 100% uptime for our Moonbeam, Moonriver, and Moonbase Alpha endpoints over the past 90 days - status.onfinality.io
We are constantly monitoring every individual node for signs of poor performance such as unresponsiveness and slow block production time. If a node is unhealthy traffic is first automatically redirected to other nodes in that region.
If an entire cluster is unable to process requests they are automatically load balanced to the next closest region.
We monitor our setup 24/7 and maintain live snapshots around the world ready to stand up new, fully synced nodes within minutes to respond to incidences or spikes in requests.
Hope this helps.
Thank you to those that took the time to put together proposals.
From a monthly cost perspective to cover all 3 networks, the proposals are as follows:
- OnFinality: 13K USD/month
- BwareLabs: 7K USD/month
- UnitedBloc: 3.6 USD/month
- Dwellir: 3K USD/month
As everyone is no doubt aware, market conditions have considerably shrunk the treasury budget in dollar terms. While having many viable options in terms of RPC providers promotes a healthy and decentralized ecosystem, this must be balanced with sustainability until more favorable conditions return.
To be sure, all four service providers are capable of providing reliable RPC services in a professional manner. While it would be wonderful if the Treasury could support all these proposals, this is just not feasible in the current climate.
As described in the original post, proposals by BwareLabs and OnFinality were approved in Q2 and Q3 for both BwareLabs and OnFinality to cover RPC services provided in the prior quarters.
The table below shows the percentage of total treasury budget allocated to RPC services for Q2, Q3 and a projection for Q4 (assuming the status quo) based on a GLMR price of $0.18 and MOVR price of $4.00.
| % of Total GLMR Treasury Budget | % of Total MOVR Treasury Budget | |
|---|---|---|
| Q2 | 24% | 37% |
| Q3 | 40% | 61% |
| Q4** | 59% | 47% |
** If proposals were approved for OnFinality and BwareLabs for RPC services provided over Q3 based on their current proposals and OnFinality were to follow the 80/20 rule for Moonbeam/Moonriver costs.
The decrease in the MOVR percentage in Q4 is only because the model shifts OnFinality to the 80/20 rule (consider costs for all 3 networks as a whole and then allocate 80% of the cost to the Moonbeam Treasury and 20% to the Moonriver Treasury).
The percentages climb even further to 80% for Moonbeam and 65% for Moonriver if all 4 proposals were to be approved.
Moreover, there are many other infrastructure public goods serving the community that may need to be funded from treasury such as block explorers, tooling, wallets and so on.
Given all of this, it is the opinion of the Treasury Council that the best way to approach this might be to restrict spending on RPC services at this time to ~35% of the Treasury Budget for each network. In dollar terms, based on the price assumptions above, this would be ~ $34K USD per quarter or a little over $11K USD per month.
In the short term, it means that some service providers may need to find ways to reduce their costs, forgo some level of funding or a combination of both. It may also mean that only a few proposals may be funded.
Longer term, assuming market conditions improve as we move into a new cycle, it means the treasury will be able to support more market participants (although the 35% level may be revisited).
In order to make an informed decision for the disbursement of funding, many questions have already been asked and answered. However, the Treasury Council has some additional feedback for each team below.
OnFinality & BwareLabs
What can be done to adjust the proposal to work within this budget while leaving room for one or more other service providers?
UnitedBloc
It would be helpful to understand if UnitedBloc could add boot nodes for Moonbeam and Moonriver alongside the RPC services under their proposal.
Dwellir
It looks like all traffic is served out of a single region. Is this expected to be the case if funding were to be granted?
While Dwellir offers the lowest cost, it is currently handling only a small portion of the traffic. How would Dwellir draw a much larger proportion of the overall traffic to its endpoints?
And if successful, how well positioned are you to handle volumes in a different order of magnitude were they to be shifted your way?
@Marc_Blockshard @Mihai_BwareLabs @brittanyseales @gustavnipe if you can answe when have time
Ben from Dwellir here. Thanks for the continued inquiry @aaron.mbf . It’s important work and I understand the considerations you are making.
In a general sense, we are a premium provider that are able to deliver cost savings due to the nature of our operations. We own and operate our own bare metal and have a unique software stack that enables us to deliver high quality, low-latency services at fair prices.
To address your questions individually.
Regional Support
As per our proposal, Dwellir is prepared to scale its operations by adding Moonbeam nodes across three key regions: the United States, Europe, and North Africa. This will enhance both service redundancy and user experience.
Attracting Users
We recognise the need to attract a larger share of overall traffic. To address this, Dwellir plans to roll out a beta platform in the fourth quarter specifically for developers and teams. This new platform will make it easier for users to integrate our high-performance endpoints into their projects.
Traffic Volume and Scalability
Currently, Dwellir handles nearly 30 billion monthly requests for our customers (https://stats.dwellir.app/). Our infrastructure is designed for elasticity and is capable of scaling to meet increased traffic demands from Moonbeam.
In conclusion, Dwellir is ready and capable of expanding its regional footprint, drawing in a larger user base, and effectively managing escalated traffic volumes, should funding be granted.
Thank you for the feedback. UnitedBloc is fully equipped and willing to add boot nodes for both Moonbeam and Moonriver alongside our RPC services as proposed. We remain committed to supporting the network’s growth and infrastructure needs.
Hey @Mihai_BwareLabs @brittanyseales if you can answer the questions from aaron
OnFinality appreciates the situation and the need to make adjustments to work within the current budget. We can make the below changes to bring down our costs and maintain quality service while meeting the needs of the community:
1. We can reduce Moonbase Alpha nodes and downsize all clusters to 2 nodes instead of 3, the price for this would be $10,000 per month. However if the community is willing to pay 6 months up-front then we can apply a significant 25% discount to bring it down to $7,500 per month.
2. We may need to slightly reduce the public rate limits to keep stability as our main priority. Our public rate limits are currently 50/s with 100 peak burst, so we will monitor and consider reducing the peak to keep a consistent 50/s - a level we have tested with common good applications such as wallets and confirmed they will still function well
3. The above does not include bootnodes. We will let the community decide whether they want us to keep our bootnodes running which are currently:
1x Moonbeam Bootnode = $500/month
2x Moonriver Bootnodes = $1000/month ($500 each)
Let me know if there are any questions here.
Hey @aaron.mbf , sorry for the late reply!
Given that we recognize the current market conditions and hold a strong belief in the future of Moonbeam networks, we are prepared to take the burden until the market improves, while making room for another provider to support the Moonbeam ecosystem.
With this in mind, we will maintain our support for the Moonbeam, Moonriver, Moonbase alpha ecosystems while revising our funding request from $7000/month to $4500/month.
thank you very much for the answers to both of you
Regarding @Mihai_BwareLabs bware, will the public rate limit remain the same, or will they be changed? If so, could you indicate the change?
Hey @jose.crypto , the public rate limit will remain the same.
Thank you for your responses.
The Treasury Council has been discussing and deliberating on various options and scenarios. Given there are many factors to consider, it’s been difficult to reach a consensus. Moreover, one of the treasury council members is a member of UnitedBloc and has therefore recused themselves from the discussions. These are just a few of the reasons why this discussion has been dragging and the council apologizes for the delay.
Funding for Q3 Services Provided and Moving to a Pre-Paid Model
Given that we are now starting Q4, the Treasury Council proposes that service providers interested in treasury funding to pay for services provided in Q3 should create a postpaid (ie. paying for services already provided) style treasury proposal as has been the convention previously.
Then, going forward, the Council suggests moving to a pre-paid model where RPC service providers are paid in advance on a quarterly basis so that they know with certainty the services they provide will be funded.
At this point, the Treasury Council would then select RPC service providers and commit to funding (as treasury reserves allow) them for Q4 2023 and Q1 2024.
However, paying for both Q3 (post-paid) and Q4 (pre-paid) services in Q4 will put a large strain on the Q4 treasury budget. Thus, the upcoming Treasury Program renewal proposal will propose that the left over Q3 budget be carried forward to help cover these costs.
Even with a carry forward, the amount of funds available for services provided in Q3 is limited so service providers should take this and general market conditions into consideration when submitting their Q3 funding proposals.
Next steps
Any RPC service providers seeking Treasury funds to cover costs incurred for services provided in Q3 should create an independent Treasury Proposal forum post in the standard format formally requesting funding.
RPC Service Providers Selected for Q4 2023 and Q1 2023 Funding
Given market conditions, it is the Treasury Council’s recommendation to fund at most two RPC service providers and keep RPC spend to 35% of the overall treasury budget.
After extensive deliberations, the Treasury Council has selected the following service providers for this period:
- UnitedBloc
- Bware Labs
UnitedBloc
Several factors contributed to the selection of UnitedBloc, including:
- Price Point - UnitedBloc is providing a very high quality of service and has proposed a very reasonable monthly service fee given the traffic they currently handle.
- Decentralization - UnitedBloc is a decentralized service providing a public good - this is a model that should be encouraged.
- Community Engagement - UnitedBloc is composed of many long time community members. The fact that they were able to very quickly gain RPC market share is a testament to both their community engagement and the trust the community places in UnitedBloc
- Boot Nodes - UnitedBloc is willing to provide Boot Nodes at no extra cost
Bware Labs
Here are some of the reasons Bware Labs was selected:
- Proven and Professional team - Bware Labs was an early supporter of the Moonbeam ecosystem. Over this time, they have demonstrated that they are a solid and reliable team.
- Paid Option - The Treasury Council feels it’s important to support at least one service provider that provides a Paid Option for projects that have a need for dedicated endpoints.
- Decentralization - Bware’s Platform is such that independent operators are able to sign up and participate increasing the diversity of participants.
- Price - Bware Labs reduced their funding considerably to align with market conditions and the limited Treasury budget.
Next steps
UnitedBloc and BwareLabs should create an independent Treasury Proposal forum post in the standard format formally requesting funding for Q4 RPC services.
RPC Providers that were NOT Selected Q4 2023 and Q1 2024 Funding
OnFinality
OnFinality shares a lot of the same qualities as Bware:
- Early supporter of the Moonbeam Ecosystem
- Proven and Professional Team
- Has a Paid Option
- Has a large share of the ecosystem RPC traffic
However, the Bware Labs service is 40% less expensive.
The Treasury Council is appreciative of the high quality of service that has been provided to the Moonbeam ecosystem over the last several years and hopes that OnFinality can find a path forward to continue to provide services in some capacity throughout the bear market until more favorable conditions allow for a larger treasury budget.
OnFinality currently handles a large amount of the public RPC requests in the Moonbeam ecosystem and also manages boot nodes for both Moonbeam and Moonriver. Should OnFinality choose to reduce service levels for the Moonbeam ecosystem, some level of disruption may be experienced by both end users and developers.
In this case, the Treasury Council would invite OnFinality to propose a “transition plan” treasury request to allow for a smooth transition for end users, projects, etc that may be hard coded to OnFinality RPC endpoints. Perhaps this would be in the form of funding for services over a 1 month period. This would take the form of a typical treasury proposal where it is proposed in the forum for consideration by the community and the treasury council. Feedback would be provided in the form of comments and once support is achieved by at least 3 council members, it could proceed to an on-chain proposal.
Dwellir
Dwellir is a reputable team with a proven and professional team with a paid option. Moreover, Dwellir quoted the lowest costs of all 4 proposals.
However, Dwellir was a relatively late comer to the Moonbeam ecosystem which has led to them having a small share of overall public RPC traffic to this point. Any attempt to steer a large proportion of public RPC traffic away from two or more of the main RPC providers over to Dwellir would no doubt be even more disrupting.
That being said, the Treasury Council hopes that Dwellir can find a path forward to continue to provide services in some capacity. Moreover, as market conditions allow, the Council will look for future opportunities to collaborate.
We are absolutely thrilled and deeply grateful for the opportunity to provide RPC services to the Moonbeam Network 
We are excited to work together and create an independent Treasury Proposal forum post as outlined. Thank you for placing your trust in UnitedBloc.
Good morning,
Firstly, congratulations to UnitedBlock and Bware - great services. @Marc_Blockshard @Mihai_BwareLabs
Thank you to all the council members @jose.crypto @dev0_sik @mtca @aaron.mbf for spending time on this. We’ve certainly learnt something from this process that will enable us to further expand our offering.
We will continue providing public RPC services for the community regardless of the outcome as we believe that Moombeam is a key player in the ecosystem. If you need anything, you know where to find us.
Thanks once again!
Best,
Ben and the Dwellir team
We have posted our proposal for Q4 2023 as requested. The proposal can be found here.
I am happy to supply the Moonbeam team with the list of UnitedBloc bootnodes for all three networks:
#MOONBASE
"bootNodes": [
"/dns/eu-02.unitedbloc.com/tcp/35060/p2p/12D3KooWG8KzsySP5pEoG5oh1MzjYJhq7QZRbNHKZiGWF7FNQHuk",
"/dns/eu-03.unitedbloc.com/tcp/37060/p2p/12D3KooWGkQhFJZAVYhDi5bp3hUm3QouwGmYugvqNLuS77TKyNZd",
"/dns/eu-01.unitedbloc.com/tcp/35060/p2p/12D3KooWAu26xXQy9Q8P9rXim3yAGkAZ38BS3xrF2J7uZUbz1A8n",
"/dns/apac-01.unitedbloc.com/tcp/35060/p2p/12D3KooWH1zRsVBRtTNiEbKHSees6ESw7UPFJBws3StgXcqTUQDt",
"/dns/sa-01.unitedbloc.com/tcp/35060/p2p/12D3KooWKbo2qnyNdea2uR55z7dfg1BVDJw15Xr9xkUHd1csQnG2",
"/dns/na-01a.unitedbloc.com/tcp/30333/p2p/12D3KooWEe1hn1YMxqzhacCDYrQVrNHyP8MXZSSjHkTwnGmKXRhh"
],
#MOONRIVER
"bootNodes": [
"/dns/na-02.unitedbloc.com/tcp/6060/p2p/12D3KooWKPrWGGSTirVgEWvvu6t4L5VXUkE9FLi5xFoi2zgqziH4",
"/dns/eu-02.unitedbloc.com/tcp/36060/p2p/12D3KooWKB4rt5SWajY7MdVEhQqyx9NRj7LyUyo2fgSzSEuYTz8j",
"/dns/eu-03.unitedbloc.com/tcp/36060/p2p/12D3KooWPaLHEeVhicbyioBWj4t9RcJTB9dhHFdcoL3nQbg5piQA",
"/dns/eu-01.unitedbloc.com/tcp/36060/p2p/12D3KooWGbSCWxP9tVAx2eWi5UG2VsDFA1vzLAcKH8b8kyNS5Rde",
"/dns/apac-02.unitedbloc.com/tcp/36060/p2p/12D3KooWPvom1ZBWpXRAPf7S8zQXYX2iNNF6dSZcBxyfPY517JMd",
"/dns/apac-01.unitedbloc.com/tcp/36060/p2p/12D3KooWAosKitTohMgbKqTtcaUvyUHpP3vbnL7het3nFBSfheZ7",
"/dns/sa-01.unitedbloc.com/tcp/36060/p2p/12D3KooWBXSNJudphGqb4TKmgfruZkY79PsN9aGYFnLgYR4ou3SH",
"/dns/na-01.unitedbloc.com/tcp/36060/p2p/12D3KooWLVmzvBxKvJyg8u4fUuBgnSBJGS8wQFXKvyHUBzJ1pAuT"
],
#MOONBEAM
"bootNodes": [
"/dns/na-02.unitedbloc.com/tcp/7060/p2p/12D3KooWS9aDbVwmLHrWKjehiGGzDSsTUVYyjvwuguz8jyFVDG2q",
"/dns/eu-02.unitedbloc.com/tcp/37060/p2p/12D3KooWRwPNbahHG9pnPd5hSyHCMZo8aV5qaJwH2qCoxTbMA3KJ",
"/dns/eu-03.unitedbloc.com/tcp/37060/p2p/12D3KooWF43rrtXewLGW8X9Pp6nyV6ESAZRnNCHVkZy1kzG1MueB",
"/dns/eu-04.unitedbloc.com/tcp/37060/p2p/12D3KooWSVwNFLrR9Bc3st9yiyjUnEx8CKSavfh3hyPeVsSdTDSn",
"/dns/eu-01.unitedbloc.com/tcp/37060/p2p/12D3KooWJTFVibb5ADHY1iUhE3iqMYiaVeXhdBWFYXrkmkxVXrBD",
"/dns/apac-02.unitedbloc.com/tcp/37060/p2p/12D3KooWJsfWGQLajaXXoqYSpn1CZbXHs7RDWqbgvsBXPJrjVGjb",
"/dns/apac-01.unitedbloc.com/tcp/37060/p2p/12D3KooWCnXymVyowL5YymVk6RNzRzFAWUvurX7eTTYAFcCbs4nj",
"/dns/sa-01.unitedbloc.com/tcp/37060/p2p/12D3KooWC3FoL2bFJ79C5CeLDjLmL2yRyozPxSjTQbm9RY95KNsV",
"/dns/na-01.unitedbloc.com/tcp/37060/p2p/12D3KooWLEZwHD8tWvWiEqKcD976wxdE5JRUnq1JG9a6VJxHUjiS"
],