Solving the XEN spamming issue

[MAR1]

Hello everyone,
I hope you’re doing well.

This post is dedicated to XEN’s presence on our blockchain. For those who don’t know, XEN is a multi-chain protocol and token, which is minted via the execution of transactions.
You can find out more at this link or read their whitepaper.

I don’t wish to judge the interest or lack of interest of such a dApp. As a builder, I simply want to point out that this token is detrimental to the smooth running of our chain. As a result, it’s hindering users of other dApps. In my case, when XEN spams the chain, The Great Escape players can no longer play. The same applies to PINK buyers, whose transactions are blocked.

When this happens, it’s extremely annoying as:

1. there’s nothing I can do
2. Moonbeam’s credibility is damaged

In concrete terms, this is what XEN spamming looks like:

The block is filled with “claimMintRewards” transactions of contract 0xb564A5767A00Ee9075cAC561c427643286F8F4E1

In the last 7 days, this represents 580 000 transactions, for example.

The problem is real for various actors:

• users, who can no longer use dApps,
• collators, who see their infrastructure over-used,
• builders, whose credibility is damaged.

The fundamental problem is that the dynamic fee model of our chain doesn’t adapt well to this spamming, which allows XEN to spam here and prevents the wallets of legit users from picking the right price for their transactions.

After consultation with various ecosystem actors, I’d like to propose several ways of dealing with this problem:

1) Ban XEN from our chain
Not sure of the feasibility, not very decentralized as they don’t do anything illegal.

2) Drastically increase fees for this specific contract.
Very heavy, and perhaps not foolproof as they could redeploy another ERC20. Would require a runtime upgrade.

3) Add an optional flag on the client, which would stop including transactions of a particular contract if it sent too many transactions in the same block.
The idea here would be to have a list of suspicious contracts, perhaps hosted on the foundation’s github. The flag would be activated when the threshold is exceeded.

4) Ask collators to stop including XEN transactions
Difficult to implement (as there is no consensus or incentives to do so).

Between the challenges of decentralization and the technical realities, it seems to me that option 3 is the most appropriate.

Still, I’m not a core dev, so I’d be very curious and interested to hear the community’s opinions on these various proposals.

I’ll conclude this post by saying that Moonbeam belongs to its users. Technology must follow usage. If we all agree that XEN is a problem, it’s up to us to come up with a solution.

[Jim_CertHum]

Thanks for posting about the issue @MAR1. I’ve also spent quite a bit of time thinking about this during the hours of troubleshooting I had to do on the UB RPC service during XEN spam storms. Eventually, my solution was lifting the artificial rate limits put in on the servers – which brings me to a point.

This is a throughput issue, and so when the throughput of the chain is increased, will that help to alleviate the problem (or just give more overhead for XEN to fill up?) In any case, I was always under the impression that gas was meant to be one counterbalance to chain spam, and that doesn’t seem to be acting as a deterrent so far.

Regarding the options you lay out –

1. banning is, as I’m sure you are aware, going to be controversial, and I don’t think would pass a governance vote based on strong ideological views of many in web3 – especially in this instance.

2. Increase fees I’ll get back to.

3. Rolling this out client flags will be challenging since node runners can just not include it, there could be a whack-a-mole scenario of new contracts, still may be considered a form of censorship (depending on what is deemed suspicious), requires someone or group to determine what is suspicious, and causes a lot of other questions (like, let’s say $STINK gets super popular and starts having the same effect as XEN on the chain. What makes one tx more deserving of blockspace than another?)

4. I think this will eventually come in the form of a module or similar implementation to 3, but will be more to do with government mandates (think tornado cash and banned address lists) and still may be a few more years away. Therefore, at least having a module like this ready may be useful, although still controversial to implement it for this purpose. Collators are definitely incentivized to help maintain a smooth running chain since collator rewards are distributed in GLMR.

Back to 2, and to reiterate, I think this is really a throughput and gas problem. Until the throughput is large enough to let XEN go wild and no one cares, if there was a way to fine-tune gas to make it unattractive to XEN while not penalizing other projects looking to grow with Moonbeam, that would be an ideal solution to me (of course, not wanting to create a situation like gas on Ethereum).

One alternative may be to artificially inflate the gas prices, but also have the Treasury subsidize and/or rebate gas fees (as a temporary solution which would also need Treasury approval), until the throughput increases.

So instead of a stick to XEN, a carrot to teams that are pushing “good” traffic through the network, that would also be a way to get them to register their contracts.

My worry with this solution is that it would potentially penalize projects that are operating in stealth, and/or users that don’t really pay that close attention. It would also create a bit more work all around. But it avoids any type of censorship debates, it would hopefully have the effect of dissuading XEN, and maybe build some community good will.

[turrizt]

hey Marin, thanks for bringing up this issue on the forum, it’s much appreciated

It seems like MBIP5 needs adjustments because it actually doesn’t solve the issue we’ve had for over a year now

as for your proposed ways:

1. I don’t think that banning XEN is actually a good thing. first of all, these are still txs that burn fees and interact with the chain. If something big comes to Moonbeam and starts spamming organically, how will we deal with it? I believe we just need to come up with an actually good long-term solution which should help fix the issue so such a thing won’t happen again. banning something contradicts the principles of decentralization and it may also set a precedent that could deter future devs

2. XEN could redeploy their contracts, making this approach only a temp fix

I think we can come up with something like dynamic throttling for contracts that spam the network. when certain thresholds are exceeded, the protocol automatically limits the execution rate of these contracts

also, I see that when XEN spam attacks arise, wallets can’t automatically adjust the gas based on the demand. It would be a good thing if wallets could automatically determine what gas prices need to be set so that they will be higher than XEN txs, ensuring the txs won’t get stuck in the mempool

It seems that Moonbeam contributors are working on short, medium, and long-term solutions rn, so I hope to hear what ideas they have

hopefully, async backing and increasing the gas limit to 60M, as discussed in Increase block gas limit by ahmadkaouk · Pull Request #2735 · moonbeam-foundation/moonbeam · GitHub, will also help in solving the issue

[Trilemma]

Thank you for starting this important discussion. As a key contributor to Moonwell, our users have been heavily impacted by this spam. Moonbeam is also not unique in this, we just have less blockspace than other networks, and our fee estimation in the node seems to be somewhat broken, so it makes for a poor user experience.

Please see this thread for an example of how it is impacting Base and other L2s: https://x.com/notnotstorm/status/1778803270750023813

The challenge, IMHO, boils down to three things:

1. Moonbeam and Ethereum networks are decentralized and permissionless. It would violate the spirit of censorship resistance if we started making value judgements about which applications are worth allowing or blocking on the network.
2. During periods of network congestion caused by XEN spam, the Moonbeam node is not providing accurate gas estimates to clients, so their transactions are underpriced, which results in non-XEN user transactions ending up “stuck” in the mempool for hours and needing to be replaced with a higher priced tx to confirm.
3. Ultimately, the long term issue is that state bloat occurs, meaning all Moonbeam nodes need to store terabytes of these spam transactions forever, although the XEN spammers only have to pay once.

As I think about the problem space, here’s how I think about this:

• Problem 1 is not solvable unless we want to move away from permissionless networks and start gatekeeping which apps can launch on Moonbeam. I think this would be a bad thing, and would put our community in a very risky situation where we are now deciding the merits of each app and making value judgements.
• Problem 2 seems like an engineering problem that can and should be solved as soon as possible. Even if people had to pay higher fees, the experience would be much better than getting your transaction “stuck” because it is underpriced.
• Problem 3 is where, IMHO, we should be focused, because it seems to me that the long-term solution is to charge state rent somehow.

To solve the long term problem, I do believe all these Ethereum networks, Moonbeam and L2s like Base, need to solve for state rent, hopefully without impacting legitimate apps like DEXes, Lending markets, Games, and NFTs, etc.

One idea I had is to evict storage for contracts that haven’t been touched in a year or so. That way if people really want to keep a low activity app around, they could interact with it at least once a year. I’m not sure if this would be viable, but it might really help to eliminate the long term state bloat.

In the short term, I think we just need to fix the gas estimation in the node. It’s clearly broken right now, although I’m not sure exactly how.

[Trilemma]

@Trilemma

One idea I had is to evict storage for contracts that haven’t been touched in a year or so. That way if people really want to keep a low activity app around, they could interact with it at least once a year. I’m not sure if this would be viable, but it might really help to eliminate the long term state bloat.

We could also dramatically increase the cost of opcodes that create new contracts. If it cost 10 GLMR to create a contract, I wouldn’t mind, since I only deploy new contracts once every few months or so… I think this was actually proposed by some of the Moonbeam engineers a few months back when this problem first surfaced.

Would anyone else’s legitimate app be broken if it cost 10 GLMR to deploy a contract?

[beamswapio]

XEN has made our life at Beamex unbearable. The costs of running the product when XEN starts spamming kills all the fees and potential revenue sharing for Products and Moonbeam users.

The issue is complex and hard to fix, chain simply doesn’t handle the spam - and we should look into fixing this somehow.

[micheleicebergnodes]

@Jim_CertHum

One alternative may be to artificially inflate the gas prices, but also have the Treasury subsidize and/or rebate gas fees (as a temporary solution which would also need Treasury approval), until the throughput increases.

So instead of a stick to XEN, a carrot to teams that are pushing “good” traffic through the network, that would also be a way to get them to register their contracts.

I really like this idea/approach Jim!!!
@turrizt I did not see your feedback specifically on this, I am curious!!!

[turrizt]

tbh, it seems the main issue is that txs are stuck in the mempool. when users submit txs, they often get stuck and aren’t sure how to proceed. what we really need, IMO, is for wallets to more accurately determine the appropriate gas fees based on current network demand. when I manually adjusted the gas settings above the market, my txs went through without issues. so, indeed, subsidizing txs during periods of high demand could be beneficial, but I’m not sure it’s a complete solution

I would love to hear @aaron.mbf thoughts. maybe he can share some insights on what ideas the core devs have in mind

[micheleicebergnodes]

Thanks for your valuable feedback, as usual!!!
I share your curiosity, quite confident our Dev Team is already coming up to an effective workaround!!!

[aaron.mbf]

Quick update - it seems like there’s been a break in terms of the XEN activity for the past while but this still remains a concern.

Putting XEN crypto aside, it would seem that there’s an issue with the core protocol under certain high load conditions.

Essentially, the mempool management needs to be made more efficient. When there is a sudden surge of transactions submitted, the collators are having difficulty getting enough transactions into a block to reach the target fullness threshold of 50%. As a result, the dynamic fee mechanism never kicks in.

The core devs are looking at possible workarounds but this would come in a future runtime upgrade.

One possible workaround would be to reduce the target fullness down from 50%. Note that it was raised from 25% to 50% in RT2801 to ease gas fees but there could be a sweet spot somewhere in between.

But the core devs are still doing analysis to look at other options and the Foundation will post back here with more info as it’s available.

There are of course other ideas in here also worth considering and I will ask for a review of the conversation in here to assess the feasibility of them. Appreciate the community’s concern and engagement to find a long term solution here…

[stakebaby]

@MAR1

0xb564A5767A00Ee9075cAC561c427643286F8F4E1

Just a quick question; I want to exclude any transaction that has to do with the XEN contract from getting indexed in the new stakeglmr platform. Are there any other related address I should be aware of?