Kusama/AssetHub RT 1001000 XCM Failed Transactions

Thanks to Ruslan from the Nova Wallet team for the help throughout the process

Introduction

Kusama and AssetHub-Kusama (AH-K) Runtime 1001000 introduced a new XCM-delivery fee mechanism that caught all Parachain teams (and most ecosystem teams) off-guard.

In this post, I’ll try to explain the problem, the solution and what I’ll be proposing the MBF to help users whose funds are locked in Moonriver-owned accounts.

The Problem

To summarize what the delivery fee means. Now, when someone sends an XCM (for example, a token transfer) from Kusama or AH-K to Moonriver, the source chain charges an additional fee to the transaction fee for sending that XCM to the destination chain. This fee is obscured to the paymentInfo endpoint in Polkadot.js SDK, which is the only way developers have to know the cost of a specific transaction in Kusama or AH-K.

So let’s go through a practical example. Alice wants to send 1 KSM from Kusama or 1 RMRK from AH-K to Bob in Moonriver. For practical reasons, let’s say the transaction fee in both chains I is 0.001 KSM. In addition, let’s say the new XCM delivery fee is 0.01 KSM. Consequently, Alice must hold at least 1.011 KSM in Kusama or 1 RMRK and 0.011 KSM in AH-K. Note that I’m not considering the existential deposit.

What happens if Alice has enough KSM to cover the transaction fee but not the XCM delivery fee? Well, that is where the problem lies! The transaction was actually executed in the origin chain, meaning that assets like KSM in Kusama or RMRK in AH-K went to a Moonriver-controlled account in each chain, but the XCM failed to be delivered. Consequently, users would not receive their transfer in Moonriver.

If the XCM can’t be delivered because of lack of funds, the call is not reverted.

Parachain to Parachain Transfers

Users also reported that their funds were stuck when doing KSM transfers from one Parachain to another Parachain. The issue here is similar to what was described above but slightly different.

Let’s say Alice in Bifrost-Kusama wants to transfer 1 KSM to Bob in Moonriver. What happens underneath is an XCM message that goes from Bifrost-Kusama to Kusama, which contains another XCM message that is then sent from Kusama to Moonriver. Do you see the problem?

The first XCM message executed in KSM was fine, as Bifrost-Kusama does not charge an XCM delivery fee. When that message is executed in Kusamam, the first part transfers the KSM from a Bifrost-owned account to the Moonriver-owned account (KSM that the user originally had in Bifrost Kusama).

The second part of that XCM actually sends another XCM message from Kusama to Moonriver to deliver the funds to Bob in Moonriver. But, who pays for the XCM-delivery fee there? Well, that is exactly the problem! There was no way to take the funds from the XCM message to pay the XCM delivery fee. Consequently, the XCM delivery failed.

The Solution

Understanding the problem is just one piece of the puzzle. Parity vaguely provided a methodology to calculate the XCM delivery fee generically. It was clear that for the problem to be solved, we needed to find a way to obtain the XCM delivery fee for each transfer and account for that at a dApp level.

Therefore, we set off to create some script that mocked an XCM message emulating the relevant token transfer, and calculating the fees. Moreover, instead of calculating an XCM delivery fee for each individual transfer, we calculated a single XCM delivery fee based on the worst-case scenario. For Kusama based transfers, the highest XCM delivery fee is when a user sends the maximum amount possible (which is U128::MAX, or 340282366920938463463374607.431768211455 KSM). For AH-K-based transfers, the highest XCM delivery fee is when a user sends the maximum amount possible and the asset with the highest index (RMRK index is 8). In both instances, the maximum value is 340282366920938463463374607431768211455). You can find the scripts in the following URLs:

From the scripts:

The Max Kusama XCM Delivery Fee in KSM is 0.001366666299
The Max AH XCM Delivery Fee in KSM is 0.001051999947

Consequently, the Moonriver dApp now checks if the user also has that balance to cover XCM delivery fees (it was actually bumped by around 10% and rounded off, so it is a bit higher).

The fix was pushed Jan 30th, and XCM transfers were reactivated.

What’s Next?

After understanding the problem and releasing a fix in the Moonriver dApp to ensure no other users were impacted, the logical next step was to find the affected users and devise a plan to unlock their funds.

Unfortunately, we can only help recover funds for transfers from Kusama to Moonriver, AssetHub Kusama to Moonriver, and parachains to Moonriver. If you transferred KSM from Moonriver to another parachain, please contact the team of the destination chain.

To find all the affected users, we created a Subsquid indexer to track all the XCM messages that failed in Kusama and AH-K, but where the funds were actually transferred to the Moonriver owned account. You can find the code of the indexer in the following URLs:

The result of the indexers is the following.

If you don’t see your account in the following list. Please respond in this forum post or contact a community moderator in Telegram or Discord.

Summary of Failed KSM Transfers - Funds Locked in Moonriver Owned Account in Kusama

Note that some transfers include a from of a parachain and an account in that parachain.

[
  {
    "blockNumber": 21573927,
    "amount": "193552087806000",
    "from": "FcRfpcdqeViSYFHFRchnqpxktYMeNg8A9dxSE828QFWhhNZ",
    "id": "0021573927-f94ef-000047",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21576212,
    "amount": "21996150000",
    "from": "FcW8WdrjoPdVLUFPWEyrRZA24bvS7Cx8PKCS3iiX7E1haGJ",
    "id": "0021576212-d427e-000042",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21579329,
    "amount": "107217134466000",
    "from": "JFFw9RfZ6527u4mefZrE5hjzdB3NvV82Zfu44CJ3W3f6dnX",
    "id": "0021579329-e64cc-000021",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21581683,
    "amount": "9832709199",
    "from": "Parachain 2001 - dgRUb8Nh6qA4ZxTHrhwMSGXjTzHaMwaFc8HsMCzpS9jjBoW - https://bifrost-kusama.subscan.io/extrinsic/5890904-2?event=5890904-9&tab=xcm_transfer",
    "id": "0021581683-46921-000003",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21582439,
    "amount": "11593960025",
    "from": "Parachain 2001 - edjPQn1n8MDxGoFNtYEBvcxPRf6gE2S7BKgQxQu8gZknycs - https://bifrost-kusama.subscan.io/extrinsic/5891160-2?event=5891160-6&tab=xcm_transfer",
    "id": "0021582439-2a20a-000003",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21583225,
    "amount": "100290626722",
    "from": "Parachain 2000 - r4SoBpxKYpUpXGTGtNpTzcP2L8faPD8xQExZG6vj55Q94SZ - https://karura.subscan.io/extrinsic/6138236-3?event=6138236-34&tab=xcm_transfer",
    "id": "0021583225-81d24-000004",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21587554,
    "amount": "9832709199",
    "from": "Parachain 2001 - ffm75STKAACSqbnc6j16Nkm9h9i2VtjTvHXpkZy32QxsD2s - https://bifrost-kusama.subscan.io/extrinsic/5892906-2?event=5892906-5&tab=xcm_transfer",
    "id": "0021587554-d471c-000004",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21588460,
    "amount": "1517360819000",
    "from": "JFFw9RfZ6527u4mefZrE5hjzdB3NvV82Zfu44CJ3W3f6dnX",
    "id": "0021588460-202b6-000039",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21589429,
    "amount": "154997802516000",
    "from": "Dfi679UVN5LTQbPX5wT1brdrzAHthu4P3LgVSaC8dhvprk8",
    "id": "0021589429-2b447-000046",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21590503,
    "amount": "10174891000",
    "from": "FcW8WdrjoPdVLUFPWEyrRZA24bvS7Cx8PKCS3iiX7E1haGJ",
    "id": "0021590503-c3dde-000045",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  },
  {
    "blockNumber": 21594113,
    "amount": "22255803745000",
    "from": "J6k9HwMJS2bxs3MNSwMmfYGUoqmX12g61nD46v1kdtLWj3V",
    "id": "0021594113-ffa51-000042",
    "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
  }
]

Summary of Failed RMRK Transfers - Funds Locked in Moonriver Owned Account in AssetHub Kusama

Only users transferring RMRK tokens to Moonriver were affected.

[
  {
    "blockNumber": 6284005,
    "amount": "23630577378420",
    "from": "EJkrJnW23UyTEkcfXpyAJ2zEtZepBTGbPh7z6ED4utyn1hd",
    "id": "0006284005-f7c1c-000005",
    "to": "FBeL7EFFn7yHFs4uSf432RThNjEuieS8pQWcMdRFpXZPqkW",
    "assetID": 8
  },
  {
    "blockNumber": 6285438,
    "amount": "11000000000",
    "from": "GUeNBTpyBDCGAFCPJnQuLwZT5zQnspygvD8czVzSBAe8HXe",
    "id": "0006285438-b0a77-000010",
    "to": "FBeL7EFFn7yHFs4uSf432RThNjEuieS8pQWcMdRFpXZPqkW",
    "assetID": 8
  },
  {
    "blockNumber": 6285596,
    "amount": "3350000000000",
    "from": "EtmtG8CwZfF6pm8Wf4pUqvPHJQKvi76cUZDKnVktBGMVbky",
    "id": "0006285596-cca77-000004",
    "to": "FBeL7EFFn7yHFs4uSf432RThNjEuieS8pQWcMdRFpXZPqkW",
    "assetID": 8
  },
  {
    "blockNumber": 6289790,
    "amount": "30000000000",
    "from": "EtmtG8CwZfF6pm8Wf4pUqvPHJQKvi76cUZDKnVktBGMVbky",
    "id": "0006289790-79340-000005",
    "to": "FBeL7EFFn7yHFs4uSf432RThNjEuieS8pQWcMdRFpXZPqkW",
    "assetID": 8
  }
]

The Plan to Recover Funds

We’ll propose to the Moonbeam Foundation to submit a governance proposal that will send an XCM message from Moonriver to Kusama and AH-K to unlock the funds and transfer them back to the original owners.

For KSMs locked in Kusama from another parachain, we’ll send an XCM to Kusama that executes an XCM call to transfer these funds back to the original account in the source parachain.

We’ll test everything in Chopsticks before submitting the governance proposal.

5 Likes

Hi Moonriver community,

Sorry for the wait but the problem was not straightforward. I managed to produce a solution that will return all of the funds that Moonriver can return, to their rightful owners.

Checking Balances

First, we must ensure that Moonriver’s Sovereign account in the respective chains has enough balance to “cover” all returns. This is just a safety check. The team monitors to ensure there are never more xcAssets than actual assets in the sovereign account. For example, Moonriver Sovereign Account in Kusama should always have more KSM than the total amount of xcKSM in Moonriver.

So this is the summary of the balance check:

Moonriver and AH-K
Total amount of xcRMRK: 4564368.541151054
Transferable RMRK Balance in Moonriver Sovereign Account: 4569054.0821376656
Balance (RMRK - xcRMRK): 4685.54
Locked because of XCM: 2702.157737842
Difference (Balance - Locked): 1983.382262158 
Moonriver and Kusama
Total amount of xcKSM: 72848.45636195566
Transferable KSM Balance in Moonriver Sovereign Account: 73408.056054291761
Balance (KSM - xcKSM): 559.599692336101
Locked: 479.703910398145 
Difference (Balance - Locked): 79.895781937956 

As you can see, the Difference for both cases is positive so all good!

XCM Solution Explained

Long story short, it will be 3 separate XCM messages:

Moonriver - AssetHub Kusama

The first XCM message is to Kusama Asset-Hub, asking Moonriver Sovereign account there to return the RMRK tokens to the rightful owners.

Assets will be returned with a DepositAsset XCM instruction, specifying the specific amount and account as reported in this forum before. The XCM execution is paid with surplus RMRK tokens that the Moonriver Sovereign account holds in AH-K.

If you want to check the call out, check this URL.

Moonriver - Kusama

The second XCM message is to Kusama, asking Moonriver Sovereign account there to return the KSM tokens to the rightful owners in Kusama.

Assets will be returned with a DepositAsset XCM instruction, specifying the specific amount and account as reported in this forum before. The XCM execution is paid with surplus KSM tokens that the Moonriver Sovereign account holds in Kusama.

If you want to check the call out, check this URL.

Moonriver - Kusama Parachains

The third XCM message is a bit more complex. It is one XCM message that contains two subsequent XCM messages.

The main XCM message withdraws the amount to be returned to owners in the respective parachains (Bifrost and Karura). It executes a DepositReserveAsset to transfer them to their respective sovereign accounts in Kusama. It sends a subsequent XCM message to each parachain asking them to mint the same amount of tokens in their respective derivative-KSM asset.

At the same time, DepositReserveAsset accepts XCM instructions that are appended to the XCM sent to each parachain. In that XCM message, we include a bunch of DepositAsset XCM instructions to return the derivative-KSM asset in each parachain to the rightful owner.

The XCM execution is paid with surplus KSM tokens that the Moonriver Sovereign account holds in Kusama. A small amount of extra KSM is sent to each parachain to cover the XCM execution cost.

If you want to check the call out, check this URL.

XCM Solution - Getting to the Point

We’ve created a script to calculate all this call data and batch all 3 XCM messages in a single call.

If you want to check the batch call out, check this URL.

Using Chopsticks, we were able to test the calls out. Let’s go through the steps!

First we submitted a preimage with the following details:

Preimage Hash: 0x236cafa3b2615670f6e5692f2be0d2de26c3fe543a7e2545b4db797c2966b047
Length: 1049

Next, we proposed the preimage in the Root track. Using an internal tool, we fast-tracked this Root-track proposal. When it was executed, it sent three separate XCM messages.

Moonriver & AH-K XCM Execution

The first message lands in AH-K and, when executed, returns the respective RMRK tokens to the rightful owners.

(Assets are transferred by burning them in the origin account and minting them in the destination account)

Note that the last assets.Issued call is the one that returns the unspent fees after executing the XCM.

Moonriver & Kusama XCM Execution

The second message lands in Kusama and, when executed, returns the respective KSM tokens to the rightful owners in Kusama.

Note that the last balances.Deposit call is the one that returns the unspent fees after executing the XCM.

Moonriver & Kusama Parachains XCM Execution

The third message lands in Kusama and, when executed, it crafts two more XCMs that are then sent to each parachain where we traced failed XCM transfers. Note the XCM delivery fees being taken from Moonriver’s Sovereign account.

Note that the last balances.Deposit call is the one that returns the unspent fees after executing the XCM.

Next, we can check the XCM execution in Bifrost.

Last, we can check the XCM execution in Karura.

Conclusions

Well, that is it! It was definitely a busy week figuring everything out.

We’ll propose this fix to the OpenGov Technical Council so this is whitelisted and can be executed promptly.

4 Likes

great job as always @AlbertoV19 !!

2 Likes

Thanks for the in-depth technical explanation :slight_smile:

1 Like

It seems Nova Wallet had an issue that resulted in 4 new transactions failing to execute in Kusama when moving from one parachain to Moonriver.

From Bifrost:

    {
      "blockNumber": 21698172,
      "amount": "9761780645",
      "from": "dpCjcC2CT6niKf1F6BcEMoaXzxwxGxCXAP2rUeXyX8jfxYQ",
      "decoded": "0x5c7870c421bbeafddaa7bb8d0b66bdcd19a56c2d7f7ec2c1c8b0055f102cf201",
      "id": "0021698172-d4372-000003",
      "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
    },
    {
      "blockNumber": 21698545,
      "amount": "7050554022",
      "from": "dpCjcC2CT6niKf1F6BcEMoaXzxwxGxCXAP2rUeXyX8jfxYQ",
      "decoded": "0x5c7870c421bbeafddaa7bb8d0b66bdcd19a56c2d7f7ec2c1c8b0055f102cf201",
      "id": "0021698545-d495d-000003",
      "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
    },
    {
      "blockNumber": 21698621,
      "amount": "50869219509",
      "from": "dpCjcC2CT6niKf1F6BcEMoaXzxwxGxCXAP2rUeXyX8jfxYQ",
      "decoded": "0x5c7870c421bbeafddaa7bb8d0b66bdcd19a56c2d7f7ec2c1c8b0055f102cf201",
      "id": "0021698621-3767e-000004",
      "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
    }

From MangataX:

 {
      "blockNumber": 21747850,
      "amount": "5757290626722",
      "from": "5DJ8dYmMKFWqKcUwY26BEDUNM38xz8enm3qnmYcX4Qmd3ph4",
      "decoded": "0x3679d1d8e31d312a55f7ca994773b6a4fc7a92f07d898ae86bad4f3cab303c49",
      "id": "0021747850-adb4e-000003",
      "to": "F7fq1jSB3w59f8vMShxvP5eSu3wCJbL5Am5MQ6vP6VzYLWD"
    }

They total 5.82 KSM, so the balances are still correct.

The new encoded call data for Moonriver including these new txs can be found here.

New preimage data:

Hash: 0xa44c7b9c4bb9a01f435301ea7f735bbe21b13bf59d5c92766e064e48f0a3a481
Length: 1277

Following the process described before we can propose the proposal in Root and then execute it to test if the new transfers are included.

Bifrost:

Mangata:

This looks good IMO.

1 Like

Wow, great detective work @AlbertoV19! I will bring this to the Moonriver opengov technical committee to request that we fast track the return of user funds and report back here.

1 Like

Can you help with a stuck transaction?

hey, please check this out: [Proposal MR52] Return User Assets Stuck in Moonriver Sovereign Account

there’s an ongoing referendum, so it needs to be passed first. then, the funds will be transferred back to the affected users:
https://apps.moonbeam.network/moonriver/referendum/52

Hey @fasder11, to add to what @turrizt said, your transfer is in the original list I shared.

So the referendum when executed should return you your funds.

1 Like

hey, the proposal has passed and been enacted. It seems that the funds have been returned to you: Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

Happy to report that the proposal executed correctly and all original holders should have their funds back.

Holders in Kusama → Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

Holders in Bifrost (KSM) → Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

Holders in Karura (KSM) → Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

Holders in MangataX (KSM) → Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

And RMRK Holders in AssetHub-Kusama → Subscan | Aggregate Substrate ecological network high-precision Web3 explorer

2 Likes

thanks guys, everything is in place :ok_hand:

1 Like