FRA Staking Issues (Update)
The engineering team released a new wallet update which appears to have introduced a problem where stake values fail to load. This appears to be purely cosmetic though inconvenient, and requiring that engineering continue working on the issue.
We'll post another update as progress continues to be made.
The engineering team released a new wallet update which appears to have introduced a problem where stake values fail to load. This appears to be purely cosmetic though inconvenient, and requiring that engineering continue working on the issue.
We'll post another update as progress continues to be made.
👍2
FRA Staking Issues (Update)
Engineers have identified issues with EVM operations. One of the symptoms of this issue is missing stake data in the Findora wallet. Validators have also shared information on block proposals which may be tied to the underlying issue. The team is using this information to narrow down the root cause and additional updates will be shared as progress is made.
Engineers have identified issues with EVM operations. One of the symptoms of this issue is missing stake data in the Findora wallet. Validators have also shared information on block proposals which may be tied to the underlying issue. The team is using this information to narrow down the root cause and additional updates will be shared as progress is made.
FRA Staking Wallet Display Issues (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update: Engineers have identified the issue is related to delays with a full node, causing various API calls to load slowly. One of the symptoms more easily noticed is stake values in the Findora wallet. The team is in the process of performing a version update to the full node in question in order to resolve the issue.
No action is needed by wallet users. We'll send another update when the update is completed.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update: Engineers have identified the issue is related to delays with a full node, causing various API calls to load slowly. One of the symptoms more easily noticed is stake values in the Findora wallet. The team is in the process of performing a version update to the full node in question in order to resolve the issue.
No action is needed by wallet users. We'll send another update when the update is completed.
👍3
FRA Wallet Display Issues (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update [Sept. 28 | 11PM PST]
Engineers have isolated the culprit to performance issues on a full node. To be more specific, the full node is showing that consensus and HTTP/API requests are attempting to acquire a lock on the database simultaneously, resulting in high resource utilization of the server.
With the triggering action identified, the team continues to work on resolving the problem. Once resolved, the wallet will resume displaying balance and stake information, and our tests on the new EVM staking portal can resume.
Thanks for being patient while the team works to resolve this issue. We should have another update by PST morning.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update [Sept. 28 | 11PM PST]
Engineers have isolated the culprit to performance issues on a full node. To be more specific, the full node is showing that consensus and HTTP/API requests are attempting to acquire a lock on the database simultaneously, resulting in high resource utilization of the server.
With the triggering action identified, the team continues to work on resolving the problem. Once resolved, the wallet will resume displaying balance and stake information, and our tests on the new EVM staking portal can resume.
Thanks for being patient while the team works to resolve this issue. We should have another update by PST morning.
❤🔥2👍2❤1
FRA Wallet Display Issues (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update [Oct. 3rd | 12:15 AM PST]
Our devops team have recently resolve a related issue with a graph node which led to slow performance on the new staking portal in testing. However, this does not resolve the issue where FRA stake, unbonding, and reward values display incorrectly.
The root cause of the issues impacting balances is related to the Findora wallet and block explorer. To be specific, those two products appear to be referencing the wrong API methods for retrieving this data. Because of this, the wallet and scanner are displaying incorrect values.
A solution is underway. Both the wallet and scanner are being updated to ensure the proper APIs are called, which should resolve the issue at hand.
Thank you again and we hope this update was helpful. We will send another update in the morning (PST).
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Update [Oct. 3rd | 12:15 AM PST]
Our devops team have recently resolve a related issue with a graph node which led to slow performance on the new staking portal in testing. However, this does not resolve the issue where FRA stake, unbonding, and reward values display incorrectly.
The root cause of the issues impacting balances is related to the Findora wallet and block explorer. To be specific, those two products appear to be referencing the wrong API methods for retrieving this data. Because of this, the wallet and scanner are displaying incorrect values.
A solution is underway. Both the wallet and scanner are being updated to ensure the proper APIs are called, which should resolve the issue at hand.
Thank you again and we hope this update was helpful. We will send another update in the morning (PST).
👀2👏1😁1
FRA Wallet Display Issues (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Latest Update [Oct. 3rd | 1:05 PM PST]
1— We have an engineer working on updating the Findora wallet and block scanner. This work should resolve the primary issue above re: the display of stake data.
2— The team also has an engineer who continues to work on the DB lock and performance impact to the full nodes. No new information is available since the last update, but efforts continue to resolve this problem.
3— The Rialto bridge is experiencing issues which should be resolved in the next 24 hours.
The underlying blockchain continues to run and the team is working around the clock resolving the issues above. Additional team members will be joining the effort in a few days when their national holiday comes to a close. Thank you for being patient while the team works to restore some of these services.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Latest Update [Oct. 3rd | 1:05 PM PST]
1— We have an engineer working on updating the Findora wallet and block scanner. This work should resolve the primary issue above re: the display of stake data.
2— The team also has an engineer who continues to work on the DB lock and performance impact to the full nodes. No new information is available since the last update, but efforts continue to resolve this problem.
3— The Rialto bridge is experiencing issues which should be resolved in the next 24 hours.
The underlying blockchain continues to run and the team is working around the clock resolving the issues above. Additional team members will be joining the effort in a few days when their national holiday comes to a close. Thank you for being patient while the team works to restore some of these services.
👍1
FRA Wallet Display Issues & Portal Blockers (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts. This creates a situation where wallet users have trouble interacting with Findora's staking, and leads to blockers for the launch of the EVM staking portal.
Latest Update [Oct. 4th | 3:45 PM PST]
++++++++++++++++++++++++++++
1— FINDORA WALLET:
Engineering is continuing to work on resolving several issues. On the wallet side, we are updating its API methods used to obtain accurate staking data. Engineers deployed a new version (v0.4.4), but the issue persisted. The team is making progress, and we'll provide more updates as things move forward.
2— STAKING PORTAL:
DevOps is working on resolving issues impacting the full nodes. A new RPC server was set up for interactions with the upcoming EVM staking portal. The DevOps team is reloading the database on this new RPC server to improve staking transaction speed on the EVM staking portal.
3— VALIDATORS:
The new validator jailing threshold has been lowered to more forgiving values. Stake rewards are being generated and accumulating for delegators, but these rewards are not displayed on the wallet until item 1 above is resolved. The engineering team is working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line.
++++++++++++++++++++++++++++
As mentioned in the last update, the underlying blockchain continues to run, and the team is working around the clock to resolve the issues above. Additional team members will join the effort in a few days when their national holiday ends.
Finally, and to share more details with the community, we have shared a list of issues being resolved and improvement efforts in progress. We’re working hard to resolve these launch blockers and will share updated versions of the list as progress is made. See image below.
Thank you for being patient while the team works to restore some of these services.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts. This creates a situation where wallet users have trouble interacting with Findora's staking, and leads to blockers for the launch of the EVM staking portal.
Latest Update [Oct. 4th | 3:45 PM PST]
++++++++++++++++++++++++++++
1— FINDORA WALLET:
Engineering is continuing to work on resolving several issues. On the wallet side, we are updating its API methods used to obtain accurate staking data. Engineers deployed a new version (v0.4.4), but the issue persisted. The team is making progress, and we'll provide more updates as things move forward.
2— STAKING PORTAL:
DevOps is working on resolving issues impacting the full nodes. A new RPC server was set up for interactions with the upcoming EVM staking portal. The DevOps team is reloading the database on this new RPC server to improve staking transaction speed on the EVM staking portal.
3— VALIDATORS:
The new validator jailing threshold has been lowered to more forgiving values. Stake rewards are being generated and accumulating for delegators, but these rewards are not displayed on the wallet until item 1 above is resolved. The engineering team is working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line.
++++++++++++++++++++++++++++
As mentioned in the last update, the underlying blockchain continues to run, and the team is working around the clock to resolve the issues above. Additional team members will join the effort in a few days when their national holiday ends.
Finally, and to share more details with the community, we have shared a list of issues being resolved and improvement efforts in progress. We’re working hard to resolve these launch blockers and will share updated versions of the list as progress is made. See image below.
Thank you for being patient while the team works to restore some of these services.
❤1
Latest Update [Oct. 6th | 5:35 PM PST]
The Engineering team is currently working on fixes to the wallet and has made significant progress on removing blockers to launching the EVM Staking portal.
Each of the issues we’re resolving have specific technical task owners assigned, and task owners are scheduled to meet as a group tonight for a working session. We plan to release an update on the solutions tracker and provide the community with more details once the meeting has concluded.
Rest assured that these are display issues and validator, delegator, and user funds and rewards are safe.
The Engineering team is currently working on fixes to the wallet and has made significant progress on removing blockers to launching the EVM Staking portal.
Each of the issues we’re resolving have specific technical task owners assigned, and task owners are scheduled to meet as a group tonight for a working session. We plan to release an update on the solutions tracker and provide the community with more details once the meeting has concluded.
Rest assured that these are display issues and validator, delegator, and user funds and rewards are safe.
Latest Update [Oct. 6th | 8:50 PM PST]
1— FINDORA WALLET:
Engineering has resolved the issue where balances were not being displayed. This fix is contained in the new v0.4.4 release available on https://findora.org/#wallet. However, a new cosmetic/display issue has been identified where recently unbonded FRA is not reflected in the total balance. Engineering is working on a solution and prioritizing this as an urgent fix.
2— EVM STAKING PORTAL:
The DevOps team made changes which significantly improved response time on the new EVM staking portal. The team is working on a Graph node rebuild. This rebuild will improve response time for other components of the new staking portal which continue to experience a sluggish response time.
3— VALIDATORS:
The engineering team is still working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line. DevOps identified some discrepancies on validator profile pages, such as total staked vs self staked. These discrepancies will be resolved with the Graph node rebuild.
We have marked two additional issues as RESOLVED since yesterday’s update and plan keep the community updated over the weekend.
1— FINDORA WALLET:
Engineering has resolved the issue where balances were not being displayed. This fix is contained in the new v0.4.4 release available on https://findora.org/#wallet. However, a new cosmetic/display issue has been identified where recently unbonded FRA is not reflected in the total balance. Engineering is working on a solution and prioritizing this as an urgent fix.
2— EVM STAKING PORTAL:
The DevOps team made changes which significantly improved response time on the new EVM staking portal. The team is working on a Graph node rebuild. This rebuild will improve response time for other components of the new staking portal which continue to experience a sluggish response time.
3— VALIDATORS:
The engineering team is still working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line. DevOps identified some discrepancies on validator profile pages, such as total staked vs self staked. These discrepancies will be resolved with the Graph node rebuild.
We have marked two additional issues as RESOLVED since yesterday’s update and plan keep the community updated over the weekend.
Update: [Oct 7th - 10:40 PM PST]
The Engineering team continues working on resolving issues with the Graph node and full node.
With the Graph node, queries receive no response shortly after the server loads and indexes data. This is likely why testing the v0.4.4 wallet was successful when the previous update was written, but began to fail shortly after.
The full node is showing the local DB receiving lock requests from different processes, causing high system resource utilization and degrading the server's response time.
Fixing the issue for the Graph node and full node will very likely resolve most or all issues with the wallet. I have confirmed again the engineering team is giving full attention to these fixes, treating this with urgent priority.
We'll circle back to everyone with more information tomorrow afternoon (PST) and allow the Engineering time time to focus on identifying root causes and implementing fixes.
The Engineering team continues working on resolving issues with the Graph node and full node.
With the Graph node, queries receive no response shortly after the server loads and indexes data. This is likely why testing the v0.4.4 wallet was successful when the previous update was written, but began to fail shortly after.
The full node is showing the local DB receiving lock requests from different processes, causing high system resource utilization and degrading the server's response time.
Fixing the issue for the Graph node and full node will very likely resolve most or all issues with the wallet. I have confirmed again the engineering team is giving full attention to these fixes, treating this with urgent priority.
We'll circle back to everyone with more information tomorrow afternoon (PST) and allow the Engineering time time to focus on identifying root causes and implementing fixes.
🙏2🔥1
Update: [Oct 9th - 10:00 PM PST]
Full Node
The DB lock issue on the full node is suspected to be resolved. However, testing is needed before installing this latest release on a production server. DevOps has the new version, which will go into testing tonight. This fix will improve UTXO functions, such as faster UTXO transactions.
If the new version tests well, the code will be installed into production in the next 24 to 48 hours.
Graph Node
The Graph node database shows an exceptionally high number of connections. This is leading to the Findora wallet displaying incorrect staking information. Engineering is reviewing more debugs off the Graph node to determine what is causing this behavior.
Bridged Tokens
Any tokens locked on the Rialto bridge will be released using a noscript being prepared by engineering in the next 24 to 48 hours. Tokens will be distributed back to the originating address.
Full Node
The DB lock issue on the full node is suspected to be resolved. However, testing is needed before installing this latest release on a production server. DevOps has the new version, which will go into testing tonight. This fix will improve UTXO functions, such as faster UTXO transactions.
If the new version tests well, the code will be installed into production in the next 24 to 48 hours.
Graph Node
The Graph node database shows an exceptionally high number of connections. This is leading to the Findora wallet displaying incorrect staking information. Engineering is reviewing more debugs off the Graph node to determine what is causing this behavior.
Bridged Tokens
Any tokens locked on the Rialto bridge will be released using a noscript being prepared by engineering in the next 24 to 48 hours. Tokens will be distributed back to the originating address.
Update: [October 10 - 10:00 PM]
Full Node
Last night, we received a newer build for Findora full nodes. However, issues were detected and resolved during testing. DevOps now has a newer build for full node in testing, and all appears to be going well. The team aims to upgrade the full nodes in the next 24 hours.
Note: This full node update will not resolve the incorrect staking balance. However, it will bring added stability to the network and its applications, such as the Findora wallet.
Graph Node
The engineering team has eliminated several potential root causes, allowing them to narrow their focus. They are continuing their debugging efforts. Once the root cause is identified, we can expect a solution in a speedy manner.
The team is working on these issues in real time, and hopes to provide more progress updates tomorrow.
Full Node
Last night, we received a newer build for Findora full nodes. However, issues were detected and resolved during testing. DevOps now has a newer build for full node in testing, and all appears to be going well. The team aims to upgrade the full nodes in the next 24 hours.
Note: This full node update will not resolve the incorrect staking balance. However, it will bring added stability to the network and its applications, such as the Findora wallet.
Graph Node
The engineering team has eliminated several potential root causes, allowing them to narrow their focus. They are continuing their debugging efforts. Once the root cause is identified, we can expect a solution in a speedy manner.
The team is working on these issues in real time, and hopes to provide more progress updates tomorrow.
🙏2
Update [October 11 - 10:45 PM PST]
Full Nodes
The DevOps team is testing a newer version that was provided since last night’s update. This version is in testing, and so far, the release appears to be stable.
If the team continues seeing stability, this version will be installed on all full nodes. Validators will not need to update. This is only applicable to full nodes. In the next 24 hours, we’re hoping to deliver news that all full nodes have been or are in the process of being upgraded.
Graph Node
We are seeing significant improvements on the Graph node since our previous update. The engineering team is monitoring the Graph node’s performance, and no issues have been observed in the past several hours. Previous observations showed failure events occurring in short succession of each other, so this is a great sign of progress.
If the Graph node continues showing stability, we can expect the in-wallet staking issues to be resolved within the next 24-48 hours. Like our update on the full nodes, we hope the next update will reflect a working and predictable balance and staking values in the wallet.
Note on Rialto
These issues have also impacted Rialto bridge transfers. Fixes for the Graph and full node should bring Rialto back into full service.
Next Steps
Please continue to monitor the pinned messages for updates. Once these issues are behind us — which should be very soon based on tonight's observations — the engineering team will begin drafting an RCA, which we will share with the community. We will also begin to draft a document detailing changes to our upgrade rollout process, including improved QA testing, internal escalation steps, and more.
We will have an updated Issue Tracker for the community in tomorrow's community update.
Full Nodes
The DevOps team is testing a newer version that was provided since last night’s update. This version is in testing, and so far, the release appears to be stable.
If the team continues seeing stability, this version will be installed on all full nodes. Validators will not need to update. This is only applicable to full nodes. In the next 24 hours, we’re hoping to deliver news that all full nodes have been or are in the process of being upgraded.
Graph Node
We are seeing significant improvements on the Graph node since our previous update. The engineering team is monitoring the Graph node’s performance, and no issues have been observed in the past several hours. Previous observations showed failure events occurring in short succession of each other, so this is a great sign of progress.
If the Graph node continues showing stability, we can expect the in-wallet staking issues to be resolved within the next 24-48 hours. Like our update on the full nodes, we hope the next update will reflect a working and predictable balance and staking values in the wallet.
Note on Rialto
These issues have also impacted Rialto bridge transfers. Fixes for the Graph and full node should bring Rialto back into full service.
Next Steps
Please continue to monitor the pinned messages for updates. Once these issues are behind us — which should be very soon based on tonight's observations — the engineering team will begin drafting an RCA, which we will share with the community. We will also begin to draft a document detailing changes to our upgrade rollout process, including improved QA testing, internal escalation steps, and more.
We will have an updated Issue Tracker for the community in tomorrow's community update.
🙏1
Update [October 12 - 10:35 PM PST]
Graph Node
The Graph node continues to show stability and correct data. This stability is allowing the EVM staking portal to reliably show valid stake information. While additional QA testing will be needed on the EVM staking portal, this good news brings the portal closer to public readiness.
Full Node
We are continuing to test newer versions for the full nodes, ironing out bugs and improving stability to ensure they do not fall behind. In the meantime, DevOps implemented a workaround that ensures all full nodes are operational and responsive, proactively preventing any impact the full nodes would have on the community. This workaround also ensures that the engineering team can approach their work cautiously while mitigating user impact.
_______________________
Next Steps ⤵️
_______________________
Wallet Unbonded Tokens
Now some good news...
With the Graph node showing stability and the full node workaround in place, unbonded tokens should be reflected properly in user wallets within the next 24 hours. No additional wallet upgrades will be needed. If you’re on a desktop, simply ensure you run the latest release v0.4.4.
Note: Staking FRA and claiming rewards should currently be functional in the Findora wallet. This fix is specifically for unbonded tokens.
Missing Bridged Assets (Rialto)
With the Graph node showing stability and the full node workaround in place, any users who bridged tokens over Rialto and could not retrieve them will find those tokens returned to their wallet in the next 24 hours.
________________________
Below you'll find the latest update to our tracker. We anticipate for the last item in the list regarding unbonded tokens to be resolved within the next 24 hours. I've also added notes for additional context on each item.
_________________________
TLDR: Unbonded tokens should appear in your wallet within the next 24 hours. Staking and claiming in the Findora wallet should currently be functional. Graph node is stable. Full node has a workaround to mitigate impact while engineers continue to work on it. Any missing tokens bridged over Rialto will be shown in the originator's wallet in the next 24 hours. EVM staking portal is closer to public readiness but needs QA after our retrospective is published.
Graph Node
The Graph node continues to show stability and correct data. This stability is allowing the EVM staking portal to reliably show valid stake information. While additional QA testing will be needed on the EVM staking portal, this good news brings the portal closer to public readiness.
Full Node
We are continuing to test newer versions for the full nodes, ironing out bugs and improving stability to ensure they do not fall behind. In the meantime, DevOps implemented a workaround that ensures all full nodes are operational and responsive, proactively preventing any impact the full nodes would have on the community. This workaround also ensures that the engineering team can approach their work cautiously while mitigating user impact.
_______________________
Next Steps ⤵️
_______________________
Wallet Unbonded Tokens
Now some good news...
With the Graph node showing stability and the full node workaround in place, unbonded tokens should be reflected properly in user wallets within the next 24 hours. No additional wallet upgrades will be needed. If you’re on a desktop, simply ensure you run the latest release v0.4.4.
Note: Staking FRA and claiming rewards should currently be functional in the Findora wallet. This fix is specifically for unbonded tokens.
Missing Bridged Assets (Rialto)
With the Graph node showing stability and the full node workaround in place, any users who bridged tokens over Rialto and could not retrieve them will find those tokens returned to their wallet in the next 24 hours.
________________________
Below you'll find the latest update to our tracker. We anticipate for the last item in the list regarding unbonded tokens to be resolved within the next 24 hours. I've also added notes for additional context on each item.
_________________________
TLDR: Unbonded tokens should appear in your wallet within the next 24 hours. Staking and claiming in the Findora wallet should currently be functional. Graph node is stable. Full node has a workaround to mitigate impact while engineers continue to work on it. Any missing tokens bridged over Rialto will be shown in the originator's wallet in the next 24 hours. EVM staking portal is closer to public readiness but needs QA after our retrospective is published.
❤1👍1
Update [October 13 - 11:30 PM PST]
Two fixes are top priority for engineers.
1. Unbonded Tokens Not Displayed in Wallet
2. Staked FRA Prior to Upgrade Not Displayed in Wallet
These issues are summarized in item 10 on the tracker. See previous pins for the latest tracker update.
After making some progress with the Graph node and full nodes, the Engineering team has redirected their troubleshooting and improvement focus to staking contracts. They've identified an issue that, when resolved, will have the Findora wallet display proper staking information.
The Engineering team created a fix which is under review by one of our smart contract developers. They are working together on reviewing the code which will be applied as soon as the review completes.
_________________________________
While this is a cosmetic issue, and all funds are safe, the inconvenience and concern expressed from the community is heard and understood. The team will work hard to improve our processes and share a retrospective once all priority items are addressed.
We are coming close to having all wallets fully functional and properly reflecting your balance. Thank you and we will share another update in 24 hours at the latest, though updates might be shared sooner.
_________________________________
TLDR: Engineers redirected their focus to staking contracts, found an issue, and have created a fix. The fix is under review by another smart contract engineer. If no issues are found in the code, and with the latest progress made on the Graph node and full nodes, the fix will be applied and all wallets will display correct balances.
Two fixes are top priority for engineers.
1. Unbonded Tokens Not Displayed in Wallet
2. Staked FRA Prior to Upgrade Not Displayed in Wallet
These issues are summarized in item 10 on the tracker. See previous pins for the latest tracker update.
After making some progress with the Graph node and full nodes, the Engineering team has redirected their troubleshooting and improvement focus to staking contracts. They've identified an issue that, when resolved, will have the Findora wallet display proper staking information.
The Engineering team created a fix which is under review by one of our smart contract developers. They are working together on reviewing the code which will be applied as soon as the review completes.
_________________________________
While this is a cosmetic issue, and all funds are safe, the inconvenience and concern expressed from the community is heard and understood. The team will work hard to improve our processes and share a retrospective once all priority items are addressed.
We are coming close to having all wallets fully functional and properly reflecting your balance. Thank you and we will share another update in 24 hours at the latest, though updates might be shared sooner.
_________________________________
TLDR: Engineers redirected their focus to staking contracts, found an issue, and have created a fix. The fix is under review by another smart contract engineer. If no issues are found in the code, and with the latest progress made on the Graph node and full nodes, the fix will be applied and all wallets will display correct balances.
❤3👍3🙏2
Update [October 15 - 11:02 PM]
Graph Node
Symptom: Staking Values Missing in Wallet (Cosmetic)
Current Status: Workaround Being Applied
Update: DevOps built a secondary Graph node to serve in a hot-standby role, meaning the backup Graph server can serve requests should the primary fail as we continue to fix this particular problem. This added redundancy should help reduce or avoid impact on your wallet staking information.
Wallet staking data should display properly in a few hours once the new Graph node catches up to the current block height.
Full Node (DB Lock)
Symptom: No User Impact
Current Status: Workaround Applied
Update: We are continuing to diagnose what has caused the database to lock on the full nodes. We have a current workaround that proactively restores service when this happens, but we will apply a more permanent solution once the root cause has been established.
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: Resolved
Update: All 113 unbonding transactions have been processed, and users whose unbonded tokens were missing should now show in the Findora wallet.
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: Resolved
Update: All Rialto Bridge transactions are now reflecting properly in user balances.
Graph Node
Symptom: Staking Values Missing in Wallet (Cosmetic)
Current Status: Workaround Being Applied
Update: DevOps built a secondary Graph node to serve in a hot-standby role, meaning the backup Graph server can serve requests should the primary fail as we continue to fix this particular problem. This added redundancy should help reduce or avoid impact on your wallet staking information.
Wallet staking data should display properly in a few hours once the new Graph node catches up to the current block height.
Full Node (DB Lock)
Symptom: No User Impact
Current Status: Workaround Applied
Update: We are continuing to diagnose what has caused the database to lock on the full nodes. We have a current workaround that proactively restores service when this happens, but we will apply a more permanent solution once the root cause has been established.
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: Resolved
Update: All 113 unbonding transactions have been processed, and users whose unbonded tokens were missing should now show in the Findora wallet.
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: Resolved
Update: All Rialto Bridge transactions are now reflecting properly in user balances.
👍4🙏2
Update | October 16 - 10:55 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟡
Work in Progress
Correct Data Sporadically Displays
Update: We found a query that’s causing our Graph node to lock up. This query requires an incredible amount of resources from the server. Engineers are working on reducing the resource requirements of the query and evaluating multiple solutions, choosing the one that doesn’t just resolve the issue today but also one that is most optimized and best for the long-term status of the service.
Optimization of queries and splitting certain database tables are two options engineers are evaluating.
In the meantime, DevOps is working on additional redundancy methods to reduce the recurrence of this issue so users are no longer impacted while a permanent solution is created. This is where troubleshooting efforts are focused right now.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Graph node continues to exhibit issues leading to sporadic incorrect wallet displays. DevOps is working on a more stable workaround (Work in Progress).
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟡
Work in Progress
Correct Data Sporadically Displays
Update: We found a query that’s causing our Graph node to lock up. This query requires an incredible amount of resources from the server. Engineers are working on reducing the resource requirements of the query and evaluating multiple solutions, choosing the one that doesn’t just resolve the issue today but also one that is most optimized and best for the long-term status of the service.
Optimization of queries and splitting certain database tables are two options engineers are evaluating.
In the meantime, DevOps is working on additional redundancy methods to reduce the recurrence of this issue so users are no longer impacted while a permanent solution is created. This is where troubleshooting efforts are focused right now.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Graph node continues to exhibit issues leading to sporadic incorrect wallet displays. DevOps is working on a more stable workaround (Work in Progress).
🙏7
Update | October 17 - 10:35 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟢 — Tentatively
Work in Progress
Correct Data Now Displaying
Update: Yesterday, we announced that a certain query was causing our Graph node to lock up because it required an incredible amount of resources from the server. Since last night, engineers have implemented a few different temporary workarounds which have resulted in significantly improved stability in the Findora wallet.
First, they have added redundancy through a backup node to reduce the recurrence of the issue. They have also upgraded the IO speed of a database disk and improved the VM, which have contributed to the stability of the Graph node seen today. The team will monitor the effectiveness of these temporary solutions as it continues to develop a permanent fix.
The result of all these efforts is that today, users have seen far more reliable data being displayed in their Findora wallets.
Now that the Graph node is looking more stable, the engineering team is starting to shift attention to the EVM Staking portal, and will soon be conducting the final checks needed before its public launch. More details to be released soon.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Work around now implemented for the Graph node which is showing more stability. Permanent solution in the works (Work in Progress).
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟢 — Tentatively
Work in Progress
Correct Data Now Displaying
Update: Yesterday, we announced that a certain query was causing our Graph node to lock up because it required an incredible amount of resources from the server. Since last night, engineers have implemented a few different temporary workarounds which have resulted in significantly improved stability in the Findora wallet.
First, they have added redundancy through a backup node to reduce the recurrence of the issue. They have also upgraded the IO speed of a database disk and improved the VM, which have contributed to the stability of the Graph node seen today. The team will monitor the effectiveness of these temporary solutions as it continues to develop a permanent fix.
The result of all these efforts is that today, users have seen far more reliable data being displayed in their Findora wallets.
Now that the Graph node is looking more stable, the engineering team is starting to shift attention to the EVM Staking portal, and will soon be conducting the final checks needed before its public launch. More details to be released soon.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Work around now implemented for the Graph node which is showing more stability. Permanent solution in the works (Work in Progress).
🔥7👍1🙏1