FRA Wallet Display Issues (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Latest Update [Oct. 3rd | 1:05 PM PST]
1— We have an engineer working on updating the Findora wallet and block scanner. This work should resolve the primary issue above re: the display of stake data.
2— The team also has an engineer who continues to work on the DB lock and performance impact to the full nodes. No new information is available since the last update, but efforts continue to resolve this problem.
3— The Rialto bridge is experiencing issues which should be resolved in the next 24 hours.
The underlying blockchain continues to run and the team is working around the clock resolving the issues above. Additional team members will be joining the effort in a few days when their national holiday comes to a close. Thank you for being patient while the team works to restore some of these services.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts.
Latest Update [Oct. 3rd | 1:05 PM PST]
1— We have an engineer working on updating the Findora wallet and block scanner. This work should resolve the primary issue above re: the display of stake data.
2— The team also has an engineer who continues to work on the DB lock and performance impact to the full nodes. No new information is available since the last update, but efforts continue to resolve this problem.
3— The Rialto bridge is experiencing issues which should be resolved in the next 24 hours.
The underlying blockchain continues to run and the team is working around the clock resolving the issues above. Additional team members will be joining the effort in a few days when their national holiday comes to a close. Thank you for being patient while the team works to restore some of these services.
👍1
FRA Wallet Display Issues & Portal Blockers (Update)
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts. This creates a situation where wallet users have trouble interacting with Findora's staking, and leads to blockers for the launch of the EVM staking portal.
Latest Update [Oct. 4th | 3:45 PM PST]
++++++++++++++++++++++++++++
1— FINDORA WALLET:
Engineering is continuing to work on resolving several issues. On the wallet side, we are updating its API methods used to obtain accurate staking data. Engineers deployed a new version (v0.4.4), but the issue persisted. The team is making progress, and we'll provide more updates as things move forward.
2— STAKING PORTAL:
DevOps is working on resolving issues impacting the full nodes. A new RPC server was set up for interactions with the upcoming EVM staking portal. The DevOps team is reloading the database on this new RPC server to improve staking transaction speed on the EVM staking portal.
3— VALIDATORS:
The new validator jailing threshold has been lowered to more forgiving values. Stake rewards are being generated and accumulating for delegators, but these rewards are not displayed on the wallet until item 1 above is resolved. The engineering team is working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line.
++++++++++++++++++++++++++++
As mentioned in the last update, the underlying blockchain continues to run, and the team is working around the clock to resolve the issues above. Additional team members will join the effort in a few days when their national holiday ends.
Finally, and to share more details with the community, we have shared a list of issues being resolved and improvement efforts in progress. We’re working hard to resolve these launch blockers and will share updated versions of the list as progress is made. See image below.
Thank you for being patient while the team works to restore some of these services.
Issue: The Findora wallet is displaying null values for staked, unstaking, and claimable amounts. This creates a situation where wallet users have trouble interacting with Findora's staking, and leads to blockers for the launch of the EVM staking portal.
Latest Update [Oct. 4th | 3:45 PM PST]
++++++++++++++++++++++++++++
1— FINDORA WALLET:
Engineering is continuing to work on resolving several issues. On the wallet side, we are updating its API methods used to obtain accurate staking data. Engineers deployed a new version (v0.4.4), but the issue persisted. The team is making progress, and we'll provide more updates as things move forward.
2— STAKING PORTAL:
DevOps is working on resolving issues impacting the full nodes. A new RPC server was set up for interactions with the upcoming EVM staking portal. The DevOps team is reloading the database on this new RPC server to improve staking transaction speed on the EVM staking portal.
3— VALIDATORS:
The new validator jailing threshold has been lowered to more forgiving values. Stake rewards are being generated and accumulating for delegators, but these rewards are not displayed on the wallet until item 1 above is resolved. The engineering team is working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line.
++++++++++++++++++++++++++++
As mentioned in the last update, the underlying blockchain continues to run, and the team is working around the clock to resolve the issues above. Additional team members will join the effort in a few days when their national holiday ends.
Finally, and to share more details with the community, we have shared a list of issues being resolved and improvement efforts in progress. We’re working hard to resolve these launch blockers and will share updated versions of the list as progress is made. See image below.
Thank you for being patient while the team works to restore some of these services.
❤1
Latest Update [Oct. 6th | 5:35 PM PST]
The Engineering team is currently working on fixes to the wallet and has made significant progress on removing blockers to launching the EVM Staking portal.
Each of the issues we’re resolving have specific technical task owners assigned, and task owners are scheduled to meet as a group tonight for a working session. We plan to release an update on the solutions tracker and provide the community with more details once the meeting has concluded.
Rest assured that these are display issues and validator, delegator, and user funds and rewards are safe.
The Engineering team is currently working on fixes to the wallet and has made significant progress on removing blockers to launching the EVM Staking portal.
Each of the issues we’re resolving have specific technical task owners assigned, and task owners are scheduled to meet as a group tonight for a working session. We plan to release an update on the solutions tracker and provide the community with more details once the meeting has concluded.
Rest assured that these are display issues and validator, delegator, and user funds and rewards are safe.
Latest Update [Oct. 6th | 8:50 PM PST]
1— FINDORA WALLET:
Engineering has resolved the issue where balances were not being displayed. This fix is contained in the new v0.4.4 release available on https://findora.org/#wallet. However, a new cosmetic/display issue has been identified where recently unbonded FRA is not reflected in the total balance. Engineering is working on a solution and prioritizing this as an urgent fix.
2— EVM STAKING PORTAL:
The DevOps team made changes which significantly improved response time on the new EVM staking portal. The team is working on a Graph node rebuild. This rebuild will improve response time for other components of the new staking portal which continue to experience a sluggish response time.
3— VALIDATORS:
The engineering team is still working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line. DevOps identified some discrepancies on validator profile pages, such as total staked vs self staked. These discrepancies will be resolved with the Graph node rebuild.
We have marked two additional issues as RESOLVED since yesterday’s update and plan keep the community updated over the weekend.
1— FINDORA WALLET:
Engineering has resolved the issue where balances were not being displayed. This fix is contained in the new v0.4.4 release available on https://findora.org/#wallet. However, a new cosmetic/display issue has been identified where recently unbonded FRA is not reflected in the total balance. Engineering is working on a solution and prioritizing this as an urgent fix.
2— EVM STAKING PORTAL:
The DevOps team made changes which significantly improved response time on the new EVM staking portal. The team is working on a Graph node rebuild. This rebuild will improve response time for other components of the new staking portal which continue to experience a sluggish response time.
3— VALIDATORS:
The engineering team is still working on providing node runners with a noscript for manual removal from jailed status. We also will create a way for node runners to check their jailed status via the command line. DevOps identified some discrepancies on validator profile pages, such as total staked vs self staked. These discrepancies will be resolved with the Graph node rebuild.
We have marked two additional issues as RESOLVED since yesterday’s update and plan keep the community updated over the weekend.
Update: [Oct 7th - 10:40 PM PST]
The Engineering team continues working on resolving issues with the Graph node and full node.
With the Graph node, queries receive no response shortly after the server loads and indexes data. This is likely why testing the v0.4.4 wallet was successful when the previous update was written, but began to fail shortly after.
The full node is showing the local DB receiving lock requests from different processes, causing high system resource utilization and degrading the server's response time.
Fixing the issue for the Graph node and full node will very likely resolve most or all issues with the wallet. I have confirmed again the engineering team is giving full attention to these fixes, treating this with urgent priority.
We'll circle back to everyone with more information tomorrow afternoon (PST) and allow the Engineering time time to focus on identifying root causes and implementing fixes.
The Engineering team continues working on resolving issues with the Graph node and full node.
With the Graph node, queries receive no response shortly after the server loads and indexes data. This is likely why testing the v0.4.4 wallet was successful when the previous update was written, but began to fail shortly after.
The full node is showing the local DB receiving lock requests from different processes, causing high system resource utilization and degrading the server's response time.
Fixing the issue for the Graph node and full node will very likely resolve most or all issues with the wallet. I have confirmed again the engineering team is giving full attention to these fixes, treating this with urgent priority.
We'll circle back to everyone with more information tomorrow afternoon (PST) and allow the Engineering time time to focus on identifying root causes and implementing fixes.
🙏2🔥1
Update: [Oct 9th - 10:00 PM PST]
Full Node
The DB lock issue on the full node is suspected to be resolved. However, testing is needed before installing this latest release on a production server. DevOps has the new version, which will go into testing tonight. This fix will improve UTXO functions, such as faster UTXO transactions.
If the new version tests well, the code will be installed into production in the next 24 to 48 hours.
Graph Node
The Graph node database shows an exceptionally high number of connections. This is leading to the Findora wallet displaying incorrect staking information. Engineering is reviewing more debugs off the Graph node to determine what is causing this behavior.
Bridged Tokens
Any tokens locked on the Rialto bridge will be released using a noscript being prepared by engineering in the next 24 to 48 hours. Tokens will be distributed back to the originating address.
Full Node
The DB lock issue on the full node is suspected to be resolved. However, testing is needed before installing this latest release on a production server. DevOps has the new version, which will go into testing tonight. This fix will improve UTXO functions, such as faster UTXO transactions.
If the new version tests well, the code will be installed into production in the next 24 to 48 hours.
Graph Node
The Graph node database shows an exceptionally high number of connections. This is leading to the Findora wallet displaying incorrect staking information. Engineering is reviewing more debugs off the Graph node to determine what is causing this behavior.
Bridged Tokens
Any tokens locked on the Rialto bridge will be released using a noscript being prepared by engineering in the next 24 to 48 hours. Tokens will be distributed back to the originating address.
Update: [October 10 - 10:00 PM]
Full Node
Last night, we received a newer build for Findora full nodes. However, issues were detected and resolved during testing. DevOps now has a newer build for full node in testing, and all appears to be going well. The team aims to upgrade the full nodes in the next 24 hours.
Note: This full node update will not resolve the incorrect staking balance. However, it will bring added stability to the network and its applications, such as the Findora wallet.
Graph Node
The engineering team has eliminated several potential root causes, allowing them to narrow their focus. They are continuing their debugging efforts. Once the root cause is identified, we can expect a solution in a speedy manner.
The team is working on these issues in real time, and hopes to provide more progress updates tomorrow.
Full Node
Last night, we received a newer build for Findora full nodes. However, issues were detected and resolved during testing. DevOps now has a newer build for full node in testing, and all appears to be going well. The team aims to upgrade the full nodes in the next 24 hours.
Note: This full node update will not resolve the incorrect staking balance. However, it will bring added stability to the network and its applications, such as the Findora wallet.
Graph Node
The engineering team has eliminated several potential root causes, allowing them to narrow their focus. They are continuing their debugging efforts. Once the root cause is identified, we can expect a solution in a speedy manner.
The team is working on these issues in real time, and hopes to provide more progress updates tomorrow.
🙏2
Update [October 11 - 10:45 PM PST]
Full Nodes
The DevOps team is testing a newer version that was provided since last night’s update. This version is in testing, and so far, the release appears to be stable.
If the team continues seeing stability, this version will be installed on all full nodes. Validators will not need to update. This is only applicable to full nodes. In the next 24 hours, we’re hoping to deliver news that all full nodes have been or are in the process of being upgraded.
Graph Node
We are seeing significant improvements on the Graph node since our previous update. The engineering team is monitoring the Graph node’s performance, and no issues have been observed in the past several hours. Previous observations showed failure events occurring in short succession of each other, so this is a great sign of progress.
If the Graph node continues showing stability, we can expect the in-wallet staking issues to be resolved within the next 24-48 hours. Like our update on the full nodes, we hope the next update will reflect a working and predictable balance and staking values in the wallet.
Note on Rialto
These issues have also impacted Rialto bridge transfers. Fixes for the Graph and full node should bring Rialto back into full service.
Next Steps
Please continue to monitor the pinned messages for updates. Once these issues are behind us — which should be very soon based on tonight's observations — the engineering team will begin drafting an RCA, which we will share with the community. We will also begin to draft a document detailing changes to our upgrade rollout process, including improved QA testing, internal escalation steps, and more.
We will have an updated Issue Tracker for the community in tomorrow's community update.
Full Nodes
The DevOps team is testing a newer version that was provided since last night’s update. This version is in testing, and so far, the release appears to be stable.
If the team continues seeing stability, this version will be installed on all full nodes. Validators will not need to update. This is only applicable to full nodes. In the next 24 hours, we’re hoping to deliver news that all full nodes have been or are in the process of being upgraded.
Graph Node
We are seeing significant improvements on the Graph node since our previous update. The engineering team is monitoring the Graph node’s performance, and no issues have been observed in the past several hours. Previous observations showed failure events occurring in short succession of each other, so this is a great sign of progress.
If the Graph node continues showing stability, we can expect the in-wallet staking issues to be resolved within the next 24-48 hours. Like our update on the full nodes, we hope the next update will reflect a working and predictable balance and staking values in the wallet.
Note on Rialto
These issues have also impacted Rialto bridge transfers. Fixes for the Graph and full node should bring Rialto back into full service.
Next Steps
Please continue to monitor the pinned messages for updates. Once these issues are behind us — which should be very soon based on tonight's observations — the engineering team will begin drafting an RCA, which we will share with the community. We will also begin to draft a document detailing changes to our upgrade rollout process, including improved QA testing, internal escalation steps, and more.
We will have an updated Issue Tracker for the community in tomorrow's community update.
🙏1
Update [October 12 - 10:35 PM PST]
Graph Node
The Graph node continues to show stability and correct data. This stability is allowing the EVM staking portal to reliably show valid stake information. While additional QA testing will be needed on the EVM staking portal, this good news brings the portal closer to public readiness.
Full Node
We are continuing to test newer versions for the full nodes, ironing out bugs and improving stability to ensure they do not fall behind. In the meantime, DevOps implemented a workaround that ensures all full nodes are operational and responsive, proactively preventing any impact the full nodes would have on the community. This workaround also ensures that the engineering team can approach their work cautiously while mitigating user impact.
_______________________
Next Steps ⤵️
_______________________
Wallet Unbonded Tokens
Now some good news...
With the Graph node showing stability and the full node workaround in place, unbonded tokens should be reflected properly in user wallets within the next 24 hours. No additional wallet upgrades will be needed. If you’re on a desktop, simply ensure you run the latest release v0.4.4.
Note: Staking FRA and claiming rewards should currently be functional in the Findora wallet. This fix is specifically for unbonded tokens.
Missing Bridged Assets (Rialto)
With the Graph node showing stability and the full node workaround in place, any users who bridged tokens over Rialto and could not retrieve them will find those tokens returned to their wallet in the next 24 hours.
________________________
Below you'll find the latest update to our tracker. We anticipate for the last item in the list regarding unbonded tokens to be resolved within the next 24 hours. I've also added notes for additional context on each item.
_________________________
TLDR: Unbonded tokens should appear in your wallet within the next 24 hours. Staking and claiming in the Findora wallet should currently be functional. Graph node is stable. Full node has a workaround to mitigate impact while engineers continue to work on it. Any missing tokens bridged over Rialto will be shown in the originator's wallet in the next 24 hours. EVM staking portal is closer to public readiness but needs QA after our retrospective is published.
Graph Node
The Graph node continues to show stability and correct data. This stability is allowing the EVM staking portal to reliably show valid stake information. While additional QA testing will be needed on the EVM staking portal, this good news brings the portal closer to public readiness.
Full Node
We are continuing to test newer versions for the full nodes, ironing out bugs and improving stability to ensure they do not fall behind. In the meantime, DevOps implemented a workaround that ensures all full nodes are operational and responsive, proactively preventing any impact the full nodes would have on the community. This workaround also ensures that the engineering team can approach their work cautiously while mitigating user impact.
_______________________
Next Steps ⤵️
_______________________
Wallet Unbonded Tokens
Now some good news...
With the Graph node showing stability and the full node workaround in place, unbonded tokens should be reflected properly in user wallets within the next 24 hours. No additional wallet upgrades will be needed. If you’re on a desktop, simply ensure you run the latest release v0.4.4.
Note: Staking FRA and claiming rewards should currently be functional in the Findora wallet. This fix is specifically for unbonded tokens.
Missing Bridged Assets (Rialto)
With the Graph node showing stability and the full node workaround in place, any users who bridged tokens over Rialto and could not retrieve them will find those tokens returned to their wallet in the next 24 hours.
________________________
Below you'll find the latest update to our tracker. We anticipate for the last item in the list regarding unbonded tokens to be resolved within the next 24 hours. I've also added notes for additional context on each item.
_________________________
TLDR: Unbonded tokens should appear in your wallet within the next 24 hours. Staking and claiming in the Findora wallet should currently be functional. Graph node is stable. Full node has a workaround to mitigate impact while engineers continue to work on it. Any missing tokens bridged over Rialto will be shown in the originator's wallet in the next 24 hours. EVM staking portal is closer to public readiness but needs QA after our retrospective is published.
❤1👍1
Update [October 13 - 11:30 PM PST]
Two fixes are top priority for engineers.
1. Unbonded Tokens Not Displayed in Wallet
2. Staked FRA Prior to Upgrade Not Displayed in Wallet
These issues are summarized in item 10 on the tracker. See previous pins for the latest tracker update.
After making some progress with the Graph node and full nodes, the Engineering team has redirected their troubleshooting and improvement focus to staking contracts. They've identified an issue that, when resolved, will have the Findora wallet display proper staking information.
The Engineering team created a fix which is under review by one of our smart contract developers. They are working together on reviewing the code which will be applied as soon as the review completes.
_________________________________
While this is a cosmetic issue, and all funds are safe, the inconvenience and concern expressed from the community is heard and understood. The team will work hard to improve our processes and share a retrospective once all priority items are addressed.
We are coming close to having all wallets fully functional and properly reflecting your balance. Thank you and we will share another update in 24 hours at the latest, though updates might be shared sooner.
_________________________________
TLDR: Engineers redirected their focus to staking contracts, found an issue, and have created a fix. The fix is under review by another smart contract engineer. If no issues are found in the code, and with the latest progress made on the Graph node and full nodes, the fix will be applied and all wallets will display correct balances.
Two fixes are top priority for engineers.
1. Unbonded Tokens Not Displayed in Wallet
2. Staked FRA Prior to Upgrade Not Displayed in Wallet
These issues are summarized in item 10 on the tracker. See previous pins for the latest tracker update.
After making some progress with the Graph node and full nodes, the Engineering team has redirected their troubleshooting and improvement focus to staking contracts. They've identified an issue that, when resolved, will have the Findora wallet display proper staking information.
The Engineering team created a fix which is under review by one of our smart contract developers. They are working together on reviewing the code which will be applied as soon as the review completes.
_________________________________
While this is a cosmetic issue, and all funds are safe, the inconvenience and concern expressed from the community is heard and understood. The team will work hard to improve our processes and share a retrospective once all priority items are addressed.
We are coming close to having all wallets fully functional and properly reflecting your balance. Thank you and we will share another update in 24 hours at the latest, though updates might be shared sooner.
_________________________________
TLDR: Engineers redirected their focus to staking contracts, found an issue, and have created a fix. The fix is under review by another smart contract engineer. If no issues are found in the code, and with the latest progress made on the Graph node and full nodes, the fix will be applied and all wallets will display correct balances.
❤3👍3🙏2
Update [October 15 - 11:02 PM]
Graph Node
Symptom: Staking Values Missing in Wallet (Cosmetic)
Current Status: Workaround Being Applied
Update: DevOps built a secondary Graph node to serve in a hot-standby role, meaning the backup Graph server can serve requests should the primary fail as we continue to fix this particular problem. This added redundancy should help reduce or avoid impact on your wallet staking information.
Wallet staking data should display properly in a few hours once the new Graph node catches up to the current block height.
Full Node (DB Lock)
Symptom: No User Impact
Current Status: Workaround Applied
Update: We are continuing to diagnose what has caused the database to lock on the full nodes. We have a current workaround that proactively restores service when this happens, but we will apply a more permanent solution once the root cause has been established.
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: Resolved
Update: All 113 unbonding transactions have been processed, and users whose unbonded tokens were missing should now show in the Findora wallet.
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: Resolved
Update: All Rialto Bridge transactions are now reflecting properly in user balances.
Graph Node
Symptom: Staking Values Missing in Wallet (Cosmetic)
Current Status: Workaround Being Applied
Update: DevOps built a secondary Graph node to serve in a hot-standby role, meaning the backup Graph server can serve requests should the primary fail as we continue to fix this particular problem. This added redundancy should help reduce or avoid impact on your wallet staking information.
Wallet staking data should display properly in a few hours once the new Graph node catches up to the current block height.
Full Node (DB Lock)
Symptom: No User Impact
Current Status: Workaround Applied
Update: We are continuing to diagnose what has caused the database to lock on the full nodes. We have a current workaround that proactively restores service when this happens, but we will apply a more permanent solution once the root cause has been established.
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: Resolved
Update: All 113 unbonding transactions have been processed, and users whose unbonded tokens were missing should now show in the Findora wallet.
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: Resolved
Update: All Rialto Bridge transactions are now reflecting properly in user balances.
👍4🙏2
Update | October 16 - 10:55 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟡
Work in Progress
Correct Data Sporadically Displays
Update: We found a query that’s causing our Graph node to lock up. This query requires an incredible amount of resources from the server. Engineers are working on reducing the resource requirements of the query and evaluating multiple solutions, choosing the one that doesn’t just resolve the issue today but also one that is most optimized and best for the long-term status of the service.
Optimization of queries and splitting certain database tables are two options engineers are evaluating.
In the meantime, DevOps is working on additional redundancy methods to reduce the recurrence of this issue so users are no longer impacted while a permanent solution is created. This is where troubleshooting efforts are focused right now.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Graph node continues to exhibit issues leading to sporadic incorrect wallet displays. DevOps is working on a more stable workaround (Work in Progress).
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟡
Work in Progress
Correct Data Sporadically Displays
Update: We found a query that’s causing our Graph node to lock up. This query requires an incredible amount of resources from the server. Engineers are working on reducing the resource requirements of the query and evaluating multiple solutions, choosing the one that doesn’t just resolve the issue today but also one that is most optimized and best for the long-term status of the service.
Optimization of queries and splitting certain database tables are two options engineers are evaluating.
In the meantime, DevOps is working on additional redundancy methods to reduce the recurrence of this issue so users are no longer impacted while a permanent solution is created. This is where troubleshooting efforts are focused right now.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Graph node continues to exhibit issues leading to sporadic incorrect wallet displays. DevOps is working on a more stable workaround (Work in Progress).
🙏7
Update | October 17 - 10:35 PM
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟢 — Tentatively
Work in Progress
Correct Data Now Displaying
Update: Yesterday, we announced that a certain query was causing our Graph node to lock up because it required an incredible amount of resources from the server. Since last night, engineers have implemented a few different temporary workarounds which have resulted in significantly improved stability in the Findora wallet.
First, they have added redundancy through a backup node to reduce the recurrence of the issue. They have also upgraded the IO speed of a database disk and improved the VM, which have contributed to the stability of the Graph node seen today. The team will monitor the effectiveness of these temporary solutions as it continues to develop a permanent fix.
The result of all these efforts is that today, users have seen far more reliable data being displayed in their Findora wallets.
Now that the Graph node is looking more stable, the engineering team is starting to shift attention to the EVM Staking portal, and will soon be conducting the final checks needed before its public launch. More details to be released soon.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Work around now implemented for the Graph node which is showing more stability. Permanent solution in the works (Work in Progress).
Full Node (DB Lock)
Symptom: No User Impact
Current Status: 🟢
Workaround Applied
Rialto
Symptom: Tokens transferred over Rialto weren’t appearing in wallets.
Current Status: 🟢
Resolved
Wallet Unbonded Tokens
Symptom: Unbonded FRA was not displaying correctly in the wallet.
Current Status: 🟢
Resolved
Graph Node
Symptom: Staking Values Missing or Shows Old Data in the Findora Wallet (Cosmetic)
Current Status: 🟢 — Tentatively
Work in Progress
Correct Data Now Displaying
Update: Yesterday, we announced that a certain query was causing our Graph node to lock up because it required an incredible amount of resources from the server. Since last night, engineers have implemented a few different temporary workarounds which have resulted in significantly improved stability in the Findora wallet.
First, they have added redundancy through a backup node to reduce the recurrence of the issue. They have also upgraded the IO speed of a database disk and improved the VM, which have contributed to the stability of the Graph node seen today. The team will monitor the effectiveness of these temporary solutions as it continues to develop a permanent fix.
The result of all these efforts is that today, users have seen far more reliable data being displayed in their Findora wallets.
Now that the Graph node is looking more stable, the engineering team is starting to shift attention to the EVM Staking portal, and will soon be conducting the final checks needed before its public launch. More details to be released soon.
TLDR: Unbonded FRA and stuck Rialto tokens are back in affected user wallets (Resolved). Full nodes have a stable workaround, pending permanent fix (Workaround). Work around now implemented for the Graph node which is showing more stability. Permanent solution in the works (Work in Progress).
🔥7👍1🙏1
October 18 - 10:25 PM
Update: The Graph node has been far more stable since yesterday’s update. However, despite the improvements, there was a brief period today when the Graph node became overburdened.
To address this, the engineering team has doubled the size of the VM today. This temporary solution should bring additional performance improvements of the Graph node as we work to implement a more permanent fix.
Meanwhile, we are aware of an issue with the iOS version of the Findora wallet app where the staking screen becomes stuck on "Loading." We have found that if other pages on the app freeze, reinstalling the app and importing your private keys tends to fix the issue. An engineer is tasked with running diagnostics to find the underlying problem of the stuck staking screen.
We are looking forward to finally focusing on testing and launching the new EVM Staking platform. Validators on Findora, if they’re willing, will be asked for their time to participate in additional testing of the new portal before it goes live. We know it has been a trying time, and we appreciate everyone’s patience and understanding.
TLDR: We have doubled the VM size to stabilize the Graph node as we look for a permanent fix (workaround); An engineer has been tasked with fixing the Findora wallet iOS app (work in progress); EVM Staking portal is coming soon.
Update: The Graph node has been far more stable since yesterday’s update. However, despite the improvements, there was a brief period today when the Graph node became overburdened.
To address this, the engineering team has doubled the size of the VM today. This temporary solution should bring additional performance improvements of the Graph node as we work to implement a more permanent fix.
Meanwhile, we are aware of an issue with the iOS version of the Findora wallet app where the staking screen becomes stuck on "Loading." We have found that if other pages on the app freeze, reinstalling the app and importing your private keys tends to fix the issue. An engineer is tasked with running diagnostics to find the underlying problem of the stuck staking screen.
We are looking forward to finally focusing on testing and launching the new EVM Staking platform. Validators on Findora, if they’re willing, will be asked for their time to participate in additional testing of the new portal before it goes live. We know it has been a trying time, and we appreciate everyone’s patience and understanding.
TLDR: We have doubled the VM size to stabilize the Graph node as we look for a permanent fix (workaround); An engineer has been tasked with fixing the Findora wallet iOS app (work in progress); EVM Staking portal is coming soon.
Update | October 19, 9:00PM
Summary: The team continues to make changes to make the Graph node more stable. As shared in previous updates, we have doubled the size of the VM and upgraded the IO speed of the database disk.
Though the desktop wallet has shown significant improvement today, we noticed that staking data sporadically failed to display. However, its performance today was far more reliable than previous days.
We’re hoping the updates below add to the Graph node’s stability as we implement a permanent fix.
Updates
Graph Node 🟢
We are also making the EVM Staking portal more efficient. Previously, when the portal loaded, several queries were simultaneously sent to fetch validator data off the Graph node. We have reduced that to a single query instead, which should reduce unnecessary load and improve Graph node performance.
We believe these improvements add to the stability of the Graph node and serve as a workaround. As a long-term solution, engineers are strategizing to restructure the database. This work will optimize how specific tables are structured, further enhance the Graph node’s performance, and provide a more permanent solution.
iOS Wallet 🟡
The staking screen on the iOS wallet app is still experiencing an issue where it gets stuck on the “Loading.” We have assigned an engineer to resolve the issue, and we are hopeful it will be fixed soon.
TLDR: With improvements being made to the Graph node and iOS wallet app, EVM Staking is getting much closer to being ready for its public launch.
Summary: The team continues to make changes to make the Graph node more stable. As shared in previous updates, we have doubled the size of the VM and upgraded the IO speed of the database disk.
Though the desktop wallet has shown significant improvement today, we noticed that staking data sporadically failed to display. However, its performance today was far more reliable than previous days.
We’re hoping the updates below add to the Graph node’s stability as we implement a permanent fix.
Updates
Graph Node 🟢
We are also making the EVM Staking portal more efficient. Previously, when the portal loaded, several queries were simultaneously sent to fetch validator data off the Graph node. We have reduced that to a single query instead, which should reduce unnecessary load and improve Graph node performance.
We believe these improvements add to the stability of the Graph node and serve as a workaround. As a long-term solution, engineers are strategizing to restructure the database. This work will optimize how specific tables are structured, further enhance the Graph node’s performance, and provide a more permanent solution.
iOS Wallet 🟡
The staking screen on the iOS wallet app is still experiencing an issue where it gets stuck on the “Loading.” We have assigned an engineer to resolve the issue, and we are hopeful it will be fixed soon.
TLDR: With improvements being made to the Graph node and iOS wallet app, EVM Staking is getting much closer to being ready for its public launch.
🔥2
[Official] Findora Announcement Channel pinned «Update | October 19, 9:00PM Summary: The team continues to make changes to make the Graph node more stable. As shared in previous updates, we have doubled the size of the VM and upgraded the IO speed of the database disk. Though the desktop wallet has shown…»
Brief Update - October 20 | 10:20 PM PST
Graph Node & Desktop Wallet
The Graph node has been more stable these past 48 hours compared to the performance impact we've observed these past couple of weeks.
This means the desktop wallet data display has been more reliable. However, the wallet may sporadically fail to display staking data until a permanent fix is created. While not the most ideal, the sporadic issues are temporary, and the improvements are a step towards stability and reliability.
In the meantime, the permanent fix - restructuring of the Graph node database - is expected to take approximately two weeks.
EVM Staking Dashboard
Assuming we continue seeing improvements on the Graph node, we plan to finish the EVM staking dashboard and resume QA testing next week.
iOS Mobile Wallet
An engineer is continuing to work on the iOS wallet. Currently, the staking screen becomes stuck on "Loading." We're hoping to have this resolved by early to mid next week.
——————————————
Note: Daily updates will resume on Monday.
Graph Node & Desktop Wallet
The Graph node has been more stable these past 48 hours compared to the performance impact we've observed these past couple of weeks.
This means the desktop wallet data display has been more reliable. However, the wallet may sporadically fail to display staking data until a permanent fix is created. While not the most ideal, the sporadic issues are temporary, and the improvements are a step towards stability and reliability.
In the meantime, the permanent fix - restructuring of the Graph node database - is expected to take approximately two weeks.
EVM Staking Dashboard
Assuming we continue seeing improvements on the Graph node, we plan to finish the EVM staking dashboard and resume QA testing next week.
iOS Mobile Wallet
An engineer is continuing to work on the iOS wallet. Currently, the staking screen becomes stuck on "Loading." We're hoping to have this resolved by early to mid next week.
——————————————
Note: Daily updates will resume on Monday.
👍4⚡1
Brief Update - October 24 | 12:55 AM PST
Findora Wallet
A new update is available for the FIndora wallet, v0.4.5. This new version contains an update to the RPC URL and was originally set to use new, multiple Web3 endpoints when it was published, increasing redundancy for data retrieval. However, a cosmetic error was quickly noticed which drove DevOps to redirect the RPC traffic to Findora full nodes instead.
Engineers are resolving the cosmetic issue on the Web3 endpoints which should be done relatively quickly. Once that's done, DevOps will redirect wallet RPC traffic back to the Web3 endpoints via load balancer settings.
This new wallet update is available for Windows and will soon be available for other platforms as well. We'll share more information on their progress in our next update to be shared within 24 hours.
TLDR: New wallet version 0.4.5 is available for Windows, coming soon to other platforms. RPC URL for the new wallet was updated for more redundancy but a cosmetic issue was found and soon to be resolved.
Findora Wallet
A new update is available for the FIndora wallet, v0.4.5. This new version contains an update to the RPC URL and was originally set to use new, multiple Web3 endpoints when it was published, increasing redundancy for data retrieval. However, a cosmetic error was quickly noticed which drove DevOps to redirect the RPC traffic to Findora full nodes instead.
Engineers are resolving the cosmetic issue on the Web3 endpoints which should be done relatively quickly. Once that's done, DevOps will redirect wallet RPC traffic back to the Web3 endpoints via load balancer settings.
This new wallet update is available for Windows and will soon be available for other platforms as well. We'll share more information on their progress in our next update to be shared within 24 hours.
TLDR: New wallet version 0.4.5 is available for Windows, coming soon to other platforms. RPC URL for the new wallet was updated for more redundancy but a cosmetic issue was found and soon to be resolved.
👍2🙏1