"signing_key": "PPY8hYvJB2kLdAVEShB8Qj55qSUDEGQjboZMXjW5rXTSzLvA2sKkK",
"next_secret_hash": "28f547c48fd9a01bae5430ebe5986229f165d941",
"previous_secret": "3ad61c9d1d976f42b770ae331829de72f37dc7e2",
"pay_vb": "1.13.307",
"vote_id": "1:118",
"total_votes": "14010233332",
"url": "https://salaheddinekassab.com/salah-witness.html",
"total_missed": 293,
"last_confirmed_block_num": 46301375
}


According to last_confirmed_block_num field of witness objects, the following witnesses did not upgrade:

jogersenjo1
salah8kassab
spacecrypt-witness
bitcoinsig


Similar to previous halts, it was decided to take what we call "supernode" approach for recovery. This involves:

1. Collecting signing keys of all or majority of active witnesses
2. Configuring a single node to run all witnesses who shared their signing keys
3. Trimming the index file to LIB
4. Replaying trimmed index file
5. Taking actions to prevent chain state that lead to halt
6. Replacing signing keys of active witnesess and taking over block signin from supernode

This is pretty standard way to recover halted network, that is previously used, and its well known to the community.

The call for collecting the keys was announced on 2021-12-31.
https://chat.peerplays.live/channel/witness-info?msg=TCow6GjNY4CJsSdDJ

Considering incoming New year celebration, it was not expected to have any response by 3rd Jan.

By January 2nd, 5 signatures were collected
https://chat.peerplays.live/channel/witness-info?msg=uthqrN4Amy9AtMwd9

By January 3nd, 7 signatures were collected and there was an attempt to restore the network for the first time
https://chat.peerplays.live/channel/witness-info?msg=gPa8sZ8EW2YjZ2Gbp

The index file was trimmed using the following command:

truncate -s 1481579840 ./witness_node_data_dir/blockchain/database/block_num_to_block/index

# LIB was 46299370, so 46299370 * 32 = 1481579840


Recovery process was trackable using cli_wallet
https://chat.peerplays.live/channel/witness-info?msg=dJCZPoDAHfKba3LfQ

It is important to mention that the signing keys were provided by two witnesses who did not update their software
earlier. With 7 witnesses who were already running updated software, with these 2 added, 9 out of 11
witnesses were willing to run latest update, which is a significant majority, and it was considered that consensus on running
latest update was reached.

Apparently, 7 witnesses who provided the signing keys were not enough to make LIB moving, and prevent another
network halt. Also, it appears that one of the witnesses provided invalid public/private key pair. At this point
the witnesses decided to outvote uncooperative witnesses:
https://chat.peerplays.live/channel/witness-info?msg=9A5XWaXBMuvvw9WpE

Later during the day, the witness provided valid signing key, and there was a total 7 out of 11 active witnesses
cooperating. Another call for rest of the active witnesses to join is announced:
https://chat.peerplays.live/channel/witness-info?msg=tQ43564kAzXB6qmkM

This is the list of witnesses who did and did not provided their signing keys:

hiltos-witness +
spacecrypt-witness -
royal-flush -
bitcoinsig -
houdini-witness -
eifos-witness +
miss-jana +
jogersenjo1 +
robert-hedler +
wit-super +


On January 5th, recovery process was stopped, due to imminent halt, due to not receiving signing keys from
any of 4 remaining active witnesses.
https://chat.peerplays.live/channel/witness-info?msg=ytYu9ebRedkQWkpjN

Later in the day, the recovery process was restarted, and witnesses who did not responded to our calls for help are voted out
Checkout blocks 46299399, 46299401, 46299402, 46299403, 46299404, 46299422, 46299435, 46299444, 46299584, 46299592

At 2022-01-05T21:00:00, voting results are processed, and new set of active witnesses is selected. Soon, participation
rate increased above threshold, the LIB started moving, and mainnet was restored.

On January 6th, it was announced that mainnet is restored, and provided a sync checkpoints to use

On January 7th 2022, following a successful restart, a bug has been identified that resulted in a fork of the chain. Witnesses and developers working on Peerplays are aware of the issue and actively working on a resolution.

A Root cause analysis needs to be done.. Sofie has contributed a bit to that which is helpful. If there are any other witnesses with information that will help identify the issue and/or a fix, please provide that information. With this information, the devs can figure out what caused it. After that the witnessess will need to determine who will take lead on the restart.

Peerplays Weekly Townhall Meeting January 07, 2022
https://lnkd.in/gniDDEDk

Peerplays Weekly Townhall Meeting January 21, 2022
https://youtu.be/_Pfd9h4E2Xg

Peerplays Weekly Townhall Meeting February 04, 2022
https://youtu.be/1mt-PzkJ1Og

Peerplays Weekly Townhall Meeting February 11, 2022
https://www.youtube.com/watch?v=rO3w7YSMO5w

The Peerplays blockchain is only as good as the community that surrounds it.

For this reason, we want to know what the Peerplays community wants to see from the Blockchain, and are soliciting feedback from the community to find out.

There are just a few questions, so it won’t take long, but the input provided will be of paramount importance. Not only will you get your voice heard, but you can help guide the first blockchain that is humanizing the crypto experience.

https://forms.gle/WzernbBREATa1wsS9

Thank you for taking the time to provide your feedback!

Peerplays Weekly Townhall Meeting March 18, 2022
https://www.youtube.com/watch?v=sfgJdbxGp90

Peerplays Weekly Townhall Meeting March 28, 2022
https://www.youtube.com/watch?v=4zX4rsaMoNk

Peerplays Weekly Townhall Meeting April 4, 2022
https://youtu.be/iYpi8orK7Kc

Peerplays Weekly Townhall Meeting April 11, 2022
https://www.youtube.com/watch?v=oqe_3bIjG60

Peerplays Weekly Townhall Meeting April 25, 2022
https://www.youtube.com/watch?v=XjTj6WBYnMk

Peerplays Weekly Townhall Meeting May 02, 2022
https://www.youtube.com/watch?v=posynMz-7mY

Peerplays Weekly Townhall Meeting May 09, 2022
https://www.youtube.com/watch?v=3A9U4w0fh3U

Peerplays Weekly Townhall Meeting May 24, 2022
https://www.youtube.com/watch?v=ozRKV68qKEY

Peerplays Weekly Townhall Meeting May 30, 2022
https://www.youtube.com/watch?v=hMQTdk5ZTX8

Peerplays Weekly Townhall Meeting June 06, 2022
https://www.youtube.com/watch?v=nteIs1FaEsY

Peerplays Weekly Townhall Meeting - June 13, 2022
https://youtu.be/Eb6F3VKDNEo