Proxmox VE: restart node due to lock-xxx.conf' - got timeout

From time to time I have to restart the servers due to loss of control.

Configuration:

Supermicro Ultra Server

Intel Xeon Scalable gen2
1Tb RAM
4x4Tb Samsung Data Center SSD


Sometimes I get such messages in the log

Feb 17 02:38:29 kfn1-node4 pvedaemon\[38846\]: can't lock file '/var/lock/qemu-server/lock-141.conf' - got timeout Feb 17 02:38:29 kfn1-node4 pvedaemon\[8817\]: <root@pam> end task UPID:kfn1-node4:000097BE:08EF85DA:5E49EE8B:qmstop:141:root@pam: can't lock file '/var/lock/qemu-server/lock-141.conf' - got timeout

and from this moment on I can't start / stop virtual servers.
Before this problem I always see such messages in the /var/logs/syslog:


Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520706\] CPU: 52 PID: 1649 Comm: z\_wr\_int\_6 Tainted: P O 4.15.18-21-pve #1 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520736\] Hardware name: Supermicro SYS-1029U-TR4T/X11DPU, BIOS 3.1a 07/19/2019 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520811\] RIP: 0010:buf\_hash\_insert+0xbd/0x180 \[zfs\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520828\] RSP: 0018:ffffb07b60eebcc0 EFLAGS: 00010206 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520846\] RAX: 1b0210fa010154fc RBX: ffff95e1569c91f0 RCX: 0000000000000080 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520866\] RDX: 0000000000000001 RSI: ffff96324bf587b0 RDI: ffffb07b7736d558 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520886\] RBP: ffffb07b60eebcd8 R08: ffff95e1569c9200 R09: 0000000000baf367 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520913\] R10: ffffb07b60eebce0 R11: 0000000000000000 R12: 00000000028256ab Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520934\] R13: 000000000005aac0 R14: ffff96335e5df690 R15: ffff9633b1dcbb50 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520958\] FS: 0000000000000000(0000) GS:ffff9635bf300000(0000) knlGS:0000000000000000 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.520983\] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521000\] CR2: 00007f79ef135000 CR3: 0000002adea0a003 CR4: 00000000007626e0 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521020\] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521040\] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521060\] PKRU: 55555554 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521070\] Call Trace: Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521123\] arc\_write\_done+0x125/0x3f0 \[zfs\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521164\] zio\_done+0x2d0/0xe60 \[zfs\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521179\] ? kfree+0x165/0x180 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521197\] ? spl\_kmem\_free+0x33/0x40 \[spl\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521237\] zio\_execute+0x95/0xf0 \[zfs\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521253\] taskq\_thread+0x2ae/0x4d0 \[spl\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.521268\] ? wake\_up\_q+0x80/0x80 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.522089\] ? zio\_reexecute+0x390/0x390 \[zfs\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.523021\] kthread+0x105/0x140 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.523759\] ? task\_done+0xb0/0xb0 \[spl\] Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.524503\] ? kthread\_create\_worker\_on\_cpu+0x70/0x70 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.525215\] ret\_from\_fork+0x1f/0x40 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.525909\] Code: 05 31 7c 18 00 4a 8d 3c e0 48 8b 37 48 85 f6 0f 84 c2 00 00 00 48 8b 0b 48 89 f0 31 d2 eb 0c 48 8b 40 20 83 c2 01 48 85 c0 74 2f <48> 39 08 75 ef 4c 8b 53 08 4c 39 50 08 75 e5 4c 8 b 5b 10 4c 39 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.527427\] RIP: buf\_hash\_insert+0xbd/0x180 \[zfs\] RSP: ffffb07b60eebcc0 Feb 17 01:49:32 kfn1-node4 kernel: \[1496221.528218\] ---\[ end trace 1ec84add9901e42b \]---

Does anyone have any idea how to diagnose such a

problem?

https://redd.it/ftu918
@r_linux

ForgeFed news: We sent @NGIZero the funding plan,waiting for them to approve it. The funded work,by zPlus and me, will be the spec, reference implementation (Vervis), implementation in existing forge (Pagure), and an implementation guide. Possibly work on Gitea will get funded too, we don't know yet
https://floss.social/@forgefed/103844664664981845

https://redd.it/ftpeqd
@r_linux

GNU to me T-shirts? Red Hat 4.2

Does anyone have any old *nix centric shirts laying around,
Size large or xl

I left my old r00t t shirt in a Panama city beach motel room like 18 years ago during spring break.. And like a Linux t shirt shld be free or something eheh

I'll PayPal u shipping

https://redd.it/ftwdj1
@r_linux

Why Linux bootloaders wind up being complicated
https://utcc.utoronto.ca/~cks/space/blog/linux/BootloadersComplexity

https://redd.it/ftka8u
@r_linux

Cartesi Scalability on Linux and The CTSI Utility Token

As blockchain projects mature and our understanding of the technology progresses, it becomes ever more clear what challenges impede its mainstream adoption:

* Convenience — developing and using a blockchain application is still an overwhelming task, involving a very difficult learning curve and consequently significantly reducing general interest in the technology;

* Scalability — Blockchains need to perform as well as centralized technology in order to be able to compete with it.

Full Article: https://medium.com/cartesi/cartesi-network-and-ctsi-b9a5e206fedf

https://redd.it/ftykwg
@r_linux

Deprecating support for the Linux kernel
https://guix.gnu.org/blog/2020/deprecating-support-for-the-linux-kernel/

https://redd.it/ftgjoa
@r_linux

Chafa (gnome-terminal image support)
https://hpjansson.org/blag/2020/04/01/chafa-1-4-0-now-with-sixels/

https://redd.it/ftfrqz
@r_linux

Commandline/sysadin course in r/linuxupskillchallange

This month-long challenge for anyone wanting to build Linux sysadmin skills is re-starting again this coming Monday.

"Daily lessons appear in the sub-reddit /r/linuxupskillchallenge - which is also be used for support/discussion. This is a 'rolling' course repeated each month. Does require some serious commitment, but if gaining/growing these skills was one of your New Years Resolutions, then you now have no excuse! "

There was a massive surge in subnoscriptions last month (600!), but the it's difficult to gauge how many really committed, and how many stuck through. Any feedback very welcome.

https://redd.it/fti5bl
@r_linux

Monitor refresh rate.

I have a 75 Hz monitor, yet Linux (I use KDE and the "Kubuntu" distro) doesn't want to really cooperate.
What do I do?
Like, games will show 75 FPS. But I think that's fabricated because my Display Settings in System Settings seem to just default to "Auto" each time I look at them, despite being set to 75 each time I close my settings.

https://redd.it/fu249x
@r_linux

Replace GRUB2 with systemd-boot on Ubuntu 18.04
https://blobfolio.com/2018/06/replace-grub2-with-systemd-boot-on-ubuntu-18-04/

https://redd.it/ftad0h
@r_linux

Anythingsay - A huge repository of cowsay files. Like ponysay/beesay but for everything!
https://charc0al.github.io/cowsay-files/converter/examples.html

https://redd.it/ft2e1e
@r_linux

Freeware, Free Software and the Corona Virus Crisis -- Choose your tools wisely!
https://fsfe.org/news/2020/news-20200330-01.html

https://redd.it/fu4dsq
@r_linux

Updated visual guide to installing Arch Linux

I used VMs to experiment when I wanted to learn more about how Linux works and try to install Arch. However I recently went to throw together a quick system for work and I noticed the guides I had read are now out of date enough that they can't be used start to finish. I found some fixes and stashed them in a markdown file for my own reference, but then figured they might be helpful to someone else so I worked it into a usable guide. It also has some extra "stuff I wish I'd known" type content like adding users, desktop managers and starting services, etc. I've pushed it to [my github](https://github.com/sweeney-th/install-arch-on-vm/blob/master/InstallVM.md) in case it's of use to anyone.

ps - thanks to the mod who helped me find the right way to post this!

https://redd.it/ft1qde
@r_linux

April 2020 Development Update | Krita
https://krita.org/en/item/april-development-update-2/

https://redd.it/fu5kka
@r_linux

Bash beginners guide in ePub format

The *Bash beginners guide* by Machtelt Garrels in ePub 2.0 format.

This is the repo on ourproject \[[repo](https://ourproject.org/scm/browser.php?group_id=2831&extra=bash-bg)\].

This is the link for the direct download \[[download](http://epub-manuals.ourproject.org/blog/bash-beginners-guide/)\].

The opening of issues is welcome.

https://redd.it/fu6ect
@r_linux

Zoom’s Encryption Is “Not Suited for Secrets” and Has Surprising Links To China, Researchers Discover
https://theintercept.com/2020/04/03/zooms-encryption-is-not-suited-for-secrets-and-has-surprising-links-to-china-researchers-discover/

https://redd.it/fu6wjl
@r_linux

Thoughts on Zorin OS check connectivity [opt-out] ?

Zorin OS (15.2 core at least) has got a feature named "Check Connectivity" (similar to Ubuntu/Fedora's) which HTTP/GET "[connectivity.zorin-os.com](https://connectivity.zorin-os.com)" every 30mn. It's opt-out. Of course there are easier ways to check connectivity, like querying NTP servers for instance.

They're doing a home-calling type of heartbeat. Maybe the right name would have been "Inform every 30mn the Zorin company that my computer is online" ?

Although it might not transmit sensitive data (haven't checked the packets), i'm not keen on those opt-out type of stats, kindly presented under an innocent euphemism.

Any thoughts on that ? I haven't found pages discussing that URL anywhere on internet before, which explains my post.

https://redd.it/fu5ru5
@r_linux

Linux filesystem encryption comparison: ext4 vs btrfs vs zfs vs luks vs gocryptfs
https://redd.it/fu7he5
@r_linux

Greg KH AMA coming next week, write your questions now!

Hello everyone,

[Greg Kroah-Hartman](https://en.wikipedia.org/wiki/Greg_Kroah-Hartman) will be doing an AMA next week. We wanted to give you a heads up to write out some questions in advance and we highly encourage checking out the last AMA to see if anything is already answered:

https://www.reddit.com/r/linux/comments/2ny1lz/im_greg_kroahhartman_linux_kernel_developer_ama/

Please note - there will be a new thread posted by /u/gregkh mid next week so please remember to get your questions in there.

I've left this thread unlocked to help people form questions together, but may lock it as needed.

https://redd.it/fu9sv5
@r_linux

Unix/Linux command references (credits: FOSSwire.com)
https://redd.it/fuac5b
@r_linux