Can't properly execute Bash variable as options

I have a [noscript](https://gist.github.com/mbierman/6cf22430ca0c2ddb699ac8780ef281ef) that defines a variable that becomes equal to the following. This variable , "args" includes other variables which have to be expanded to complete it.

--name=homebridge --hostname=homebridge --env=HOMEBRIDGE_CONFIG_UI_PORT=8581 --env=PATH=/opt/homebridge/bin:/var/lib/homebridge/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=S6_OVERLAY_VERSION=3.1.1.2--env=S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 --env=S6_KEEP_ENV=1 --env=ENABLE_AVAHI=0 --env=USER=root --env=HOMEBRIDGE_APT_PACKAGE=1 --env=UIX_CUSTOM_PLUGIN_PATH=/var/lib/homebridge/node_modules --env=HOME=/home/homebridge --env=npm_config_prefix=/opt/homebridge --env=npm_config_global_style=true --env=npm_config_audit=false --env=npm_config_fund=false --env=npm_config_update_notifier=false --env=npm_config_loglevel=error --env=HOMEBRIDGE_PKG_VERSION=1.0.33 --volume=/volume1/docker/homebridge:/homebridge:rw --volume=/homebridge --network=host --workdir=/homebridge --restart=always --label='org.opencontainers.image.noscript=Homebridge in Docker' --label='org.opencontainers.image.authors=oznu' --label='org.opencontainers.image.licenses=GPL-3.0' --label='org.opencontainers.image.url=https://github.com/oznu/docker-homebridge' --label='org.opencontainers.image.denoscription=Official Homebridge Docker Image' --log-driver=db --runtime=runc --detach=true -t oznu/homebridge:ubuntu

The variable is defined perfectly and returns what I need and expect.

I then want to execute the arguments in $args, like so:

`sudo docker run "$args"` or `sudo docker run $args`

The problem is I get

sudo docker run '
--name=homebridge --hostname=homebridge --env=HOMEBRIDGE_CONFIG_UI_PORT=8581 --env=PATH=/opt/homebridge/bin:/var/lib/homebridge/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=S6_OVERLAY_VERSION=3.1.1.2--env=S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 --env=S6_KEEP_ENV=1 --env=ENABLE_AVAHI=0 --env=USER=root --env=HOMEBRIDGE_APT_PACKAGE=1 --env=UIX_CUSTOM_PLUGIN_PATH=/var/lib/homebridge/node_modules --env=HOME=/home/homebridge --env=npm_config_prefix=/opt/homebridge --env=npm_config_global_style=true --env=npm_config_audit=false --env=npm_config_fund=false --env=npm_config_update_notifier=false --env=npm_config_loglevel=error --env=HOMEBRIDGE_PKG_VERSION=1.0.33 --volume=/volume1/docker/homebridge:/homebridge:rw --volume=/homebridge --network=host --workdir=/homebridge --restart=always --label='\''org.opencontainers.image.noscript=Homebridge in Docker'\'' --label='\''org.opencontainers.image.authors=oznu'\'' --label='\''org.opencontainers.image.licenses=GPL-3.0'\'' --label='\''org.opencontainers.image.url=https://github.com/oznu/docker-homebridge'\'' --label='\''org.opencontainers.image.denoscription=Official Homebridge Docker Image'\'' --log-driver=db --runtime=runc --detach=true -t oznu/homebridge:ubuntu'

which fails. Obviously I'm not escaping something properly or something like that but I'm not seeing how to solve it.

If I simply echo the entire command rather than executing it, it comes out fine and if executed, works but I want this to work interactively.

sudo docker run --name=homebridge --hostname=homebridge --env=HOMEBRIDGE_CONFIG_UI_PORT=8581

--env=PATH=/opt/homebridge/bin:/var/lib/homebridge/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --env=S6_OVERLAY_VERSION=3.1.1.2 --env=S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 --env=S6_KEEP_ENV=1 --env=ENABLE_AVAHI=0 --env=USER=root --env=HOMEBRIDGE_APT_PACKAGE=1 --env=UIX_CUSTOM_PLUGIN_PATH=/var/lib/homebridge/node_modules --env=HOME=/home/homebridge --env=npm_config_prefix=/opt/homebridge --env=npm_config_global_style=true --env=npm_config_audit=false --env=npm_config_fund=false --env=npm_config_update_notifier=false --env=npm_config_loglevel=error --env=HOMEBRIDGE_PKG_VERSION=1.0.33 --volume=/volume1/docker/homebridge:/homebridge:rw --volume=/homebridge --network=host --workdir=/homebridge --restart=always --label='org.opencontainers.image.noscript=Homebridge in Docker' --label='org.opencontainers.image.authors=oznu' --label='org.opencontainers.image.licenses=GPL-3.0' --label='org.opencontainers.image.url=https://github.com/oznu/docker-homebridge' --label='org.opencontainers.image.denoscription=Official Homebridge Docker Image' --log-driver=db --runtime=runc --detach=true -t oznu/homebridge:ubuntu

https://redd.it/106vxnb
@r_bash

Escape Whitespace in command not working

Why is this not working?

$ touch "this is a test"
$ echo "./this is a test" | sed -e 's/^a-zA-Z0-9,._+@%-//\\&/g'
./this\ is\ a\ test
$ rm "$(echo "./this is a test" | sed -e 's/^a-zA-Z0-9,._+@%-//\\&/g')"

but this does

$ rm ./this\ is\ a\ \ test

https://redd.it/106ulbu
@r_bash

SCREAMINGCASE variables - is it really essential?

I read someone who mentioend that if you don't use SCREAMING\CASE then you have zero chance of accidently overwriting important environment variables. Also I find camelCase a lot nicer to read.

For that reason, I always use camelCase for my own variables, but will continue to use SCREAMING_CASE for environment variables

What's your thoughts?

https://redd.it/107aw18
@r_bash

A function that uses coprocs to parallelize loops in a way that is orders of magnitude faster than forking

LINK TO CODE ON GITHUB

I wrote a function that parallelizes a loop using bash coprocs instead of forking. To run N tasks at a time, the code forks off N coprocs and then pipes data to them, passing a new input each time one finishes its current task.

The syntax is much like the basic usage of xargs -P or parallel. for example: finding the sha256sum of all files under the current path is done via:

source /path/to/forkruncoproc.bash
find ./ -type f | forkruncoproc sha256sum

Why would you want to use this over standard forking / xargs -P / parallel? Well, a couple of reasons:

1. Its a pure-bash solution - no external dependencies are required (with the exception of printf, but its hard to imagine a system that has bash but not printf). NOTE: bash 4.0+ is needed, since this is when coprocs were introduced into bash.

2. It is fast.

It is so fast because forking has a lot of overhead - see man fork(2) for all the stuff that is copied into the child process during a fork syscall. To run a function with M different inputs using N parallel processes:

traditional forking requires `M` fork syscalls, plus `M` calls to `jobs -r | wc -l` to control the number of active current forks.

using forkruncoprocs only requires `N` forked coprocs. Once these are setup, data is passed to/from them without additional forks.

This means that, in particular for loops that have numerous inputs to parallelize over but where each individual input runs very fast, there is a HUGE speed increase. How huge? Heres an example:

On my machine I tested/timed computing the sha256sum of every file under `/lib*/`. (which are symlinks to /usr/lib* on my machine). This comes out to about ~85k files, most of which are very small (and as such computing their sha256sum is very fast for each individual file). Here's how standard forking / paralel / xargs -P / forkruncoproc performed (on an admittedly beefy 28-core i9-7940x-based machine running fedora):

runcontrolledfork() {
mapfile -t inArgs < <(find /usr/lib -type f)
nArgs=${#inArgs[@]}
nProcs=$(which nproc 2>/dev/null 1>/dev/null && nproc || grep -cE '^processor.: ' /proc/cpuinfo)
for nn in "${inArgs@}"; do
sha256sum "$nn" &
(( $(jobs -rp | wc -l) >= ${nProcs} )) && wait -nf
done
}

runparallel() {
find /usr/lib* -type f | xargs -l1 -P$(which nproc 2>/dev/null 1>/dev/null && nproc || grep -cE '^processor.*: ' /proc/cpuinfo) sha256sum
}

runxargs() {
find /usr/lib -type f | parallel -j $(which nproc 2>/dev/null 1>/dev/null && nproc || grep -cE '^processor.: ' /proc/cpuinfo) sha256sum
}

runforkruncoproc() {
find /usr/lib -type f | forkrun_coproc sha256sum
}

time run_controlled_fork

real 28m16.651s
user 20m21.887s
sys 32m26.817s

time run_parallel

real 3m59.612s
user 5m46.280s
sys 9m19.121s


time run_xargs

real 1m24.514s
user 1m30.911s
sys 3m44.941s


time run_forkrun_coproc

real 0m18.589s
user 1m14.336s
sys 2m45.950s

In terms of real-world execution time, `forkrun_coproc` was

\~91x faster than simple forking (just shy of 2 orders of magnitude)
\~13x faster than `parallel`
\~4.5x faster than xargs -P

forkrun_coproc, was also the lightest on cpu cycles used (followed close-ish-ly by xargs -P).

https://redd.it/107f1mw
@r_bash

I give up: WTF is #ifs!

23 years of Bash and today I come across this in code I need to maintain. Very first line is:

\#ifs!/bin/bash

What the hell is #ifs doing before the ! ? Googling stuff like this is pretty futile; can anyone enlighten me?

EDIT: The answer is - this is a typo which someone made and is the reason I had to look at the noscript in the first place! Duh! Git history to the rescue!

https://redd.it/107i7vx
@r_bash

Help with function command format and execution

So I've been compiling a good sized list if bash functions and I've hit a bit of a snag.

I've made some functions for portability between a few systems I run.

So I set up a few firewall functions that work well.

However, I decided to create a function that could serve as a catch-all, so-to-speak.

The individual *-{fwd,ufw,ipfw} functions run properly but when i try to run allowip 8000 it's not running as I had expected.

The general idea is by running $ allowport 8000 the function would check if firewall-cmd is installed then run those commands. If firewall-cmd isn't found check for ufw and run those. If neither are found, check for and run the iptables commands.

Unfortunately, when I run $ allowip 8000 the only output I get is sudo: ufw: command not found. This would be OK as I don't have ufw installed on my main system. Since I use firewalld on my main system (Fedora), the function should just run the first set of commands then exit but it doesn't. I've probably missed something obvious but I'm still learning.

If anyone has an idea on how to get it to run properly, I would very much appreciate it.

At this point, since the individual functions work, I'm just looking to understand more about what it's doing versus what it's supposed to be doing.

&#x200B;

&#x200B;

function allowip-fwd() {

sudo firewall-cmd --zone=home --add-source="$1" --permanent

sudo firewall-cmd --reload

}

&#x200B;

function blockip-fwd() {

sudo firewall-cmd --zone=home --remove-source="$1" --permanent

sudo firewall-cmd --reload

}

&#x200B;

function allowport-fwd() {

sudo firewall-cmd --zone=home --add-port="$1"/tcp --permanent

sudo firewall-cmd --reload

}

&#x200B;

function blockport-fwd() {

sudo firewall-cmd --zone=home --remove-port="$1"/tcp --permanent

sudo firewall-cmd --reload

}

&#x200B;

function allowip-ufw() {

sudo ufw allow from "$1"

}

&#x200B;

function blockip-ufw() {

sudo ufw delete allow from "$1"

}

&#x200B;

function allowport-ufw() {

sudo ufw allow "$1"/tcp

}

&#x200B;

function blockport-ufw() {

sudo ufw delete allow "$1"/tcp

}

&#x200B;

function allowip-ipfw() {

sudo iptables -I INPUT -s "$1" -j ACCEPT

sudo iptables -A OUTPUT -d "$1" -j ACCEPT

}

&#x200B;

function blockip-ipfw() {

sudo iptables -I INPUT -s "$1" -j DROP

sudo iptables -A OUTPUT -d "$1" -j DROP

}

&#x200B;

function allowport-ipfw() {

sudo iptables -I INPUT -p tcp -m tcp --dport "$1" -j ACCEPT

sudo iptables -A OUTPUT -p tcp -m tcp --dport "$1" -j ACCEPT

}

&#x200B;

function blockport-ipfw() {

sudo iptables -I INPUT -p tcp -m tcp --dport "$1" -j REJECT

sudo iptables -A OUTPUT -p tcp -m tcp --dport "$1" -j REJECT

}

&#x200B;

function allowip() {

if ! -x "$(command -v firewall-cmd)" ; then

sudo firewall-cmd --zone=home --add-source="$1" --permanent

sudo firewall-cmd --reload

elif ! -x "$(command -v ufw)" ; then

sudo ufw allow from "$1"

elif ! -x "$(command -v iptables)" ; then

sudo iptables -I INPUT -s "$1" -j ACCEPT

sudo iptables -A OUTPUT -d "$1" -j ACCEPT

else

echo "Something is wrong with your firewall... exiting" >&2

fi

}

&#x200B;

function blockip() {

if ! [ -x "$(command -v firewall-cmd)" \]; then

sudo firewall-cmd --zone=home --remove-source="$1" --permanent

sudo firewall-cmd --reload

elif ! [ -x "$(command -v ufw)" \]; then

sudo ufw delete allow from "$1"

elif ! [ -x "$(command -v iptables)" \]; then

sudo iptables -I

INPUT -s "$1" -j DROP

sudo iptables -A OUTPUT -d "$1" -j DROP

else

echo "Something is wrong with your firewall... exiting" >&2

fi

}

&#x200B;

function allowport() {

if ! [ -x "$(command -v firewall-cmd)" \]; then

sudo firewall-cmd --zone=home --add-port="$1"/tcp --permanent

sudo firewall-cmd --reload

elif ! [ -x "$(command -v ufw)" \]; then

sudo ufw allow "$1"/tcp

elif ! [ -x "$(command -v iptables)" \]; then

sudo iptables -I INPUT -p tcp -m tcp --dport "$1" -j ACCEPT

sudo iptables -A OUTPUT -p tcp -m tcp --dport "$1" -j ACCEPT

else

echo "Something is wrong with your firewall... exiting" >&2

fi

}

&#x200B;

function blockport() { # Function to cover most used firewalls on other distributions

if ! [ -x "$(command -v firewall-cmd)" \]; then # Check if firewall-cmd is installed

sudo firewall-cmd --zone=home --remove-port="$1"/tcp --permanent

sudo firewall-cmd --reload

elif ! [ -x "$(command -v ufw)" \]; then # If no firewall-cmd check for ufw

sudo ufw delete allow "$1"/tcp

elif ! [ -x "$(command -v iptables)" \]; then # If ufw unavailable, resort to iptables

sudo iptables -I INPUT -p tcp -m tcp --dport "$1" -j REJECT

sudo iptables -A OUTPUT -p tcp -m tcp --dport "$1" -j REJECT

else

echo "Something is wrong with your firewall... exiting" >&2 # If no Firewalld, ufw or iptables
exit

fi
}

https://redd.it/107larf
@r_bash

How to read a text file and delete files listed in the file?

I am using a noscript that makes a list of audio files in a directory, then it concatenates the files via ffmpeg into one big file, then I want it to read the text file again and delete all the component files that went into the big file. But I'm having trouble doing it because the noscript keeps breaking the lines of the text file into multiple strings, instead i want it to find the file listed and delete it.

Here is the noscript:

#!/bin/bash

read -p 'Type the noscript of the book here: ' noscript
read -p 'Type the extension of the book here: ' ext
for f in *.$ext; do echo "file '$f'" >> book_order.txt; done
ffmpeg -safe 0 -f concat -i book_order.txt -c:a copy $noscript.$ext && \
IFS='' && \
for f in `cat book_order.txt`; do rm -r `$f`; done && \
rm -f book_order.txt

I set IFS to blank because it was initially separating each line of the file by blank spaces. This makes it set $f to the full `file ./book-part1.mp3` line of the text file. I want the second for loop to have an array of the file commands and then execute the file command to get the file location and then delete it.

I know the IFS thing is wrong because now it doesn't have a delimiter at all. I know by default it uses blank spaces and newlines i thought initially i could use `IFS='\n'` but it just read that as `'n'` and later found that the for loop may just be stripping the new line altogether.

Does anyone know how I could achieve this?

https://redd.it/107mw17
@r_bash

Grep 2 strings to print missing items - what am I doing wrong??

tag1 (file) - just an example, these will be strings instead of numbers

1,2,3,4,5,6,7,8

&#x200B;

tag2 (file)

3

4

7

&#x200B;

GOAL: Print missing items from tag1 that SHOULD BE in tag2

&#x200B;



egrep -v -f tag2 tag1 \# Returns no output - should be 1,2,5,6,8

&#x200B;

I've tried doing this in bash with strings and files as well, getting frustrated :(

TIA

https://redd.it/107jn0o
@r_bash

How to select the latest created directory among multiple directories?

Hi everyone!

A beginner at bash here, still trying to understand stuff. I have a noscript which runs a program in a docker container like

For file in subdirectory

Do

Docker run

/Path/to/directory/subdirectory

Done


However, the folder called "directory" has multiple subdirectories, named on numbers like

102011, 102345, etc


Now I want to always select 102345 (the largest number) in the run because it was created last. How can I add a chunk which specifies: (in all runs of my for loop)?


For file in subdirectory

Do

Docker run

/Path/to/directory/latest_subdirectory

Done

https://redd.it/107qjhd
@r_bash

Deleting folders greater than X days old?

I'm cobbling together a backup noscript to do a database export and copy that and a number of other files to our file server. Thus far, it's working exactly as intended (which is good), but however, since i don't want to overwrite everything on a daily basis, i am leaving every backup in a folder with a time stamped name, eg:

/mnt/smb/backups/daily/2023-01-08
/mnt/smb/backups/daily/2023-01-07
/mnt/smb/backups/daily/2023-01-06
/mnt/smb/backups/daily/2023-01-05

Each backup is 500MB, so while not huge, they will snowball pretty quickly. So now, I would like to start removing backups older than a few days old. Well, I'd like to preserve one backup per month, except I'd also like to keep the last 5 days of backups as well.

I can do the monthly backup just by running

cp *0101 /smb/backups/monthly/

Now, I need to delete the oldest backups.

It appears i have two choices:

find /path/ -mtime +14 -type d | xargs rm -rf;

Or there is package i've just learned about called tmpreaper which looks like it could be pretty simple, but which has a very long scary warning.

Figured i'd give the entire background.

Any thoughts appreciated.

(i'm doing backups this way, because it honestly be easier for my coworkers to do a recovery if they ever need to. I've created a document outlining what files go where, etc)

https://redd.it/107omg4
@r_bash

pkill -u $username vs pkill -9 -u $username, difference and flag explanation?

We have this code so that we close all processes of a user and log them out before changing name on them.

pkill -u $unam 2> /dev/null
pkill -9 -u $unam 2> /dev/null
usermod -l $newnam $unam

We're not 100% about the details, though.

What does "pkill -u jackiie" do, or more specifically, what does the -u flag do? Does it kill all the processes of the effective user ID of "jackiie"?

And "pkill -9 -u jackiie", does this kill the user itself, so that it forces a "log out"? Is the -u flag needed?

https://redd.it/107imam
@r_bash

i=0; ((i++)); echo "${i}${?}" # why 11 not 10?!!

Hi all!

I cannot understand why $? is set to error in the line below

i=0; ((i++)); echo "${i}${?}"

>11

But if I init with 1 there is no error:

i=1; ((i++)); echo "${i}${?}"

>20

Can you explain it?

https://redd.it/1086qqx
@r_bash

Catch error in command

Hi all, Is there a way to use "try" as in Python to capture errors when executing commands?

I’m running the following:

snapshotid=$(head -n 10 /home/egonzalez/ansible/awsoutput.txt)

for snapshot in $snapshotid ; do
aws ec2 modify-snapshot-tier --snapshot-id "${snapshot}" --storage-tier archive
done

The modify-snapshot-tier command will return the error message containing the word "already in progress" my idea is to capture the error of the command and if it contains the word "already in progress" delete id snapshot from the file to follow with other snapshot IDs.


Regards,

https://redd.it/1089cr8
@r_bash

Script can handle spaces in file names when run without sudo, but can't when run with sudo

I have a noscript that finds all pdf files in certain folder and then moves them depending in their date. However, since I move them in the shared folder, I need to run noscript with sudo in order to make permission to move them. The problem is, that with sudo, for some reason, it can't handle spaces in names. So I created a test noscript:

#/!bin/bash
OIFS="$IFS"
IFS=$'\n'
for f in find /mnt/test -maxdepth 1 -type f -name "*.pdf"
do
echo "$f"
done
IFS="$OIFS"

When run it without sudo I get the result:

/mnt/test/I love spaces.pdf

However, when I run it with sudo, I get:

/m
t/test/I love spaces.pdf

And if I run it without OIFC and IFC thing, I get this result both with sudo and without:

/mnt/test/I
love
spaces.pdf

So for some reason, when run with sudo, it removes letter "n"

https://redd.it/1089bp3
@r_bash

I wrote an MVP testing library in Bash to make it as easy as possible to add test suites for commandline-driven tools
https://github.com/pmarreck/tinytestlib

https://redd.it/108ewcp
@r_bash

How to run commannds with dynamic parameters inside a bash scipt?

For an exercise, I need to run the same command hundreds of times, but each time with a different second parameter. Fortunately I have all of them in a file (block.txt). My idea was to read the values and run the command for each value as the argument inside a bash noscript. However, passing the value to the command as a variable doesn't seem to work and it interprets my command as ./aes128-cbc-crackme "00112233445566778800112233445566" "$b". Any alternative methods or ideas of how to overcome this are highly appreciated.

Here's the code:

https://preview.redd.it/y6w9p4o6k9ba1.png?width=567&format=png&auto=webp&v=enabled&s=38343f724cfd443f5f6e04c089c5b910607f3156

https://redd.it/108hx6u
@r_bash

Bash help, terrible with linux, not sure what the name of this would be

Hey all,

I have a noscript I am working on in which there is a variable that indicates the number of variables (in the example there are 4 variables - The DESIRED variables, not total variables in use). I am trying to set variable TEMP to be "VARX" in the for loop, so VAR1 on first pass, VAR2 on 2nd, etc. I am not sure what the syntax is to accomplish that.

&#x200B;

The NUMVARS will be defined in the noscript, not input or anything like that (that is WAAAAY over my head, and not desired). The noscript itself is lengthy, and has numerous calls to use VAR1, VAR2 in for loops, looking to make the whole thing dynamic based on the variables defined at the top, so the NUMBER of variables may be 4 now, but could be 6 later, soo NUMVARS changes to 6 at the other 2 VARS are defined, and it prevents dozens of lines of code changes.

&#x200B;

Would this be a dynamic variable? I am not sure what the name would be for what I am trying to do.

&#x200B;

#!/bin/bash
NUMVARS=4
VAR1=ONE
VAR2=TWO
VAR3=THREE
VAR4=FOUR
#
for ((X=1; X<="$NUMVARS"; X++)); do
#
TEMP="$VAR${X}"
#
echo TEMP $X - $TEMP
done


&#x200B;

&#x200B;

Output from above:

TEMP 1 - 1
TEMP 2 - 2
TEMP 3 - 3
TEMP 4 - 4


&#x200B;

&#x200B;

DESIRED OUTPUT:

TEMP 1 - ONE
TEMP 2 - TWO
TEMP 3 - THREE
TEMP 4 - FOUR

https://redd.it/108vt9o
@r_bash

Modify an IPTable Rule to Include Destination Port

I need to create a noscript that reads several iptable rules and adds a destination port. Is there a way to do this without manually parsing the iptables output as text via awk? Ideally I'm hoping there's a way to ask iptables for individual pieces of data about a certain rule so I can save those into variables and then delete the rule and recreate it with the destination port added.

https://redd.it/108wpai
@r_bash

automated corrupt archive testing help

Hi all,

I've been using the snippet below to create a text file with all the corrupt rar filenames in it.

find . -type f -iname '*.cbr' -exec unrar t {} \; 2>"cbrerrors.txt" >"cbroutput.txt"

I can't help but feel, that for a bash guru it would take about 10 seconds to put together a mod that just moves the corrupt files to another location. I don't suppose I could get an assist please ?

Cheers

https://redd.it/10930zt
@r_bash