set -e, but show error message and line-number

I usually use set -e in my bash noscripts.

Unfortunately, you only get a non-zero exit value if a command fails, but no warning message.

Is there a simple way to get and error message and the line-number, if
a bash noscript stops because a command returned a non-zero status?

https://redd.it/187fr0c
@r_bash

incremental Backup

Hey

working on a rsync noscript to create incremental backups. I’ve used many different sources around the net including chatgpt to get to where I’m at. I understand approx 75% of whats going on and I’ve run to the point where I’m not grasping why somethings are the way they are... The noscript creates three directories: daily, weekly, monthly. each of these has sub directories ie, (daily)2023-11-30-02:00, 2023-11-29-02:00, etc (weekly)46,47,48. The noscript is supposed to delete backups past a defined date (I’ll use “daily” as example here on out) $DAILY_RETENTION_DAYS. TDirectory name correctly corresponds to the date the directory was actually created but the mtime is the date I first ran the noscript which is nov 9. All the directories are Nov 9 mtime regardless of when they were actually created. I assume this is due to how links operate. Why are the directories mtime off?

!/bin/bash

# debugging
export PS4='${LINENO}: ' #Prints out the line number being executed by debug
set -xv #Turn on debugging




######### Change Directory as Needed - Must Match Source Dir Name ###########

DESTOPT="MyFiles"


# Define source and destination servers and directories
SOURCESERVER="username@192.168.100.55"
SOURCEDIR="/srv/dev-disk-by-uuid-532c2087-63ea-4d82-9194-b2d49e371aa9/$DESTOPT"
DESTSERVER="localhost"
DESTROOT="/media/Storage01"
DESTDIR="$DESTROOT/$DESTOPT"

# Define backup retention periods in days
DAILYRETENTIONDAYS=7
WEEKLYRETENTIONDAYS=30
MONTHLYRETENTIONDAYS=365

# Define current date/time for use in backup directory name
CURRENTDATE=date "+%Y-%m-%d-%H:%M"
CURRENTWEEK=`date "+%V"`
CURRENTMONTH=date "+%m"

# Define logs
if -d $DEST_ROOT/log ; then
echo "log directory exists"
else
mkdir -p $DESTROOT/log
fi

LOGDIR="$DESTROOT/log"
LOGFILE="$DESTOPT"-"$CURRENTDATE.log"

# Define rsync options
RSYNCOPTS="-rlthv --exclude=.Trash* --delete-after --itemize-changes --link-dest=$DESTDIR/current --log-file=$LOGDIR/$LOGFILE"

# Define notification threshold
DAILYTHRESHOLD=20
WEEKLYTHRESHOLD=100
MONTHLYTHRESHOLD=400

# Create backup directories on local destination server
mkdir -p "$DESTDIR/daily/$CURRENTDATE"
mkdir -p "$DESTDIR/weekly/$CURRENTWEEK"
mkdir -p "$DESTDIR/monthly/$CURRENTMONTH"
mkdir -p "$DESTDIR/current"

# Run rsync to perform daily backup
RSYNCOUTPUT=$(rsync $RSYNCOPTS $SOURCESERVER:$SOURCEDIR/ $DESTDIR/daily/$CURRENTDATE/)

# Count number of files changed
DAILYNUMCHANGED=$(echo "$RSYNCOUTPUT" | grep -c "^[\>\<cd]")

# Send email notification if number of files changed exceeds threshold
if [[ $DAILYNUMCHANGED -gt $DAILYTHRESHOLD ]]; then
sendmail -t < /home/user/Mail/backupthresholdexceeded.txt
fi

# Remove daily backups older than retention period
# 11-13-23 commented out as results are usually deleting all directories
#find $DESTDIR/daily -maxdepth 1 -type d -mtime +$DAILYRETENTIONDAYS -exec rm -rf {} \;

# Update symlink to most recent daily backup
rm -rf $DESTDIR/current && ln -s $DESTDIR/daily/$CURRENTDATE $DESTDIR/current


#
# Run rsync to perform weekly backup on Monday
if [ `date +%u` -eq 1 ]; then
rsyncoutput=$(rsync $RSYNCOPTS $SOURCESERVER:$SOURCEDIR/ $DESTDIR/weekly/$CURRENTWEEK/ 2>&1)
# Count number of files changed
WEEKLYNUMCHANGED=$(echo "$rsyncoutput" | grep -E '^sent|^total size' | awk '{print $NF}' | paste -sd+ - | bc)

# Remove weekly backups older than retention period
# find $DESTDIR/weekly -maxdepth 1 -type d -mtime +$WEEKLYRETENTIONDAYS -exec rm -rf {} \;

# Update symlink to most recent weekly backup
rm -f $DESTDIR/weekly/current && ln -s $DESTDIR/weekly/$CURRENTWEEK

$DESTDIR/weekly/current

# Send email notification if number of files changed exceeds threshold
if [[ $WEEKLYNUMCHANGED -gt $NOTIFICATIONTHRESHOLD ]]; then
sendmail -t < /home/user/Mail/backupthresholdexceeded.txt
fi
fi
#
# Run rsync to perform monthly backup on 1st day of month
if `date +%d` -eq 1 ; then
rsyncoutput=$(rsync $RSYNCOPTS $SOURCESERVER:$SOURCEDIR/ $DESTDIR/monthly/$CURRENTMONTH/ 2>&1)

# Count number of files changed
MONTHLYNUMCHANGED=$(echo "$rsyncoutput" | grep -E '^sent|^total size' | awk '{print $NF}' | paste -sd+ - | bc)

# Remove monthly backup older than retention period
# find $DESTDIR/monthly -maxdepth 1 -type d -mtime +$MONTHLYRETENTIONDAYS -exec rm -rf {} \;

# Update symlink to most rescent monthly backup
rm -f $DESTDIR/monthly/current && ln -s $DESTDIR/monthly/$CURRENTMONTH $DESTDIR/monthly/current

# Send email notification if number of files changed exceeds threshold
if [ $MONTHLY_NUM_CHNAGED -gt $NOTIFICATION_THRESHOLD ]; then
sendmail -t < /home/user/Mail/backupthresholdexceeded.txt
fi
fi

# Turn off debugging
set +xv

ls -la

user@backup:/media/Storage01/myfiles/daily$ ls -la
total 72
drwxrwxr-x 18 user user 4096 Nov 30 01:00 .
drwxrwxr-x 5 user user 4096 Nov 30 01:00 ..
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-15-09:25
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-17-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-18-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-19-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-20-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-20-14:07
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-21-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-22-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-23-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-24-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-25-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-26-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-27-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-28-01:00
drwxrwxr-x 13 user user 4096 Nov 9 16:30 2023-11-29-01:00

Notice I have commented out the lines to delete directories past the defined retention time as it deletes all the directories because mtime is Nov 9th. How do I fix this?

https://redd.it/187p2ly
@r_bash

String comparison unexpected results?

I have the following noscript:

if sqlite3 webapp/instance/app.db "SELECT EXISTS(SELECT 1 FROM users WHERE username='admin');" > /dev/null
then
read -p "admin user exists, overwrite? " answer
if "$answer" = "y" ;
then
echo "Removing old admin account..."
sed -i '/ADMINPASSWORD/d' .env
sqlite3 webapp/instance/app.db "DELETE FROM users WHERE username='admin';"
ADMINPWMAYBE=ADMINPASSWORD
else
ADMINPWMAYBE=NULLPASSWORD
fi
fi

I run this and it goes as follows:

admin user exists, overwrite? y



There is no output and it doesn't delete the old ADMIN\PASSWORD

I have added echo "$answer" and it is indeed "y"

so what am I doing wrong?

https://redd.it/187meau
@r_bash

grep to output only needed capturing group
https://shnoscripts.com/grep-to-output-only-needed-capturing-group/

https://redd.it/187tjxd
@r_bash

Calculating with Logs in Bash...

I think BC can do it, or maybe EXPR, but can't find enough documentation or examples even.


I want to calculate this formula and display a result in a noscript I am building...


N = Log2 (S^L)

It's for calculating the password strength of a given password.

I have S and I have L, i need to calculate N. Short of generating Log tables and storing them in an array, I am stuck in finding an elegant solution.

Here are the notes I have received on how it works...

\----

**Password Entropy**

Password entropy is a measure of the randomness or unpredictability of a password. It is often expressed in bits and gives an indication of the strength of a password against brute-force attacks. The formula to calculate password entropy is:

\[ \\text{Entropy} = \\log\2(\\text{Number of Possible Combinations}) \]

Where:

(\\text{Entropy}) is the password entropy in bits.
( \\log_2 ) is the base-2 logarithm.
(\\text{Number of Possible Combinations}) is the total number of possible combinations of the characters used in the password.

The formula takes into account the length of the password and the size of the character set.

Here's a step-by-step guide to calculating password entropy:

Determine the Character Set:


Identify the character set used in the password. This includes uppercase letters, lowercase letters, numbers, and special characters.

Calculate the Size of the Character Set ((S)):


Add up the number of characters in the character set.

Determine the Password Length ((L)):


Identify the length of the password.

Calculate the Number of Possible Combinations ((N)):


Raise the size of the character set ((S)) to the power of the password length ((L)). \[ N = S\^L \]

Calculate the Entropy ((\\text{Entropy})):


Take the base-2 logarithm of the number of possible combinations ((N)). [ \\text{Entropy} = \\log_2(N) \]

This entropy value gives an indication of the strength of the password. Generally, higher entropy values indicate stronger passwords that are more resistant to brute-force attacks. Keep in mind that the actual strength of a password also depends on other factors, such as the effectiveness of the password generation method and the randomness of the chosen characters.

https://redd.it/18804ax
@r_bash

Trying to understand what the period does in a regular expression/wild card?

I'm a little confused about the difference here:

This is true (returns 0 exit code): [[ "aramel" =\~ c* \]\]; echo $?

This is false (returns 1): [[ "aramel" =\~ c.* \]\]; echo $?

(notice the period after the 'c' in the latter.

&#x200B;

Now, both will return a success if instead of writing 'aramel' we writ the word 'caramel'.

So it seems like one of them requires the c and the other does not?

&#x200B;

I've been asking and searching but I am still unsure about what specifically the dot affects.

https://redd.it/185ti39
@r_bash

Basename breaks everything somehow

\#!/bin/bash

baseFolder=\~/gameconverter/psx

&#x200B;

outputFolder=\~/gameconverter/output

rm -rf ${outputFolder}

mkdir ${outputFolder}

&#x200B;

nonSplitted=$(find ${baseFolder} -maxdepth 1 -not -regex ".*(Disc [12\]).*")

disc1=$(find ${baseFolder} -maxdepth 1 -regex ".*(Disc [1\]).*")

disc2=$(find ${baseFolder} -maxdepth 1 -regex ".*(Disc [2\]).*")

&#x200B;

for game in "${nonSplitted[@\]}"; do

gameName=$(basename "${game}")

input="${baseFolder}/${gameName}/${gameName}.cue"

output="${outputFolder}/${gameName}.chd"

chdman createcd -i "${input}" -o "${output}"

done

There are at least 20 psx games in basefolder but somehow, if I use basename or ##*/, it only runs once but with echo ${game} shows all the subfolders, why?

https://redd.it/188m8ch
@r_bash

Getting "read -p" to work within do loop reading files

I'm trying to read a file into my noscript, and prompt for input between each read. When I execute it, the prompt does not occur and only two lines are printed. Removing the "read yn" line means all the files.txt lines do print.

user@local ~/code/bash/interactivefilecopy> source pl.sh
in loop file001.txt
in loop file003.txt

user@local ~/code/bash/interactivefilecopy> cat pl.sh
while read -r linein; do
echo in loop $linein
read -p "whatever" yn
done <files.txt

user@local ~/code/bash/interactivefilecopy> cat files.txt
file001.txt
file002.txt
file003.txt

What am I doing wrong?

Thank you in advance.

https://redd.it/188ltyx
@r_bash

Functions and Libraries

So...... I have started moving all my snips of valuable functions that I use into a bunch of library files which I will make available to anyone who wants. The hardest part is documenting everything so it is actually useful.

My next step, once this step is done, is two make a "bash make" tool, that scans your noscript, scans the libraries that are called using `source` and then builds a single file containing only what is needed. Single file is easier for distribution.

BUT!!!! I have a question: Some of my functions from abc.lib.sh are needed in xyz.lib.sh as well as getting used by mainnoscript.sh. The kicker comes in that if I `source abc.lib.sh` in both the other files, the function loads twice which causes an error.

I can do a test before the source command to see if it is already loaded. I just want to know what is common practice for sequence of events.


I am currently doing:

1. declare statements
2. source statements
3. functions
4. main code

https://redd.it/18ae4xk
@r_bash

Help needed with my noscript

So basically i made simple neofetch, and i want to add ascii on left of the text.

i was thinking about creating something like list of ascii that i can change

github repo with code: https://github.com/Kotuu3/shfetch

https://redd.it/18akcqk
@r_bash

Why learn bash if there's python?

I don't get it. Python is inherently a simpler language(in terms of syntax mainly which obviously makes it easier to learn and stuffs).

Why should I learn bash when I can do everything else in python?

https://redd.it/18amkcn
@r_bash

Backup database with rsync to remote server, how?

My brain is stuck trying to figure out how to do this with rsync in a noscript.

This is what I have so far, and its just very wrong:

docker compose exec -T database pg_dump -U user teslamate | rsync -az 192.168.1.100:/media/user/backup/teslamate/"teslamate.bck.$(date)

any suggestions?

https://redd.it/18aobj0
@r_bash

Essential Bash Keyboard Shortcuts to Speed Up Your Workflow
https://linuxiac.com/essential-bash-keyboard-shortcuts/

https://redd.it/18av3w9
@r_bash

Bash noscript with ffmpeg uses 71 GB memory, how do I fix it?

&#x200B;

https://preview.redd.it/zyznrwd4oe4c1.png?width=1330&format=png&auto=webp&s=cf9d3e9f41de3a4274b6a7183760f70f632fd74f

&#x200B;

* [I am trying to generate a mosaic from 4 input videos using filter complex](https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos)
* The mosaic is generated inside a for loop because start time is different in each iteration on the mosaic
* Then I cut a part of the video as specified by timestamps in array (basically 4 parts) and finally join them
* Then delete the intermediate stuff
* [Here is the bash noscript that does all this](https://pastebin.com/vB9GMMj4)
* When tested on 640x480 the whole thing works perfectly
* When run on 1920x1080, first 2 iterations work well, 3rd iteration gets a KILLED 9 error meaning I guess it ran out of memory
* I am on Apple M1 16GB Sonoma 14 if that helps
* How can I resolve this?

https://redd.it/18b3yg6
@r_bash

Detecting hot keys and inserting text into stdin

Right now i have a program using a simple loop to get input:

while true
do
  read -ep "" string
  echo "Text Entered: $string"
done

I would like to be able to detect a hotkey combination, such as ctrl+1, and from that, insert into the currently edited stdin.
to be clear, i don't want to edit the input after getting it, but instead add text into the input. for example how some programs automatically add a closing bracket when you enter an opening one.

is this possible? where should i look? i have already looked around and can't find anything for the inserting text.

https://redd.it/18bfn2z
@r_bash

Help creating AWK noscript that compares values ?

I need to create a awk noscript that does the following. Finds the specific Max value and Min value for a specific hour found in files that represent a day. So I have a directory with files multiple files like this

day1.csv
day2.csv
day3.csv
day4.csv
day5.csv
day6.csv
day.7.csv

Here is an example of what a few lines in each file would look like

Google,2023-11-30T04:00:00.000Z,38322391063,dev1
Google,2023-11-30T05:00:00.000Z,17091898399,dev1
Google,2023-11-30T06:00:00.000Z,19000746641,dev1
Facebook,2023-11-30T04:00:00.000Z,38322391063,dev1
Facebook,2023-11-30T05:00:00.000Z,17091898399,dev1
Facebook,2023-11-30T06:00:00.000Z,19000746641,dev1

if we take a look at the second field 2023-11-30T04:00:00.000Z , 04 = 4th hour.

So what I am wondering is how would I go about going through all the day files , getting the lines with the specific hours and their first field (Protocol) and 3rd field (usage). So for example if I wanted to see the Max and Min value for the 4th hour for Google over the 7 day files , I would have to find each line in each file that has Google as a first field and contains 04 as the hour in its date field and then store its 3rd field , the usage , and then compare all those lines to see which is the Highest value and which is the lowest. If someone could help me understand how to do this with awk I would really appreciate it.

https://redd.it/18bk8h2
@r_bash

Replacing white spaces with underscores. I originally put this on the Linux help sub but people here might also like it... I may edit this more and made one quick edit just now. This topic comes up a lot for those new to Linux ...

(This is a recent version I just finished. Before anyone points this out, yes, I know there are many ways to do this and you can do it with just one line. I am fully aware that most of those in this sub already know how to deal with the issue this noscript addresses but I felt like sharing it anyway because - why not?)

A few points:

*The help menu is there for a reason and that's why it doesn't have the simple one line and instead informs the user on how the tool works. I think noscripts should be helpful for not only technical people but folks new to Linux too.

*The find command does have flaws here and I was hesitant to specify max depth - but I did this because I felt it was a good balance between specifying no depth an using a simple max depth of only 1

*Wild cards are powerful - and I always have to think if I should use them - but I decided to go for it. Other than that, all I can say is I make the best decisions I can in the limited time I have to write this full knowing they always have flaws and if I obsessed over everything, I would get nothing done. So I move fast but unlike Facebook, I try "not" to break things, haha.

\#!/bin/bash -e

&#x200B;

if [[ "$1" == "-h" \]\]; then

echo "================== Help Menu =================="

echo This noscript requires two arguments

echo "Ex: filerename [ \~/Desktop \] [ \\"file name\\" \]"

echo "Changes 'file name' in \~/Desktop to file_name'"

echo -e "Use quotes or wildcards to reference the file"

echo "[ \~/Desktop \] [ file*name (or) \\"file name\\" \]: \\"file_name\\""

echo "================== filerename ================="

exit 0

fi

&#x200B;

varfind=$(find "$1" -maxdepth 2 -iwholename "*$2*")

&#x200B;

&#x200B;

if [[ -z "$varfind" \]\]; then

echo No results - please enter a valid string

echo Also consider altering your syntax

else

echo "$varfind"

read -r -p "The file(s) above match the string you entered.

Do you want to rename them? Enter \\"yes\\" or \\"no\\": " ans

if [[ "$ans" == "yes" \]\]; then

echo OK - processing files now

logloc=$HOME/tmp/remove-whitespace-rename.log

&#x200B;

echo "$varfind" | while IFS= read -r fileitem

do

if mv -v "$fileitem" "$(echo "${fileitem}" | sed 's/ /_/g')" | tee -a "$logloc"; then

echo "success!"

underscore=$(echo "$fileitem" | grep _ ) &&

echo renamed "$1" "$underscore" &&

\#if [[ -e "${fileitem}" \]\]; then

sleep 1s && wait

exit 0

else

echo "Error: failed to rename file"

echo Check your syntax or review

exit 1

fi

done

else

echo OK - exiting noscript

exit 0

fi

fi

&#x200B;

https://redd.it/18bmgr3
@r_bash

nohup not working?

I have a simple fzf launcher (below) that I want to call from a sway bindsym $mod+d like this:

foot -e fzf-launcher

... ie it pops up a terminal and runs the noscript, the user picks a .desktop file and the noscript runs it with gtk-launcher. When I run the noscript from a normal terminal it works fine, but when I run it as above, it fails to launch anything - actually it starts the process but the process gets SIGHUP as soon as the noscript terminates.

The only way I've got it to work is to add a 'trap "" HUP' just before the gtk-launcher - in other words, the nohup doesn't seem to be working.

Has something changed in nohup or am I misunderstanding something here?

Here's the noscript 'fzf-launcher' - see the 3rd line from the end:

#!/bin/bash
# shellcheck disable=SC2016

locations=( "$HOME/.local/share/applications" "/usr/share/applications" )

#print out the available categories:
grep -r '^Categories' "${locations[@]}" | cut -d= -f2- | tr ';' '\n' | sort -u|column

selected_app=$(
find "${locations[@]}" -name '*.desktop' |
while read -r desktop; do
cat=$( awk -F= '/^Categories/ {print $2}' "$desktop" )
name=${desktop##*/} # filename
name=${name%.*} # basename .desktop
echo "$name '$cat' $desktop"
done |
column -t |
fzf -i --reverse --height 15 --ansi --preview 'echo {} | awk "{print \$3}" | xargs -- grep -iE "name=|exec="' |
awk '{print $3}'
)

if [[ "$selected_app" ]]; then
app="${selected_app##*/}"
# we need this trap otherwise the launched app dies when this noscript
# exits - but only when run as 'foot -e fzf-launcher':
trap '' SIGHUP # !!!! why is this needed? !!!!
nohup gtk-launch "$app" > /dev/null 2>&1 & disown $!
fi

https://redd.it/18c26qs
@r_bash

Cut, dash, behaviour in noscript using stdin creates surprising result

Can anyone help explain below? In short, when the input is cat'd and read as input cut behaves differently, and seems to consider - a delimiter. Why would it do this?

$ cat in

M – Municipality

$ cat in|cut -c 1-4

M –

$ echo 'M – ' | cut -c 1-4

M –

$ cat `mycut.sh`

while read X ; do echo $X | cut -c 1-4 ; done

$ cat in|./mycut4.sh

M

What....? Where did the dash go...? Thanks all!

https://redd.it/18c8so8
@r_bash

Help with selecting a Chrome or Firefox tab/window for typing

Hello,

I would like to put together a noscript where the noscript selects a particular Chrome or Firefox tab and then types in a character every few minutes (to keep the website from timing out). Does anyone know how to do this in Bash in a Wayland DE? Thank you!

https://redd.it/18c8cag
@r_bash