Need to read file and remove content from results.

I have a file that generate all the bad IP for my firewall from several site I have a line to delete any of my IPs but would loved to tell it to remove any ips in a file instead of adding them to my .sh file here is the command below can anyone tell me what to change to tell it to omit whitelistips.txt

curl -sk $IPBAN $FW $MAIL $BLOCKIP $DEB $DES |\

grep -oE '[0-9]{1,3}+[.][0-9]{1,3}+[.][0-9]{1,3}+[.][0-9]{,3}+(/[0-9]{2})?' |\

awk 'NR > 0 {print $1}' | sort -u | grep -v XXX.182.158.* | grep -v 10.10.20.* | grep -v XXX.153.56.212 | grep -v XX.230.162.184 | grep -v XXX.192.189.32 | grep -v XXX.192.189.33 | grep -v >

so I need to replace grep -v XXX.153.56.212 | grep -v XX.230.162.184 | grep -v XXX.192.189.32 | grep -v XXX.192.189.33 | grep -v > with my file name

If you can modify my code because I am not a developer It would be much appreciated.

https://redd.it/15y3q95
@r_bash

{: command not found, shellcheck shows no errors though.

Using shellcheck and no errors, but keeping getting `{: command not found`,

Just using aws cli to create cloudfront invalidation and store the output in variable `invalidation_ouput` Then take that `invalidation_output` and extract the `invalidation_id` and `invalidation_status`.

And bash noscripts asynchronous? Does it wait till the aws cloudfront command to get an output?

My errors are specifically showing on `invalidation_id` line and `invalidation_status` line. The aws cloudfront doesn't seem to be succeeding.

`test.sh`

# 1. Create invalidation
invalidation_output=$(aws cloudfront create-invalidation --region "${REGION}" --distribution-id "${CLOUDFRONT_ID}" --paths '/*')

invalidation_id=$("${invalidation_output}" | jq -r '.Invalidation.Id')
invalidation_status=$("${invalidation_output}" | jq -r '.Invalidation.Status')

https://redd.it/15y8gdm
@r_bash

Assigning tar parameters using variables... help?

Bash noscripting newbie here, I'm trying to write a noscript that generates tar archives of some model data that I've been generating. I have set years for the start and end of each archive. Here's what I'm working with so far:

#!/bin/bash -l

shopt -s extglob

### DEFINE START AND END YEARS HERE
srtYr=$(2451)
endYr=$(2500)

cd /usr/scratch/data/SF_SU_55Ma_init/ocn/
tar -czvf /usr/archive/SF_SU_55Ma_init/ocn/SF_SU_55Ma_init.$srtYr-$endYr.ocn.tar.gz SF_SU_55Ma_init.ocn.h.{$srtYr..$endYr}-??.nc
cd /usr/scratch/data/SF_SU_55Ma_init/atm/
tar -czvf /usr/archive/SF_SU_55Ma_init/atm/SF_SU_55Ma_init.$srtYr-$endYr.ocn.tar.gz SF_SU_55Ma_init.atm.h.{$srtYr..$endYr}-??.nc
cd /usr/scratch/data/SF_SU_55Ma_init/ice/
tar -czvf /usr/archive/SF_SU_55Ma_init/ice/SF_SU_55Ma_init.$srtYr-$endYr.ocn.tar.gz SF_SU_55Ma_init.ice.h.{$srtYr..$endYr}-??.nc

So, I am trying to set the start and end years in both the name of the tar files, and in the brace expansion that defines the range of files to use. This doesn't work, though, as I get these errors:

bash: 2451: command not found
bash: 2500: command not found
tar: SF_SU_55Ma_init.ocn.h.{..}-??.nc: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: SF_SU_55Ma_init.atm.h.{..}-??.nc: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: SF_SU_55Ma_init.ice.h.{..}-??.nc: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors

It looks like I'm not able to call variables within the brace expansion. Also, I'm not sure what those initial bash errors might be. Does anyone know what I'm doing wrong, or what I could do for a workaround? Thanks in advance!

https://redd.it/15yb8v2
@r_bash

Can't pass installation command to function in POSIX noscript

I wrote this noscript to install rust alongside other tools:

#!/bin/sh

set -e

install_dependency() {
id_dependency="$1"
id_denoscription="$2"
id_installation_command="$3"

if ! command -v "$id_dependency"; then
echo "Installing '$id_denoscription' ('$id_dependency' binary)..."
$id_installation_command
echo "Installed '$id_dependency' successfully.
Please make sure '$id_dependency' binary is in your \$PATH."
exit 1
fi
}

install_dependency rustc Rust 'curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh'

The problem appears when I run it:

Installing 'Rust' ('rustc' binary)...
curl: option --proto: is badly used here
curl: try 'curl --help' or 'curl --manual' for more information

I don't understand why: I expect everything work fine, as I didn't quote `$id_installation_command`. While I can run `curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh` directly (and it works) I need this function to install several tools and not to duplicate code several times.

https://redd.it/15yoxle
@r_bash

POSIX-compliant noscript to generate DocOpt-compatible help messages with colors

I've written a noscript to generate help messages with colors. Here is how it looks:

​

https://preview.redd.it/7ghkhdhglsjb1.png?width=1920&format=png&auto=webp&s=1e97f83ef40249060034a208cf8320f732e1dc29

Firstly, I tell my program explanation (Colorful help message generator. here as the first argument), then after double dashes I list all options should be put in my help message in this format: -<short>/--<long>='Some denoscription'. All options and their denoscriptions are automatically aligned to form a table.

You can preview a generated message, or the whole help() function generated via this noscript. Also, it's possible to copy the output via xclip to clipboard when --copy/-c is passed.

In this example I've obtained this Bash code generated:

help() {
echo -e "\e30mColorful help message generator.

\e[36mOptions:
\e[31m--help|-h \e[35mPrint help
\e[31m--version|-v \e[35mPrint version
\e[31m--text-color|-t \e[35mSpecify text color
\e[31m--header-color|-h \e[35mSpecify header color
\e[31m--option-color|-o \e[35mSpecify option color
\e[31m--option-denoscription-color|-O \e[35mSpecify option denoscription color
\e[31m--preview|-p \e[35mWhether to preview generated message
\e[31m--copy|-c \e[35mWhether to copy generated function to clipboard"
}


​

[https://redd.it/15ytyu5
@r_bash

Backslash doesn't seem to remain literal inside double quotes when doesn't follow any special character in dash

Dash [man page](https://manpages.ubuntu.com/manpages/focal/en/man1/sh.1.html) states that:

> The backslash inside double quotes is historically weird, and serves to quote only the following characters:
> $ ` " \ <newline>
> Otherwise it remains literal.

I've tried to execute `echo "1\2"` and expected to see `1\2` while got `1`. In Bash it works as expected. Is it me misunderstanding man page or what? When I see `it remains literal` phrase, I interpret it automatically as `it is being displayed as it is`.

https://redd.it/15z0lm8
@r_bash

Send output of tail to another tail

I am looking to condense the following into one line

I am plumbing logs that are of the format keyword.randomstring.joblog, looking for the last line to have the keyword rc in it.

I currently use

ls -tr $LogDirectory | grep keyword | tail -1 then manually copy that to tail -1 $LogDirectory/FilenameHere | grep rc

Can this be condensed to one line?

Thanks!

https://redd.it/15zdfnf
@r_bash

How Ctrl-V prevents Terminal Driver Sending SIGINT to foreground group?

Hello,

Sorry if this question is a little mercurial and pedantic...or obvious. Trying to fully understand some behaviour rigorously.

If I am running in interactive bash session and I run something like trap report_SIGINT SIGINT to a function that just logs the reception of the SIGINT, then I Ctrl -V Ctrl - C to implement the character literally as in the bash manual I get the Ctrl - C as the manual would suggest literally printed. All good so far according to the manual.

However, this causes a problem I can't quite account for:

Since the terminal driver acts on things before readline library receives stuff from it, why doesn't the driver receive the Ctrl - C from the keyboard and then send the SIGINT to the foreground process group (which bash is in when there is no active foreground childprocess) meaning that bash receives a SIGINT? You can see that bash doesn't receive this from the trap...nothing is outputted to my logging file after Ctrl - V then Ctrl - C. I thought perhaps it turns off the intr value in terminal driver from Ctrl - C but running stty -a TERMINAL on the terminal (from a second terminal) after Ctrl - V shows no change.

Thanks for any and all help!

https://redd.it/15zcpdq
@r_bash

slack noscript

hello not sure if this is the right place for this

i have 0 noscripting experience and my boss asked me to create a noscript that will produce all of our users who are on slack and show them what build version they are on

example user 1 slack build intel
example user 2 slack build silicon

https://redd.it/15zi0is
@r_bash

Can someone tell me why my code works fine outputting to a terminal but gets "stuck" outputting to a pipe?

Usually I try and parse down the code to a easy-to-understand example, but Im really not sure what is causing this behavior. So....the code is here. It is a prototype for a new way of distributing inputs to several forked workers that is much faster than the standard "pipe --> while read; do..." loop. On my system, distributing 1 million lines (printf '%%s\n' {1..1000000}) to 28 different workers only takes ~0.3 seconds longer than the printf call itself. But that isnt the point of this post, so back on topic...

To see this behavior im trying to fix, run

# source function
source <(curl https://github.com/jkool702/forkrun/blob/main/mySplit.bash)

# this works as expected
printf '%s\n' {1..10000} | mySplit

# this gets stuck
printf '%s\n' {1..100000} | mySplit | wc -l

Ive tried a lot of different ways to fix this without success. best I can tell, mySplit runs fully and cleans up. There are no forked processes left running nor any file denoscriptors left open. mySplit passes all the data it is supposed to pass to the pipe, runs every line of code it is supposed to, and calls its exit trap. And then...it just sits there and does nothing instead of closing and sending an EOF down the pipe.

If whatever you are piping its output to expects an EOF it will wait for it forever. As such, something like wc -l will never give any output. Piping to cat instead will print everything to the screen but then just sit there not closing and giving you back the terminal until you press <crtl> + c.

Any ideas what would cause this?

Thanks in advance.

https://redd.it/15zlyld
@r_bash

Why doesn't echo 1 1 1 | uniq, do what I think it does?

I read uniq could take the stdout and filter out the unique entries.

echo 1<tab>1<tab>1 | uniq

echo 1 1 1 | uniq

both do nothing but print out 1 1 1. What do you suggest I do to put just one 1 using the uniq command so I can understand this command?

I guess I could do this echo -e "1\n1\n1\n" | uniq but I don't understand why the three ones separated by spaces and tabs don't work like I thought.

https://redd.it/15zogw4
@r_bash

How can I output a file into a csv table but merging columns if it has the same value

I have the following example :


1)
--+----------------------------------+-------------+-------------------+
| Node | Pods count | Containers count |
--+----------------------------------+-------------+-------------------+
| www-wwwwwww-wwwwwwww-wwwwwwwww0 | 42 | 53 |
| www-wwwwwww-wwwwwwww-wwwwwwwww1 | 42 | 41 |
| www-wwwwwww-wwwwwwww-wwwwwwwww2 | 40 | 51 |
| www-wwwwwww-wwwwwwww-wwwwwwwww3 | 25 | 41 |
| www-wwwwwww-wwwwwwww-wwwwwwwww4 | 53 | 70 |
--+----------------------------------+-------------+-------------------+

2)
NAMESPACE IMAGE
ns1 image-ns1:v1
ns1 image-ns1:v1
ns2 image-ns2:v1
ns2 image-ns2:v2
ns3 image-ns3:v1
ns4 image-ns4:v1
ns4 image-ns4:v2

I want 2) to look like this :

--+----------------------------+-------------------+
| NAMESPACE | IMAGE |
--+----------------------------+-------------------+
| ns1 | image-ns1:v1 |
| | image-ns1:v1 |
--+----------------------------+-------------------+
| ns2 | image-ns2:v1 |
| | image-ns2:v2 |
--+----------------------------+-------------------+
| ns3 | image-ns3:v1 |
--+----------------------------+-------------------+
| ns4 | image-ns4:v1 |
| | image-ns4:v2 |
--+----------------------------+-------------------+

This is the function I used for the table

function csvtableheader () {
table=$(echo "$1"|sed -e 's/^/,/' -e 's/$/,/' -e 's/,/,| /g' |column -t -s,)
line=$(echo "$table" |head -n1 | sed -e 's/^|/-/g' -e 's/.$//' -e 's/|/+/g')
echo ${line}
echo "$table" | head -n1
echo ${line}
echo "${table}" | tail -n +2
echo ${line}
}

I used it from here @ jabend

How can I achieve this? Thank you

https://redd.it/16045nj
@r_bash

Pipelight - Automation pipelines but easier. (v0.6.14)

Hi guys!

I needed something to glue commands together but with
- more simplicity and verbosity than a bash noscript,
- and much lighter than most cicd software that doesn't fit on my servers.

So I made a tool that has matured over the years and I share it today with you.

https://pipelight.dev/

It heavily goes against some well established standards in the automation/cicd field that I and some others don't approve to a point It may disconcert some of you! 😈

But I am truly convinced this is the way to do things
especially when you have the power of a linux laying in your hands and thus
I am working my ass off through pain and pleasure to bring
a good cicd software to our beloved foss community.

https://redd.it/1604zsl
@r_bash

Issue with GitHub Script

I would like to preface that I am by no means fluent in noscripting on Linux.. I am trying to make a github noscript work to create timelapse from unifi protect camera of a building project. Link to the project is below:

https://github.com/sfeakes/UniFi-Timelapse/blob/master/unifi-timelapse.sh

The savesnap piece works like a charm and I have a cron job running to capture the screenshots. The issue arises when I try to run the createvideo function. It would seem that the noscript creates the $snapTemp directory and creates the files.list inside of that directory, then cd to the $snapTemp and tries to find the files.list in whcih it states it cannot find it.. I am at a loss and grasping at straws so far. I am sorry if this is not the usual or appropriate post to this sub. TIA

ant@antlinux1:~$ ./unifi-timelapse.sh createvideo "1050Build1" today
Creating video of 1050Build1 from today's images
./unifi-timelapse.sh: line 55: ./Timelapse/1050Build1/temp-2023-08-24 21:30/files.list: No such file or directory
ERROR no files found

The only things on the noscript I have changed are the name and IP to the single camera and SNAP_BASE to "./Timelapse". Also I think it is worth mentioning the noscript lives in the Home directory of the logged on user. This is also running on a VM of Ubuntu Desktop 20.04.6 LTS. If additional info is needed, please advise.

https://redd.it/160ml2g
@r_bash

What happens if I change the IFS to IFS=":"?

I tried to experiment with changing the IFS to something else besides the <space><tab><newline> default but it doesn't do want I thought it would.

export IFS=":"
echo random:words:trying:this:thing:out

Shouldn't this echo out random words trying this thing out because we have the : as the new delimiter so all of these should be separate arguments going into echo? Does anyone have a simpler example of changing the IFS?

https://redd.it/160pmgc
@r_bash

PhotoDup noscript help

I have a noscript that I use to remove duplicate images on my Linux Box, It was created many years ago and I have lost my touch and want to adjust it since updating the directory structure on my server.

Basically all I want to add to it is if the file name of the duplicate set has a (1) or -Copy at the end of the file name and they 2 files are in the same directory, remove the one with the longer filename or -copy or (1) at the end of the name.
Currently it will automatically delete images that are in the ZZInboundImages directory that have a duplicate, BUT sometimes if both images are in the ZZInboundImages directory ones will have the -copy or (1) at the end of the file name, but the noscript will not delete that one it will delete the one with the original shorter name. I would like to have it delete the other one.

Please help.
First I run this on my images directory

sudo rm /tmp/Duplist.lst;sudo rm -rf /NAS/Images/.deleted/Def;find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate >> /tmp/Duplist.lst

Then I run this noscript to remove the duplicates that are listed in the Duplist.lst file created above


#!/bin/bash
blue=$(tput setaf 4)
while read -r -u3 md5 path
do
[[ -z $md5 ]] && continue
[[ $prevmd5 != $md5 ]] && prevmd5=$md5 && continue
echo " "
grep $md5 /tmp/Duplist.lst
echo " "
dupepath=./.deleted/DefDupe/${path#./}
printf '\e[1;31mdel: `%s` (%s)\n\e[m' "$path" "$md5"
ls -alth "$path"
ftbd="$path"
autodel="ZZInboundImages"
echo "File to be deleted is: $ftbd"
# ls -alth /NAS/"$path" | awk '{print $5} File:$path' >> /NAS/savedspace.txt
echo " "
if [[ "$ftbd" =~ "$autodel" ]];
then
echo "Auto Deleting file"
# read -p "Delete this file (y/n)? " -n1 ans
# $ans = "y"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] &&
ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
echo "File $path deleted Automatically"
else
echo "Not Deleted Automatically"
read -p "Delete this file (y/n)? " -n1 ans
printf '\nYou entered: %s\n' "$ans"
[ ${ans,,} =~ ^(y|ye|yes)$ ] && ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] && ls -alth /NAS/'$path' | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# echo rm "$path"
fi
done 3< /tmp/Duplist.lst

https://redd.it/160opzt
@r_bash

yabai cycle native fullscreen apps - my first real shell noscript

technically just sh, but r/bash seems to be the only place I can really share this.

But fairly recently I made my first shell noscript that was more than a glorified macro, and as such I decided the best option is to share it with people who are far more knowledgeable about shell noscripting generally so they can tear it to smithereens, and maybe I'll come out of it with some new tips!

Without further ado: [github](https://github.com/Zynh0722/skhd/blob/d05ddeda5df393f3a9e41fab9a78d1c8e2dca0f1/yabai-swap-to-fullscreen.sh)

#!/bin/sh
fullscreen_displays=$(yabai -m query --spaces | jq -c 'map(select(."is-native-fullscreen" == true))')

if [ $(echo $fullscreen_displays | jq 'length') -gt 0 ]; then
# This should be at max length 1. Though this may likely break in a multimonitor environment
visible_fullscreen_displays=$(echo $fullscreen_displays | jq -c 'map(select(."is-visible" == true))')

goto_first_fs=false
if [ $(echo $visible_fullscreen_displays | jq 'length') -gt "0" ]; then
visible_id=$(echo $visible_fullscreen_displays | jq -c '.[0].index')
next_index=$(($visible_id + 1))
if [ $(echo $fullscreen_displays | jq "map(select(.index == $next_index)) | length") -gt "0" ]; then
yabai -m space --focus $next_index
else
goto_first_fs=true
fi
else
goto_first_fs=true
fi

if $goto_first_fs; then
yabai -m space --focus $(echo $fullscreen_displays | jq 'sort_by(.index) | .[0].index')
fi
else
yabai -m space --focus 1
fi

Im interfacing with two other programs, namely yabai, which outputs in json, and jq, to parse and select from that json.

The target of the program is to query yabai for native fullscreen apps with their own dedicated workspaces, if there are none, go to workspace 0, which always exists. Otherwise go to the first native fullscreen app, or the next one if you are already on one.

I had briefly considered using nushell, since it has json parsing built in, but I felt like using this as an excuse to learn proper shell noscripting. As such jq is my friend. While I am generally looking for shell noscripting specific advice or pointers, yabai or jq (or json handling more broadly) specific advice is welcome.

The main things that I think might be improvable is the control structure, I'm like 90% sure I can get the same logic with less nested ifs and without the awful boolean flag. But when I wrote this at 4am I couldn't be bothered to figure it out.

The other thing that *feels* wrong to me is the liberal application of the `$()` syntax. Is there anything wrong with using it? are there any gotchas I should be worrying about? or is it simply the best way to use the values from other commands?

Also, if this post doesn't belong here, let me know and I'll take it down, was just looking for a place to share and potentially seek insight, and this seemed to be as good a place as any


E: Markdown mode isn't the default, so half my formatting got borked, had to fix it

https://redd.it/160wj9s
@r_bash

Another JQ query

I have a JSON data set that looks like this...

{
"Sent Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
} ,
"Received Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}
}

As you can see some of the keys have spaces in them. If I try...

$ jq '.Sent Messages' file.json

I get an error. I have used SED to alter all the keys, but is there a way to use JQ to query this as it stands?

https://redd.it/1616f56
@r_bash

Creating a variable name dynamically?

Okay so I'm about to write a noscript to assemble and edit a video playlist. It'll give each item a number, display it's filename, display it's length (get that from ffprobe) etc. When the playlist is complete it'll be written out to a text file.

What I would like to do is store each bit of information in relevant variables, like this:

videoname1, videopath1, videoduration1

when the user asks to put a second video, a new set of variables should be created:

videoname2, videopath2, videoduration2

Repeat if a third video is added, fourth etc.

I know how to make a loop that counts by incrementing a variable. How could I take that variable and use it to name a new variable?

https://redd.it/161cpe0
@r_bash

Is this book good for someone starting with bash? If not what will you recommend?
https://redd.it/161t9lo
@r_bash

I wrote Q*Bert in pure shell noscript
https://redd.it/162c5ni
@r_bash