Can someone tell me why my code works fine outputting to a terminal but gets "stuck" outputting to a pipe?

Usually I try and parse down the code to a easy-to-understand example, but Im really not sure what is causing this behavior. So....the code is here. It is a prototype for a new way of distributing inputs to several forked workers that is much faster than the standard "pipe --> while read; do..." loop. On my system, distributing 1 million lines (printf '%%s\n' {1..1000000}) to 28 different workers only takes ~0.3 seconds longer than the printf call itself. But that isnt the point of this post, so back on topic...

To see this behavior im trying to fix, run

# source function
source <(curl https://github.com/jkool702/forkrun/blob/main/mySplit.bash)

# this works as expected
printf '%s\n' {1..10000} | mySplit

# this gets stuck
printf '%s\n' {1..100000} | mySplit | wc -l

Ive tried a lot of different ways to fix this without success. best I can tell, mySplit runs fully and cleans up. There are no forked processes left running nor any file denoscriptors left open. mySplit passes all the data it is supposed to pass to the pipe, runs every line of code it is supposed to, and calls its exit trap. And then...it just sits there and does nothing instead of closing and sending an EOF down the pipe.

If whatever you are piping its output to expects an EOF it will wait for it forever. As such, something like wc -l will never give any output. Piping to cat instead will print everything to the screen but then just sit there not closing and giving you back the terminal until you press <crtl> + c.

Any ideas what would cause this?

Thanks in advance.

https://redd.it/15zlyld
@r_bash

Why doesn't echo 1 1 1 | uniq, do what I think it does?

I read uniq could take the stdout and filter out the unique entries.

echo 1<tab>1<tab>1 | uniq

echo 1 1 1 | uniq

both do nothing but print out 1 1 1. What do you suggest I do to put just one 1 using the uniq command so I can understand this command?

I guess I could do this echo -e "1\n1\n1\n" | uniq but I don't understand why the three ones separated by spaces and tabs don't work like I thought.

https://redd.it/15zogw4
@r_bash

How can I output a file into a csv table but merging columns if it has the same value

I have the following example :


1)
--+----------------------------------+-------------+-------------------+
| Node | Pods count | Containers count |
--+----------------------------------+-------------+-------------------+
| www-wwwwwww-wwwwwwww-wwwwwwwww0 | 42 | 53 |
| www-wwwwwww-wwwwwwww-wwwwwwwww1 | 42 | 41 |
| www-wwwwwww-wwwwwwww-wwwwwwwww2 | 40 | 51 |
| www-wwwwwww-wwwwwwww-wwwwwwwww3 | 25 | 41 |
| www-wwwwwww-wwwwwwww-wwwwwwwww4 | 53 | 70 |
--+----------------------------------+-------------+-------------------+

2)
NAMESPACE IMAGE
ns1 image-ns1:v1
ns1 image-ns1:v1
ns2 image-ns2:v1
ns2 image-ns2:v2
ns3 image-ns3:v1
ns4 image-ns4:v1
ns4 image-ns4:v2

I want 2) to look like this :

--+----------------------------+-------------------+
| NAMESPACE | IMAGE |
--+----------------------------+-------------------+
| ns1 | image-ns1:v1 |
| | image-ns1:v1 |
--+----------------------------+-------------------+
| ns2 | image-ns2:v1 |
| | image-ns2:v2 |
--+----------------------------+-------------------+
| ns3 | image-ns3:v1 |
--+----------------------------+-------------------+
| ns4 | image-ns4:v1 |
| | image-ns4:v2 |
--+----------------------------+-------------------+

This is the function I used for the table

function csvtableheader () {
table=$(echo "$1"|sed -e 's/^/,/' -e 's/$/,/' -e 's/,/,| /g' |column -t -s,)
line=$(echo "$table" |head -n1 | sed -e 's/^|/-/g' -e 's/.$//' -e 's/|/+/g')
echo ${line}
echo "$table" | head -n1
echo ${line}
echo "${table}" | tail -n +2
echo ${line}
}

I used it from here @ jabend

How can I achieve this? Thank you

https://redd.it/16045nj
@r_bash

Pipelight - Automation pipelines but easier. (v0.6.14)

Hi guys!

I needed something to glue commands together but with
- more simplicity and verbosity than a bash noscript,
- and much lighter than most cicd software that doesn't fit on my servers.

So I made a tool that has matured over the years and I share it today with you.

https://pipelight.dev/

It heavily goes against some well established standards in the automation/cicd field that I and some others don't approve to a point It may disconcert some of you! 😈

But I am truly convinced this is the way to do things
especially when you have the power of a linux laying in your hands and thus
I am working my ass off through pain and pleasure to bring
a good cicd software to our beloved foss community.

https://redd.it/1604zsl
@r_bash

Issue with GitHub Script

I would like to preface that I am by no means fluent in noscripting on Linux.. I am trying to make a github noscript work to create timelapse from unifi protect camera of a building project. Link to the project is below:

https://github.com/sfeakes/UniFi-Timelapse/blob/master/unifi-timelapse.sh

The savesnap piece works like a charm and I have a cron job running to capture the screenshots. The issue arises when I try to run the createvideo function. It would seem that the noscript creates the $snapTemp directory and creates the files.list inside of that directory, then cd to the $snapTemp and tries to find the files.list in whcih it states it cannot find it.. I am at a loss and grasping at straws so far. I am sorry if this is not the usual or appropriate post to this sub. TIA

ant@antlinux1:~$ ./unifi-timelapse.sh createvideo "1050Build1" today
Creating video of 1050Build1 from today's images
./unifi-timelapse.sh: line 55: ./Timelapse/1050Build1/temp-2023-08-24 21:30/files.list: No such file or directory
ERROR no files found

The only things on the noscript I have changed are the name and IP to the single camera and SNAP_BASE to "./Timelapse". Also I think it is worth mentioning the noscript lives in the Home directory of the logged on user. This is also running on a VM of Ubuntu Desktop 20.04.6 LTS. If additional info is needed, please advise.

https://redd.it/160ml2g
@r_bash

What happens if I change the IFS to IFS=":"?

I tried to experiment with changing the IFS to something else besides the <space><tab><newline> default but it doesn't do want I thought it would.

export IFS=":"
echo random:words:trying:this:thing:out

Shouldn't this echo out random words trying this thing out because we have the : as the new delimiter so all of these should be separate arguments going into echo? Does anyone have a simpler example of changing the IFS?

https://redd.it/160pmgc
@r_bash

PhotoDup noscript help

I have a noscript that I use to remove duplicate images on my Linux Box, It was created many years ago and I have lost my touch and want to adjust it since updating the directory structure on my server.

Basically all I want to add to it is if the file name of the duplicate set has a (1) or -Copy at the end of the file name and they 2 files are in the same directory, remove the one with the longer filename or -copy or (1) at the end of the name.
Currently it will automatically delete images that are in the ZZInboundImages directory that have a duplicate, BUT sometimes if both images are in the ZZInboundImages directory ones will have the -copy or (1) at the end of the file name, but the noscript will not delete that one it will delete the one with the original shorter name. I would like to have it delete the other one.

Please help.
First I run this on my images directory

sudo rm /tmp/Duplist.lst;sudo rm -rf /NAS/Images/.deleted/Def;find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate >> /tmp/Duplist.lst

Then I run this noscript to remove the duplicates that are listed in the Duplist.lst file created above


#!/bin/bash
blue=$(tput setaf 4)
while read -r -u3 md5 path
do
[[ -z $md5 ]] && continue
[[ $prevmd5 != $md5 ]] && prevmd5=$md5 && continue
echo " "
grep $md5 /tmp/Duplist.lst
echo " "
dupepath=./.deleted/DefDupe/${path#./}
printf '\e[1;31mdel: `%s` (%s)\n\e[m' "$path" "$md5"
ls -alth "$path"
ftbd="$path"
autodel="ZZInboundImages"
echo "File to be deleted is: $ftbd"
# ls -alth /NAS/"$path" | awk '{print $5} File:$path' >> /NAS/savedspace.txt
echo " "
if [[ "$ftbd" =~ "$autodel" ]];
then
echo "Auto Deleting file"
# read -p "Delete this file (y/n)? " -n1 ans
# $ans = "y"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] &&
ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
echo "File $path deleted Automatically"
else
echo "Not Deleted Automatically"
read -p "Delete this file (y/n)? " -n1 ans
printf '\nYou entered: %s\n' "$ans"
[ ${ans,,} =~ ^(y|ye|yes)$ ] && ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] && ls -alth /NAS/'$path' | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# echo rm "$path"
fi
done 3< /tmp/Duplist.lst

https://redd.it/160opzt
@r_bash

yabai cycle native fullscreen apps - my first real shell noscript

technically just sh, but r/bash seems to be the only place I can really share this.

But fairly recently I made my first shell noscript that was more than a glorified macro, and as such I decided the best option is to share it with people who are far more knowledgeable about shell noscripting generally so they can tear it to smithereens, and maybe I'll come out of it with some new tips!

Without further ado: [github](https://github.com/Zynh0722/skhd/blob/d05ddeda5df393f3a9e41fab9a78d1c8e2dca0f1/yabai-swap-to-fullscreen.sh)

#!/bin/sh
fullscreen_displays=$(yabai -m query --spaces | jq -c 'map(select(."is-native-fullscreen" == true))')

if [ $(echo $fullscreen_displays | jq 'length') -gt 0 ]; then
# This should be at max length 1. Though this may likely break in a multimonitor environment
visible_fullscreen_displays=$(echo $fullscreen_displays | jq -c 'map(select(."is-visible" == true))')

goto_first_fs=false
if [ $(echo $visible_fullscreen_displays | jq 'length') -gt "0" ]; then
visible_id=$(echo $visible_fullscreen_displays | jq -c '.[0].index')
next_index=$(($visible_id + 1))
if [ $(echo $fullscreen_displays | jq "map(select(.index == $next_index)) | length") -gt "0" ]; then
yabai -m space --focus $next_index
else
goto_first_fs=true
fi
else
goto_first_fs=true
fi

if $goto_first_fs; then
yabai -m space --focus $(echo $fullscreen_displays | jq 'sort_by(.index) | .[0].index')
fi
else
yabai -m space --focus 1
fi

Im interfacing with two other programs, namely yabai, which outputs in json, and jq, to parse and select from that json.

The target of the program is to query yabai for native fullscreen apps with their own dedicated workspaces, if there are none, go to workspace 0, which always exists. Otherwise go to the first native fullscreen app, or the next one if you are already on one.

I had briefly considered using nushell, since it has json parsing built in, but I felt like using this as an excuse to learn proper shell noscripting. As such jq is my friend. While I am generally looking for shell noscripting specific advice or pointers, yabai or jq (or json handling more broadly) specific advice is welcome.

The main things that I think might be improvable is the control structure, I'm like 90% sure I can get the same logic with less nested ifs and without the awful boolean flag. But when I wrote this at 4am I couldn't be bothered to figure it out.

The other thing that *feels* wrong to me is the liberal application of the `$()` syntax. Is there anything wrong with using it? are there any gotchas I should be worrying about? or is it simply the best way to use the values from other commands?

Also, if this post doesn't belong here, let me know and I'll take it down, was just looking for a place to share and potentially seek insight, and this seemed to be as good a place as any


E: Markdown mode isn't the default, so half my formatting got borked, had to fix it

https://redd.it/160wj9s
@r_bash

Another JQ query

I have a JSON data set that looks like this...

{
"Sent Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
} ,
"Received Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}
}

As you can see some of the keys have spaces in them. If I try...

$ jq '.Sent Messages' file.json

I get an error. I have used SED to alter all the keys, but is there a way to use JQ to query this as it stands?

https://redd.it/1616f56
@r_bash

Creating a variable name dynamically?

Okay so I'm about to write a noscript to assemble and edit a video playlist. It'll give each item a number, display it's filename, display it's length (get that from ffprobe) etc. When the playlist is complete it'll be written out to a text file.

What I would like to do is store each bit of information in relevant variables, like this:

videoname1, videopath1, videoduration1

when the user asks to put a second video, a new set of variables should be created:

videoname2, videopath2, videoduration2

Repeat if a third video is added, fourth etc.

I know how to make a loop that counts by incrementing a variable. How could I take that variable and use it to name a new variable?

https://redd.it/161cpe0
@r_bash

Is this book good for someone starting with bash? If not what will you recommend?
https://redd.it/161t9lo
@r_bash

I wrote Q*Bert in pure shell noscript
https://redd.it/162c5ni
@r_bash

me pueden recomendar unos libros para practicar basH?

hola comunidad queria saber si ud me podian ayudar a buscar unos buenos libros ud saben que estudiar siempre paga pa tras

https://redd.it/162ej2v
@r_bash

bash-object: A Bash library to construct and manipulate heterogenous data hierarchies.
https://github.com/bash-bastion/bash-object

https://redd.it/162h8zj
@r_bash

Do I need to export these set of variables?

I'm reading the Linux Command Line and for an example they recommend we add these two lines to the .bashrc file.

export HISTCONTROL=ignoredups
export HISTSIZE=1000

But why would you export these two variables? The child processes (if there any any) don't seem to benefit from us exporting them. I assume we only needed to export variables if they are needed for our noscripts or some other processes but these just set the shell configuration.

https://redd.it/162jnt0
@r_bash

Simple terminal clock

alias clock='while true ; do clear; date | cut -b 23-40 ; sleep 1; done;'
clock

https://redd.it/162qrth
@r_bash

What's the best way not to run a specific command as sudo when the entire noscript was run as sudo?

I wrote the following post installation noscript, which should pull all required things to my OS.

Go installation requires sudo on my system, but go install <package> command doesn't. My goal is to be able to type my password at noscript startup and then go away while everything is being installed. Now, I don't know how to make go packages installed to my $HOME/go/bin. I supposed that I can temporarily cancel sudo effect just for go install <package> command, but I didn't know how. What comes to my mind - is to run go install <package> with a specific user id. But... how do I know what the current user id was? I don't like that go packages are being installed in root directory...

#!/usr/bin/env dash

reset='\e0m'
light_white='\e[97m'
background_red='\e[41m'
background_green='\e[42m'

error() {
local in_message="$1"
echo "$light_white${background_red}error: $in_message$reset">&2
exit 1
}

download_safely() {
local in_url="$1"
local in_output="$2"

if command -v wget 2> /dev/null; then
wget --output-document "$in_output" "$in_url"
else
curl --output "$in_output" "$in_url"
fi
}

install_go() {
echo "$light_white${background_green}Installing 'Go'...$reset"

local archive='go1.21.0.linux-amd64.tar.gz'
download_safely https://go.dev/dl/$archive $archive
rm --recursive --force /usr/local/go
tar --directory /usr/local --extract --gzip --file $archive
export PATH="$PATH:/usr/local/go/bin"

echo "$light_white${background_green}'Go' installation completed...$reset"
}

install_go_cli_tool() {
local in_name="$1"

echo "$light_white${background_green}Installing Go tool '$in_name'...$reset"
go install "github.com/$in_name@latest" # <- How to target to my $HOME folder instead of /root?

echo "$light_white${background_green}Go tool '$in_name' installation completed...$reset"
}

install_go_cli_tools() {
local tools='charmbracelet/gum charmbracelet/pop junegunn/fzf'

for tool in $tools; do
install_go_cli_tool "$tool"
done

export PATH="$PATH:$HOME/go/bin"
}

[ "$(id --user)" -ne 0 && error 'root privileges required to install software'

installgo
installgoclitools

https://redd.it/162vx3d
@r_bash

Naming something dynamically

Okay so a couple of days ago I asked how to name variables dynamically.

specifically, to use one variable containing a number, and then a text string, and combine them together to create the name of a new variable.

For example, if

num=1

and we have the word

fart

I would like to create a new variable using $num and "fart" to name it

1fart

This question was never actually answered, instead, people told me to use arrays instead of variables.

This is annoying, because I'm going to have to name those arrays anyway, because they'll be created by a loop that increments $num.

So my question is still valid and important and actually needs answering, please.

https://redd.it/1635udb
@r_bash

Help with loops

I am quite new to bash, but not programming and I am having a little issue trying to create the correct syntax

I have the code below which should cycle through drives 'md1' then 'md2' etc etc. Now I need to cycle through drives 'md1p1', 'md2p1', 'md3p1' etc. I am not quite sure how to add that on after the $.

for i in {1..23}

do

echo Disk $i

df /dev/md$i

done

Any help would be great.

https://redd.it/163ll5f
@r_bash

Found this list of carefully selected collections of fantastic online Bash noscripting tutorials for beginners! Happy Learning!
https://coursesity.com/blog/bash-noscripting-tutorials/

https://redd.it/163nlf0
@r_bash

Regex Extract and replace within file

Ok, so this is dumb, but I want to edit the .nfo files for each episode of Iron Chef America to add the theme ingredient into the episode noscript.

So each episode has it's own .nfo file. I want to read through the .nfo file, find the plot, strip the Theme (I've formatted all of the plots so that EVERY episodes plot begins with "Theme: ", followed by the theme ingredient, followed by either a line break or a "]".

I spent a couple of hours on this yesterday, but realized that I'm just so lost with grep. I'm usually pretty good with regex, but grep appears to work very different from how I'm used to.

here are two relevent .nfo file excerpts.

Example #1:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles</noscript>

Would become:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles (Chocolate)</noscript>

Example #2:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan</noscript>

Would become:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan (American Kobe beef)</noscript>

https://redd.it/163pvi0
@r_bash