PhotoDup noscript help
I have a noscript that I use to remove duplicate images on my Linux Box, It was created many years ago and I have lost my touch and want to adjust it since updating the directory structure on my server.
Basically all I want to add to it is if the file name of the duplicate set has a (1) or -Copy at the end of the file name and they 2 files are in the same directory, remove the one with the longer filename or -copy or (1) at the end of the name.
Currently it will automatically delete images that are in the ZZInboundImages directory that have a duplicate, BUT sometimes if both images are in the ZZInboundImages directory ones will have the -copy or (1) at the end of the file name, but the noscript will not delete that one it will delete the one with the original shorter name. I would like to have it delete the other one.
Please help.
First I run this on my images directory
sudo rm /tmp/Duplist.lst;sudo rm -rf /NAS/Images/.deleted/Def;find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate >> /tmp/Duplist.lst
Then I run this noscript to remove the duplicates that are listed in the Duplist.lst file created above
#!/bin/bash
blue=$(tput setaf 4)
while read -r -u3 md5 path
do
[[ -z $md5 ]] && continue
[[ $prevmd5 != $md5 ]] && prevmd5=$md5 && continue
echo " "
grep $md5 /tmp/Duplist.lst
echo " "
dupepath=./.deleted/DefDupe/${path#./}
printf '\e[1;31mdel: `%s` (%s)\n\e[m' "$path" "$md5"
ls -alth "$path"
ftbd="$path"
autodel="ZZInboundImages"
echo "File to be deleted is: $ftbd"
# ls -alth /NAS/"$path" | awk '{print $5} File:$path' >> /NAS/savedspace.txt
echo " "
if [[ "$ftbd" =~ "$autodel" ]];
then
echo "Auto Deleting file"
# read -p "Delete this file (y/n)? " -n1 ans
# $ans = "y"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] &&
ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
echo "File $path deleted Automatically"
else
echo "Not Deleted Automatically"
read -p "Delete this file (y/n)? " -n1 ans
printf '\nYou entered: %s\n' "$ans"
[ ${ans,,} =~ ^(y|ye|yes)$ ] && ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] && ls -alth /NAS/'$path' | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# echo rm "$path"
fi
done 3< /tmp/Duplist.lst
https://redd.it/160opzt
@r_bash
I have a noscript that I use to remove duplicate images on my Linux Box, It was created many years ago and I have lost my touch and want to adjust it since updating the directory structure on my server.
Basically all I want to add to it is if the file name of the duplicate set has a (1) or -Copy at the end of the file name and they 2 files are in the same directory, remove the one with the longer filename or -copy or (1) at the end of the name.
Currently it will automatically delete images that are in the ZZInboundImages directory that have a duplicate, BUT sometimes if both images are in the ZZInboundImages directory ones will have the -copy or (1) at the end of the file name, but the noscript will not delete that one it will delete the one with the original shorter name. I would like to have it delete the other one.
Please help.
First I run this on my images directory
sudo rm /tmp/Duplist.lst;sudo rm -rf /NAS/Images/.deleted/Def;find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate >> /tmp/Duplist.lst
Then I run this noscript to remove the duplicates that are listed in the Duplist.lst file created above
#!/bin/bash
blue=$(tput setaf 4)
while read -r -u3 md5 path
do
[[ -z $md5 ]] && continue
[[ $prevmd5 != $md5 ]] && prevmd5=$md5 && continue
echo " "
grep $md5 /tmp/Duplist.lst
echo " "
dupepath=./.deleted/DefDupe/${path#./}
printf '\e[1;31mdel: `%s` (%s)\n\e[m' "$path" "$md5"
ls -alth "$path"
ftbd="$path"
autodel="ZZInboundImages"
echo "File to be deleted is: $ftbd"
# ls -alth /NAS/"$path" | awk '{print $5} File:$path' >> /NAS/savedspace.txt
echo " "
if [[ "$ftbd" =~ "$autodel" ]];
then
echo "Auto Deleting file"
# read -p "Delete this file (y/n)? " -n1 ans
# $ans = "y"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] &&
ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
echo "File $path deleted Automatically"
else
echo "Not Deleted Automatically"
read -p "Delete this file (y/n)? " -n1 ans
printf '\nYou entered: %s\n' "$ans"
[ ${ans,,} =~ ^(y|ye|yes)$ ] && ls -alth "$path" | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# [[ ${ans,,} =~ ^(y|ye|yes)$ ]] && ls -alth /NAS/'$path' | awk '{print $5}' >> /NAS/savedspace.txt && mkdir -p "${dupepath%/}" ; mv "$path" "$dupepath"
# echo rm "$path"
fi
done 3< /tmp/Duplist.lst
https://redd.it/160opzt
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
yabai cycle native fullscreen apps - my first real shell noscript
technically just sh, but r/bash seems to be the only place I can really share this.
But fairly recently I made my first shell noscript that was more than a glorified macro, and as such I decided the best option is to share it with people who are far more knowledgeable about shell noscripting generally so they can tear it to smithereens, and maybe I'll come out of it with some new tips!
Without further ado: [github](https://github.com/Zynh0722/skhd/blob/d05ddeda5df393f3a9e41fab9a78d1c8e2dca0f1/yabai-swap-to-fullscreen.sh)
#!/bin/sh
fullscreen_displays=$(yabai -m query --spaces | jq -c 'map(select(."is-native-fullscreen" == true))')
if [ $(echo $fullscreen_displays | jq 'length') -gt 0 ]; then
# This should be at max length 1. Though this may likely break in a multimonitor environment
visible_fullscreen_displays=$(echo $fullscreen_displays | jq -c 'map(select(."is-visible" == true))')
goto_first_fs=false
if [ $(echo $visible_fullscreen_displays | jq 'length') -gt "0" ]; then
visible_id=$(echo $visible_fullscreen_displays | jq -c '.[0].index')
next_index=$(($visible_id + 1))
if [ $(echo $fullscreen_displays | jq "map(select(.index == $next_index)) | length") -gt "0" ]; then
yabai -m space --focus $next_index
else
goto_first_fs=true
fi
else
goto_first_fs=true
fi
if $goto_first_fs; then
yabai -m space --focus $(echo $fullscreen_displays | jq 'sort_by(.index) | .[0].index')
fi
else
yabai -m space --focus 1
fi
Im interfacing with two other programs, namely yabai, which outputs in json, and jq, to parse and select from that json.
The target of the program is to query yabai for native fullscreen apps with their own dedicated workspaces, if there are none, go to workspace 0, which always exists. Otherwise go to the first native fullscreen app, or the next one if you are already on one.
I had briefly considered using nushell, since it has json parsing built in, but I felt like using this as an excuse to learn proper shell noscripting. As such jq is my friend. While I am generally looking for shell noscripting specific advice or pointers, yabai or jq (or json handling more broadly) specific advice is welcome.
The main things that I think might be improvable is the control structure, I'm like 90% sure I can get the same logic with less nested ifs and without the awful boolean flag. But when I wrote this at 4am I couldn't be bothered to figure it out.
The other thing that *feels* wrong to me is the liberal application of the `$()` syntax. Is there anything wrong with using it? are there any gotchas I should be worrying about? or is it simply the best way to use the values from other commands?
Also, if this post doesn't belong here, let me know and I'll take it down, was just looking for a place to share and potentially seek insight, and this seemed to be as good a place as any
E: Markdown mode isn't the default, so half my formatting got borked, had to fix it
https://redd.it/160wj9s
@r_bash
technically just sh, but r/bash seems to be the only place I can really share this.
But fairly recently I made my first shell noscript that was more than a glorified macro, and as such I decided the best option is to share it with people who are far more knowledgeable about shell noscripting generally so they can tear it to smithereens, and maybe I'll come out of it with some new tips!
Without further ado: [github](https://github.com/Zynh0722/skhd/blob/d05ddeda5df393f3a9e41fab9a78d1c8e2dca0f1/yabai-swap-to-fullscreen.sh)
#!/bin/sh
fullscreen_displays=$(yabai -m query --spaces | jq -c 'map(select(."is-native-fullscreen" == true))')
if [ $(echo $fullscreen_displays | jq 'length') -gt 0 ]; then
# This should be at max length 1. Though this may likely break in a multimonitor environment
visible_fullscreen_displays=$(echo $fullscreen_displays | jq -c 'map(select(."is-visible" == true))')
goto_first_fs=false
if [ $(echo $visible_fullscreen_displays | jq 'length') -gt "0" ]; then
visible_id=$(echo $visible_fullscreen_displays | jq -c '.[0].index')
next_index=$(($visible_id + 1))
if [ $(echo $fullscreen_displays | jq "map(select(.index == $next_index)) | length") -gt "0" ]; then
yabai -m space --focus $next_index
else
goto_first_fs=true
fi
else
goto_first_fs=true
fi
if $goto_first_fs; then
yabai -m space --focus $(echo $fullscreen_displays | jq 'sort_by(.index) | .[0].index')
fi
else
yabai -m space --focus 1
fi
Im interfacing with two other programs, namely yabai, which outputs in json, and jq, to parse and select from that json.
The target of the program is to query yabai for native fullscreen apps with their own dedicated workspaces, if there are none, go to workspace 0, which always exists. Otherwise go to the first native fullscreen app, or the next one if you are already on one.
I had briefly considered using nushell, since it has json parsing built in, but I felt like using this as an excuse to learn proper shell noscripting. As such jq is my friend. While I am generally looking for shell noscripting specific advice or pointers, yabai or jq (or json handling more broadly) specific advice is welcome.
The main things that I think might be improvable is the control structure, I'm like 90% sure I can get the same logic with less nested ifs and without the awful boolean flag. But when I wrote this at 4am I couldn't be bothered to figure it out.
The other thing that *feels* wrong to me is the liberal application of the `$()` syntax. Is there anything wrong with using it? are there any gotchas I should be worrying about? or is it simply the best way to use the values from other commands?
Also, if this post doesn't belong here, let me know and I'll take it down, was just looking for a place to share and potentially seek insight, and this seemed to be as good a place as any
E: Markdown mode isn't the default, so half my formatting got borked, had to fix it
https://redd.it/160wj9s
@r_bash
GitHub
skhd/yabai-swap-to-fullscreen.sh at d05ddeda5df393f3a9e41fab9a78d1c8e2dca0f1 · Zynh0722/skhd
Contribute to Zynh0722/skhd development by creating an account on GitHub.
Another JQ query
I have a JSON data set that looks like this...
{
"Sent Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
} ,
"Received Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}
}
As you can see some of the keys have spaces in them. If I try...
$ jq '.Sent Messages' file.json
I get an error. I have used SED to alter all the keys, but is there a way to use JQ to query this as it stands?
https://redd.it/1616f56
@r_bash
I have a JSON data set that looks like this...
{
"Sent Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
} ,
"Received Messages":
{
"To": "Name1",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name2",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}, {
"To": "Name3",
"Created": "DATE TIME UTC",
"Message Type": "Text",
"Text": "Blah blah blah"
}
}
As you can see some of the keys have spaces in them. If I try...
$ jq '.Sent Messages' file.json
I get an error. I have used SED to alter all the keys, but is there a way to use JQ to query this as it stands?
https://redd.it/1616f56
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
Creating a variable name dynamically?
Okay so I'm about to write a noscript to assemble and edit a video playlist. It'll give each item a number, display it's filename, display it's length (get that from ffprobe) etc. When the playlist is complete it'll be written out to a text file.
What I would like to do is store each bit of information in relevant variables, like this:
videoname1, videopath1, videoduration1
when the user asks to put a second video, a new set of variables should be created:
videoname2, videopath2, videoduration2
Repeat if a third video is added, fourth etc.
I know how to make a loop that counts by incrementing a variable. How could I take that variable and use it to name a new variable?
https://redd.it/161cpe0
@r_bash
Okay so I'm about to write a noscript to assemble and edit a video playlist. It'll give each item a number, display it's filename, display it's length (get that from ffprobe) etc. When the playlist is complete it'll be written out to a text file.
What I would like to do is store each bit of information in relevant variables, like this:
videoname1, videopath1, videoduration1
when the user asks to put a second video, a new set of variables should be created:
videoname2, videopath2, videoduration2
Repeat if a third video is added, fourth etc.
I know how to make a loop that counts by incrementing a variable. How could I take that variable and use it to name a new variable?
https://redd.it/161cpe0
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
Is this book good for someone starting with bash? If not what will you recommend?
https://redd.it/161t9lo
@r_bash
https://redd.it/161t9lo
@r_bash
me pueden recomendar unos libros para practicar basH?
hola comunidad queria saber si ud me podian ayudar a buscar unos buenos libros ud saben que estudiar siempre paga pa tras
https://redd.it/162ej2v
@r_bash
hola comunidad queria saber si ud me podian ayudar a buscar unos buenos libros ud saben que estudiar siempre paga pa tras
https://redd.it/162ej2v
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
bash-object: A Bash library to construct and manipulate heterogenous data hierarchies.
https://github.com/bash-bastion/bash-object
https://redd.it/162h8zj
@r_bash
https://github.com/bash-bastion/bash-object
https://redd.it/162h8zj
@r_bash
GitHub
GitHub - bash-bastion/bash-object: Manipulate heterogenous data hierarchies in Bash.
Manipulate heterogenous data hierarchies in Bash. Contribute to bash-bastion/bash-object development by creating an account on GitHub.
Do I need to export these set of variables?
I'm reading the Linux Command Line and for an example they recommend we add these two lines to the .bashrc file.
export HISTCONTROL=ignoredups
export HISTSIZE=1000
But why would you export these two variables? The child processes (if there any any) don't seem to benefit from us exporting them. I assume we only needed to export variables if they are needed for our noscripts or some other processes but these just set the shell configuration.
https://redd.it/162jnt0
@r_bash
I'm reading the Linux Command Line and for an example they recommend we add these two lines to the .bashrc file.
export HISTCONTROL=ignoredups
export HISTSIZE=1000
But why would you export these two variables? The child processes (if there any any) don't seem to benefit from us exporting them. I assume we only needed to export variables if they are needed for our noscripts or some other processes but these just set the shell configuration.
https://redd.it/162jnt0
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
Simple terminal clock
alias clock='while true ; do clear; date | cut -b 23-40 ; sleep 1; done;'
clock
https://redd.it/162qrth
@r_bash
alias clock='while true ; do clear; date | cut -b 23-40 ; sleep 1; done;'
clock
https://redd.it/162qrth
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
What's the best way not to run a specific command as sudo when the entire noscript was run as sudo?
I wrote the following post installation noscript, which should pull all required things to my OS.
Go installation requires sudo on my system, but
#!/usr/bin/env dash
reset='\e0m'
light_white='\e[97m'
background_red='\e[41m'
background_green='\e[42m'
error() {
local in_message="$1"
echo "$light_white${background_red}error: $in_message$reset">&2
exit 1
}
download_safely() {
local in_url="$1"
local in_output="$2"
if command -v wget 2> /dev/null; then
wget --output-document "$in_output" "$in_url"
else
curl --output "$in_output" "$in_url"
fi
}
install_go() {
echo "$light_white${background_green}Installing 'Go'...$reset"
local archive='go1.21.0.linux-amd64.tar.gz'
download_safely https://go.dev/dl/$archive $archive
rm --recursive --force /usr/local/go
tar --directory /usr/local --extract --gzip --file $archive
export PATH="$PATH:/usr/local/go/bin"
echo "$light_white${background_green}'Go' installation completed...$reset"
}
install_go_cli_tool() {
local in_name="$1"
echo "$light_white${background_green}Installing Go tool '$in_name'...$reset"
go install "github.com/$in_name@latest" # <- How to target to my $HOME folder instead of /root?
echo "$light_white${background_green}Go tool '$in_name' installation completed...$reset"
}
install_go_cli_tools() {
local tools='charmbracelet/gum charmbracelet/pop junegunn/fzf'
for tool in $tools; do
install_go_cli_tool "$tool"
done
export PATH="$PATH:$HOME/go/bin"
}
[ "$(id --user)" -ne 0 && error 'root privileges required to install software'
installgo
installgoclitools
https://redd.it/162vx3d
@r_bash
I wrote the following post installation noscript, which should pull all required things to my OS.
Go installation requires sudo on my system, but
go install <package> command doesn't. My goal is to be able to type my password at noscript startup and then go away while everything is being installed. Now, I don't know how to make go packages installed to my $HOME/go/bin. I supposed that I can temporarily cancel sudo effect just for go install <package> command, but I didn't know how. What comes to my mind - is to run go install <package> with a specific user id. But... how do I know what the current user id was? I don't like that go packages are being installed in root directory...#!/usr/bin/env dash
reset='\e0m'
light_white='\e[97m'
background_red='\e[41m'
background_green='\e[42m'
error() {
local in_message="$1"
echo "$light_white${background_red}error: $in_message$reset">&2
exit 1
}
download_safely() {
local in_url="$1"
local in_output="$2"
if command -v wget 2> /dev/null; then
wget --output-document "$in_output" "$in_url"
else
curl --output "$in_output" "$in_url"
fi
}
install_go() {
echo "$light_white${background_green}Installing 'Go'...$reset"
local archive='go1.21.0.linux-amd64.tar.gz'
download_safely https://go.dev/dl/$archive $archive
rm --recursive --force /usr/local/go
tar --directory /usr/local --extract --gzip --file $archive
export PATH="$PATH:/usr/local/go/bin"
echo "$light_white${background_green}'Go' installation completed...$reset"
}
install_go_cli_tool() {
local in_name="$1"
echo "$light_white${background_green}Installing Go tool '$in_name'...$reset"
go install "github.com/$in_name@latest" # <- How to target to my $HOME folder instead of /root?
echo "$light_white${background_green}Go tool '$in_name' installation completed...$reset"
}
install_go_cli_tools() {
local tools='charmbracelet/gum charmbracelet/pop junegunn/fzf'
for tool in $tools; do
install_go_cli_tool "$tool"
done
export PATH="$PATH:$HOME/go/bin"
}
[ "$(id --user)" -ne 0 && error 'root privileges required to install software'
installgo
installgoclitools
https://redd.it/162vx3d
@r_bash
Naming something dynamically
Okay so a couple of days ago I asked how to name variables dynamically.
specifically, to use one variable containing a number, and then a text string, and combine them together to create the name of a new variable.
For example, if
num=1
and we have the word
fart
I would like to create a new variable using $num and "fart" to name it
1fart
This question was never actually answered, instead, people told me to use arrays instead of variables.
This is annoying, because I'm going to have to name those arrays anyway, because they'll be created by a loop that increments $num.
So my question is still valid and important and actually needs answering, please.
https://redd.it/1635udb
@r_bash
Okay so a couple of days ago I asked how to name variables dynamically.
specifically, to use one variable containing a number, and then a text string, and combine them together to create the name of a new variable.
For example, if
num=1
and we have the word
fart
I would like to create a new variable using $num and "fart" to name it
1fart
This question was never actually answered, instead, people told me to use arrays instead of variables.
This is annoying, because I'm going to have to name those arrays anyway, because they'll be created by a loop that increments $num.
So my question is still valid and important and actually needs answering, please.
https://redd.it/1635udb
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
Help with loops
I am quite new to bash, but not programming and I am having a little issue trying to create the correct syntax
I have the code below which should cycle through drives 'md1' then 'md2' etc etc. Now I need to cycle through drives 'md1p1', 'md2p1', 'md3p1' etc. I am not quite sure how to add that on after the $.
for i in {1..23}
do
echo Disk $i
df /dev/md$i
done
Any help would be great.
https://redd.it/163ll5f
@r_bash
I am quite new to bash, but not programming and I am having a little issue trying to create the correct syntax
I have the code below which should cycle through drives 'md1' then 'md2' etc etc. Now I need to cycle through drives 'md1p1', 'md2p1', 'md3p1' etc. I am not quite sure how to add that on after the $.
for i in {1..23}
do
echo Disk $i
df /dev/md$i
done
Any help would be great.
https://redd.it/163ll5f
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community
Found this list of carefully selected collections of fantastic online Bash noscripting tutorials for beginners! Happy Learning!
https://coursesity.com/blog/bash-noscripting-tutorials/
https://redd.it/163nlf0
@r_bash
https://coursesity.com/blog/bash-noscripting-tutorials/
https://redd.it/163nlf0
@r_bash
Coursesity
9 Bash Scripting Tutorials For Beginners in 2023
Hello to all the tech enthusiasts and coding enthusiasts! In today's fast-paced world, efficiency and automation play a vital role. One valuable skill that can greatly contribute to this is Bash noscripting. Are you excited to join us on a journey where we…
Regex Extract and replace within file
Ok, so this is dumb, but I want to edit the .nfo files for each episode of Iron Chef America to add the theme ingredient into the episode noscript.
So each episode has it's own .nfo file. I want to read through the .nfo file, find the plot, strip the Theme (I've formatted all of the plots so that EVERY episodes plot begins with "Theme: ", followed by the theme ingredient, followed by either a line break or a "]".
I spent a couple of hours on this yesterday, but realized that I'm just so lost with grep. I'm usually pretty good with regex, but grep appears to work very different from how I'm used to.
here are two relevent .nfo file excerpts.
Example #1:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles</noscript>
Would become:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles (Chocolate)</noscript>
Example #2:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan</noscript>
Would become:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan (American Kobe beef)</noscript>
https://redd.it/163pvi0
@r_bash
Ok, so this is dumb, but I want to edit the .nfo files for each episode of Iron Chef America to add the theme ingredient into the episode noscript.
So each episode has it's own .nfo file. I want to read through the .nfo file, find the plot, strip the Theme (I've formatted all of the plots so that EVERY episodes plot begins with "Theme: ", followed by the theme ingredient, followed by either a line break or a "]".
I spent a couple of hours on this yesterday, but realized that I'm just so lost with grep. I'm usually pretty good with regex, but grep appears to work very different from how I'm used to.
here are two relevent .nfo file excerpts.
Example #1:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles</noscript>
Would become:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: Chocolate]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-27 06:15:10</dateadded>
<noscript>Flay vs. Bowles (Chocolate)</noscript>
Example #2:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan</noscript>
Would become:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot><!CDATA[Theme: American Kobe beef
Chefs: Iron Chef Bobby Flay takes on Contemporary American specialist Beau MacMillan]></plot>
<outline />
<lockdata>false</lockdata>
<dateadded>2023-08-26 03:34:27</dateadded>
<noscript>Flay vs. MacMillan (American Kobe beef)</noscript>
https://redd.it/163pvi0
@r_bash
Pastebin
<?xml version="1.0" encoding="utf-8" standalone="yes"?><episodedetails> <pl - Pastebin.com
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
timefunc: a function for creating a line-by-line execution time profile for bash code with very minimal overhead
CODE IS HOSTED ON GITHUB HERE
`timefunc` produces a cumulative line-by-line execution time profile for bash noscripts and functions. It is "cumulative" in the sense that if a given line is run multiple times (e.g., because it is in a loop) it will show to total cumulative time spent running that particular line, as well as the command that was run and how many times said command was run.
`timefunc` works by cleverly using the DEBUG trap to keep track of the cumulative time taken per line (as given by `$LINENO`) as the function runs, and then once it is finished running it takes this info and produces a human-usable time-profile report with it. Times are determined by recording start/stop time via the bash builtin `$EPOCHREALTIME` variable, and then the difference is computed and added to a per-line running total.
Note: `line-by-line` refers to lines that you would see after sourcing the function (call if `ff`) and running `declare -f ff`, not to lines in the original function definition. e.g., lines like `echo 1; echo 2` will be broken up into separate lines.
DEPENDENCIES: Bash 5+ (due to use of $EPOCHREALTIME), various GNU coreutils (cat, sed, grep, sort, cut, head, tail, ...)
Note: One could, without much effort, tweak this to use
USAGE
# First, source `timefunc`
source <(curl 'https://raw.githubusercontent.com/jkool702/timeprofile/main/timefunc.bash')
# run timefunc on already-sourced function gg, optionally with arguments for gg
timefunc gg [args]
# run timefunc on not-already-sourced function gg, optionally with arguments for gg, and specify where to source function gg from
timefunc -s "$gg_src" gg [args]
# run timefunc on noscript hh, optionally with arguments for hh
timefunc -S hh [args]
# note: anything passed via STDIN to timefunc will be redirected to the STDIN of the function/noscript being time profiled. e.g.,
echo 'stuff on STDIN' | timefunc gg
See the help text at the top of `timefunc` for more details
EXAMPLE
This is one of the functions I used for testing
gg() {
trap 'echo goodbye' EXIT
trap 'echo -n' DEBUG
echo 'start test'
printf '%s\n' 'loop 1 test'
for kk in $(sleep 1; seq 1 10); do
echo 'hi'
sleep 0.1s
echo $(echo bye; sleep 0.4s)
done
echo 'loop 1 test done'; echo 'loop 2 test'
for kk in {1..10}; do
if (( kk == ( ( kk / 2 ) 2 ) )); then
echo even; sleep 0.2s
elif (( kk == ( ( kk / 3 ) 3 ) )); then echo 'odd3'; sleep 0.3s
fi
echo
done
echo 'loop 2 test done'; echo 'loop 3 test (nested loops in subshell)'
(
for hh in {1..10}; do
for ii in $(seq 1 $gg); do
for jj in $(seq 1 $(( hh + ii ))); do
echo nope >/dev/null
sleep 0.01s
done
done
done
)
echo 'loop 3 test (nested loops in subshell) done';
echo 'loop 4 test (nested functions)'
ff() {
for kk in {1..10}; do
for ll in {1..5}; do
echo $(( $kk $ll ))
sleep 0.02s
done
done
}
ff
echo 'loop 4 test (nested functions) done';
echo 'loop test 5 (nested function in subshell)';
(
ff
)
echo 'loop test 5 (nested function in subshell) done';
echo 'test complete'
}
Running
CODE IS HOSTED ON GITHUB HERE
`timefunc` produces a cumulative line-by-line execution time profile for bash noscripts and functions. It is "cumulative" in the sense that if a given line is run multiple times (e.g., because it is in a loop) it will show to total cumulative time spent running that particular line, as well as the command that was run and how many times said command was run.
`timefunc` works by cleverly using the DEBUG trap to keep track of the cumulative time taken per line (as given by `$LINENO`) as the function runs, and then once it is finished running it takes this info and produces a human-usable time-profile report with it. Times are determined by recording start/stop time via the bash builtin `$EPOCHREALTIME` variable, and then the difference is computed and added to a per-line running total.
Note: `line-by-line` refers to lines that you would see after sourcing the function (call if `ff`) and running `declare -f ff`, not to lines in the original function definition. e.g., lines like `echo 1; echo 2` will be broken up into separate lines.
DEPENDENCIES: Bash 5+ (due to use of $EPOCHREALTIME), various GNU coreutils (cat, sed, grep, sort, cut, head, tail, ...)
Note: One could, without much effort, tweak this to use
date (and not $EPOCHREALTIME) to generate the timestamps and make it compatible with earlier bash versions, but this would come at the cost of significantly increasing the overhead from generating the time profile.USAGE
# First, source `timefunc`
source <(curl 'https://raw.githubusercontent.com/jkool702/timeprofile/main/timefunc.bash')
# run timefunc on already-sourced function gg, optionally with arguments for gg
timefunc gg [args]
# run timefunc on not-already-sourced function gg, optionally with arguments for gg, and specify where to source function gg from
timefunc -s "$gg_src" gg [args]
# run timefunc on noscript hh, optionally with arguments for hh
timefunc -S hh [args]
# note: anything passed via STDIN to timefunc will be redirected to the STDIN of the function/noscript being time profiled. e.g.,
echo 'stuff on STDIN' | timefunc gg
See the help text at the top of `timefunc` for more details
EXAMPLE
This is one of the functions I used for testing
timefunc that contains loops and nested loops, uses subshells and command substitutions, defines functions and calls said functions locally, and defines its own DEBUG and EXIT traps. Many sleep calls are included to check that the execution times and the commands are matching up properly.gg() {
trap 'echo goodbye' EXIT
trap 'echo -n' DEBUG
echo 'start test'
printf '%s\n' 'loop 1 test'
for kk in $(sleep 1; seq 1 10); do
echo 'hi'
sleep 0.1s
echo $(echo bye; sleep 0.4s)
done
echo 'loop 1 test done'; echo 'loop 2 test'
for kk in {1..10}; do
if (( kk == ( ( kk / 2 ) 2 ) )); then
echo even; sleep 0.2s
elif (( kk == ( ( kk / 3 ) 3 ) )); then echo 'odd3'; sleep 0.3s
fi
echo
done
echo 'loop 2 test done'; echo 'loop 3 test (nested loops in subshell)'
(
for hh in {1..10}; do
for ii in $(seq 1 $gg); do
for jj in $(seq 1 $(( hh + ii ))); do
echo nope >/dev/null
sleep 0.01s
done
done
done
)
echo 'loop 3 test (nested loops in subshell) done';
echo 'loop 4 test (nested functions)'
ff() {
for kk in {1..10}; do
for ll in {1..5}; do
echo $(( $kk $ll ))
sleep 0.02s
done
done
}
ff
echo 'loop 4 test (nested functions) done';
echo 'loop test 5 (nested function in subshell)';
(
ff
)
echo 'loop test 5 (nested function in subshell) done';
echo 'test complete'
}
Running
gg (without time profiling) takes just overGitHub
timeprofile/timefunc.bash at main · jkool702/timeprofile
tool for measuring the execution time of all commands called by a shell noscript or shell function - jkool702/timeprofile
16.5 seconds
time gg
real 0m16.575s
user 0m0.355s
sys 0m0.200s
Running
timefunc gg >/dev/null
1.1: 0.001325 sec { (2x) "${@}"; (1x) ${noscriptFlag}; (2x) :; (1x) ff; }
1.3: 0.002744 sec { (1x) echo 'start test'; (10x) for kk in {1..10}; }
1.4: 0.002519 sec { (10x) :; (1x) printf '%s\n' 'loop 1 test'; }
1.6: 1.030974 sec { (50x) for ll in {1..5}; }
1.7: 0.015178 sec { (50x) echo $(( $kk $ll )); (10x) for kk in $(sleep 1; seq 1 10); }
1.8: 1.117169 sec { (10x) echo 'hi'; (50x) sleep 0.02s; }
1.9: 1.022775 sec { (10x) sleep 0.1s; }
1.10: 4.180431 sec { (10x) echo $(echo bye; sleep 0.4s); }
1.12: 0.000239 sec { (1x) echo 'loop 1 test done'; }
1.13: 0.000216 sec { (1x) echo 'loop 2 test (nested loops)'; }
1.16: 0.001946 sec { (10x) for kk in {1..10}; }
1.19: 0.010337 sec { (50x) for jj in {1..5}; }
1.20: 0.010285 sec { (50x) (( ( kk + jj ) == ( ( ( kk + jj )/ 2 ) 2 ) )); }
1.21: 0.007913 sec { (25x) echo even; }
1.22: 5.058182 sec { (25x) sleep 0.2s; }
1.24: 0.005089 sec { (25x) (( ( kk + jj ) == ( ( ( kk + jj ) / 3 ) 3 ) )); }
1.25: 0.001776 sec { (8x) echo 'odd3'; }
1.26: 2.418957 sec { (8x) sleep 0.3s; }
1.29: 0.012242 sec { (50x) echo; }
1.32: 0.000239 sec { (1x) echo 'loop 2 test (nested loops) done'; }
1.33: 0.000222 sec { (1x) echo 'loop 3 test (nested loops in subshell)'; }
1.48: 0.000253 sec { (1x) echo 'loop 3 test (nested loops in subshell) done'; }
1.49: 0.000234 sec { (1x) echo 'loop 4 test (nested functions)'; }
1.50: 0.000203 sec { (1x) ff; }
1.51: 0.000246 sec { (1x) echo 'loop 4 test (nested functions) done'; }
1.52: 0.000220 sec { (1x) echo 'loop test 5 (nested function in subshell)'; }
1.55: 0.000252 sec { (1x) echo 'loop test 5 (nested function in subshell) done'; }
1.56: 0.000326 sec { (1x) echo 'test complete'; }
TOTAL TIME TAKEN: 17.335308 seconds
SUBSHELL COMMANDS
2.1: 0.002224 sec { (1x) :); (10x) echo $(echo bye; sleep 0.4s)); (1x) ff; }
2.3: 0.006185 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); (9x) for kk in {1..10}; (1x) for kk in {1..10}); }
2.6: 1.015416 sec { (49x) for ll in {1..5}; (1x) for ll in {1..5}); (1x) seq 1 10); (1x) sleep 1; }
2.7: 0.014421 sec { (49x) echo $(( $kk $ll )); (1x) echo $(( $kk $ll ))); }
2.8: 1.119910 sec { (49x) sleep 0.02s; (1x) sleep 0.02s); }
2.9: 4.026090 sec { (10x) echo bye; (10x) sleep 0.4s); }
2.10: 0.001833 sec { (10x) echo $(echo bye; sleep 0.4s)); }
2.36: 0.002011 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); }
2.39: 0.002030 sec { (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); }
2.42: 0.013214 sec { (64x) for jj in $(seq 1 $(( hh + ii ))); (1x) for jj in $(seq 1 $(( hh + ii )))); }
2.43: 0.015589 sec { (64x) echo nope > /dev/null; (1x) echo nope > /dev/null); }
2.44: 0.789913 sec { (64x) sleep 0.01s; (1x) sleep 0.01s); }
2.54: 0.000217 sec { (1x) ff); }
time profile for gg has been saved to /root/timeprofile.gg
The overall execution time increased by ~760 ms to 17.335 seconds, an increase of ~4.6%. Its worth noting that this does not include the ~1 second needed at the end after the function has finished running to actually generate the time profile from the raw timing data. Its also, however, worth noting that the "total execution time" in the time profile report includes the time taken by the DEBUG trap to keep track of the cumulative time taken for each line, but the per-line totals do not (the DEBUG trap basically stops the timer by setting tStop, then figures out the time difference, then starts a
time gg
real 0m16.575s
user 0m0.355s
sys 0m0.200s
Running
timefunc gg gives the following output:timefunc gg >/dev/null
1.1: 0.001325 sec { (2x) "${@}"; (1x) ${noscriptFlag}; (2x) :; (1x) ff; }
1.3: 0.002744 sec { (1x) echo 'start test'; (10x) for kk in {1..10}; }
1.4: 0.002519 sec { (10x) :; (1x) printf '%s\n' 'loop 1 test'; }
1.6: 1.030974 sec { (50x) for ll in {1..5}; }
1.7: 0.015178 sec { (50x) echo $(( $kk $ll )); (10x) for kk in $(sleep 1; seq 1 10); }
1.8: 1.117169 sec { (10x) echo 'hi'; (50x) sleep 0.02s; }
1.9: 1.022775 sec { (10x) sleep 0.1s; }
1.10: 4.180431 sec { (10x) echo $(echo bye; sleep 0.4s); }
1.12: 0.000239 sec { (1x) echo 'loop 1 test done'; }
1.13: 0.000216 sec { (1x) echo 'loop 2 test (nested loops)'; }
1.16: 0.001946 sec { (10x) for kk in {1..10}; }
1.19: 0.010337 sec { (50x) for jj in {1..5}; }
1.20: 0.010285 sec { (50x) (( ( kk + jj ) == ( ( ( kk + jj )/ 2 ) 2 ) )); }
1.21: 0.007913 sec { (25x) echo even; }
1.22: 5.058182 sec { (25x) sleep 0.2s; }
1.24: 0.005089 sec { (25x) (( ( kk + jj ) == ( ( ( kk + jj ) / 3 ) 3 ) )); }
1.25: 0.001776 sec { (8x) echo 'odd3'; }
1.26: 2.418957 sec { (8x) sleep 0.3s; }
1.29: 0.012242 sec { (50x) echo; }
1.32: 0.000239 sec { (1x) echo 'loop 2 test (nested loops) done'; }
1.33: 0.000222 sec { (1x) echo 'loop 3 test (nested loops in subshell)'; }
1.48: 0.000253 sec { (1x) echo 'loop 3 test (nested loops in subshell) done'; }
1.49: 0.000234 sec { (1x) echo 'loop 4 test (nested functions)'; }
1.50: 0.000203 sec { (1x) ff; }
1.51: 0.000246 sec { (1x) echo 'loop 4 test (nested functions) done'; }
1.52: 0.000220 sec { (1x) echo 'loop test 5 (nested function in subshell)'; }
1.55: 0.000252 sec { (1x) echo 'loop test 5 (nested function in subshell) done'; }
1.56: 0.000326 sec { (1x) echo 'test complete'; }
TOTAL TIME TAKEN: 17.335308 seconds
SUBSHELL COMMANDS
2.1: 0.002224 sec { (1x) :); (10x) echo $(echo bye; sleep 0.4s)); (1x) ff; }
2.3: 0.006185 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); (9x) for kk in {1..10}; (1x) for kk in {1..10}); }
2.6: 1.015416 sec { (49x) for ll in {1..5}; (1x) for ll in {1..5}); (1x) seq 1 10); (1x) sleep 1; }
2.7: 0.014421 sec { (49x) echo $(( $kk $ll )); (1x) echo $(( $kk $ll ))); }
2.8: 1.119910 sec { (49x) sleep 0.02s; (1x) sleep 0.02s); }
2.9: 4.026090 sec { (10x) echo bye; (10x) sleep 0.4s); }
2.10: 0.001833 sec { (10x) echo $(echo bye; sleep 0.4s)); }
2.36: 0.002011 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); }
2.39: 0.002030 sec { (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); }
2.42: 0.013214 sec { (64x) for jj in $(seq 1 $(( hh + ii ))); (1x) for jj in $(seq 1 $(( hh + ii )))); }
2.43: 0.015589 sec { (64x) echo nope > /dev/null; (1x) echo nope > /dev/null); }
2.44: 0.789913 sec { (64x) sleep 0.01s; (1x) sleep 0.01s); }
2.54: 0.000217 sec { (1x) ff); }
time profile for gg has been saved to /root/timeprofile.gg
The overall execution time increased by ~760 ms to 17.335 seconds, an increase of ~4.6%. Its worth noting that this does not include the ~1 second needed at the end after the function has finished running to actually generate the time profile from the raw timing data. Its also, however, worth noting that the "total execution time" in the time profile report includes the time taken by the DEBUG trap to keep track of the cumulative time taken for each line, but the per-line totals do not (the DEBUG trap basically stops the timer by setting tStop, then figures out the time difference, then starts a
timefunc: a function for creating a line-by-line execution time profile for bash code with very minimal overhead
CODE IS HOSTED ON GITHUB [HERE](https://github.com/jkool702/timeprofile/blob/main/timefunc.bash)
***
`timefunc` produces a cumulative line-by-line execution time profile for bash noscripts and functions. It is "cumulative" in the sense that if a given line is run multiple times (e.g., because it is in a loop) it will show to total cumulative time spent running that particular line, as well as the command that was run and how many times said command was run.
`timefunc` works by cleverly using the DEBUG trap to keep track of the cumulative time taken per line (as given by `$LINENO`) as the function runs, and then once it is finished running it takes this info and produces a human-usable time-profile report with it. Times are determined by recording start/stop time via the bash builtin `$EPOCHREALTIME` variable, and then the difference is computed and added to a per-line running total.
Note: `line-by-line` refers to lines that you would see after sourcing the function (call if `ff`) and running `declare -f ff`, not to lines in the original function definition. e.g., lines like `echo 1; echo 2` will be broken up into separate lines.
***
DEPENDENCIES: Bash 5+ (due to use of $EPOCHREALTIME), various GNU coreutils (cat, sed, grep, sort, cut, head, tail, ...)
Note: One could, without much effort, tweak this to use `date` (and not `$EPOCHREALTIME`) to generate the timestamps and make it compatible with earlier bash versions, but this would come at the cost of significantly increasing the overhead from generating the time profile.
***
USAGE
# First, source `timefunc`
source <(curl 'https://raw.githubusercontent.com/jkool702/timeprofile/main/timefunc.bash')
# run timefunc on already-sourced function gg, optionally with arguments for gg
timefunc gg [args]
# run timefunc on not-already-sourced function gg, optionally with arguments for gg, and specify where to source function gg from
timefunc -s "$gg_src" gg [args]
# run timefunc on noscript hh, optionally with arguments for hh
timefunc -S hh [args]
# note: anything passed via STDIN to timefunc will be redirected to the STDIN of the function/noscript being time profiled. e.g.,
echo 'stuff on STDIN' | timefunc gg
See the help text at the top of `timefunc` for more details
***
EXAMPLE
This is one of the functions I used for testing `timefunc` that contains loops and nested loops, uses subshells and command substitutions, defines functions and calls said functions locally, and defines its own DEBUG and EXIT traps. Many sleep calls are included to check that the execution times and the commands are matching up properly.
gg() {
trap 'echo goodbye' EXIT
trap 'echo -n' DEBUG
echo 'start test'
printf '%s\n' 'loop 1 test'
for kk in $(sleep 1; seq 1 10); do
echo 'hi'
sleep 0.1s
echo $(echo bye; sleep 0.4s)
done
echo 'loop 1 test done'; echo 'loop 2 test'
for kk in {1..10}; do
if (( kk == ( ( kk / 2 ) * 2 ) )); then
echo even; sleep 0.2s
elif (( kk == ( ( kk / 3 ) * 3 ) )); then echo 'odd3'; sleep 0.3s
fi
echo
done
echo 'loop 2 test done'; echo 'loop 3 test (nested loops in subshell)'
(
for hh in {1..10}; do
for ii in $(seq 1 $gg); do
for jj in $(seq 1 $(( hh + ii ))); do
echo nope >/dev/null
sleep 0.01s
done
done
done
)
echo 'loop 3 test (nested loops in subshell) done';
echo 'loop 4 test (nested functions)'
ff() {
for kk in {1..10}; do
for ll in {1..5}; do
echo $(( $kk ** $ll ))
sleep 0.02s
done
done
}
ff
echo 'loop 4 test (nested functions) done';
echo 'loop test 5 (nested function in subshell)';
(
ff
)
echo 'loop test 5 (nested function in subshell) done';
echo 'test complete'
}
Running `gg` (without time profiling) takes just over
CODE IS HOSTED ON GITHUB [HERE](https://github.com/jkool702/timeprofile/blob/main/timefunc.bash)
***
`timefunc` produces a cumulative line-by-line execution time profile for bash noscripts and functions. It is "cumulative" in the sense that if a given line is run multiple times (e.g., because it is in a loop) it will show to total cumulative time spent running that particular line, as well as the command that was run and how many times said command was run.
`timefunc` works by cleverly using the DEBUG trap to keep track of the cumulative time taken per line (as given by `$LINENO`) as the function runs, and then once it is finished running it takes this info and produces a human-usable time-profile report with it. Times are determined by recording start/stop time via the bash builtin `$EPOCHREALTIME` variable, and then the difference is computed and added to a per-line running total.
Note: `line-by-line` refers to lines that you would see after sourcing the function (call if `ff`) and running `declare -f ff`, not to lines in the original function definition. e.g., lines like `echo 1; echo 2` will be broken up into separate lines.
***
DEPENDENCIES: Bash 5+ (due to use of $EPOCHREALTIME), various GNU coreutils (cat, sed, grep, sort, cut, head, tail, ...)
Note: One could, without much effort, tweak this to use `date` (and not `$EPOCHREALTIME`) to generate the timestamps and make it compatible with earlier bash versions, but this would come at the cost of significantly increasing the overhead from generating the time profile.
***
USAGE
# First, source `timefunc`
source <(curl 'https://raw.githubusercontent.com/jkool702/timeprofile/main/timefunc.bash')
# run timefunc on already-sourced function gg, optionally with arguments for gg
timefunc gg [args]
# run timefunc on not-already-sourced function gg, optionally with arguments for gg, and specify where to source function gg from
timefunc -s "$gg_src" gg [args]
# run timefunc on noscript hh, optionally with arguments for hh
timefunc -S hh [args]
# note: anything passed via STDIN to timefunc will be redirected to the STDIN of the function/noscript being time profiled. e.g.,
echo 'stuff on STDIN' | timefunc gg
See the help text at the top of `timefunc` for more details
***
EXAMPLE
This is one of the functions I used for testing `timefunc` that contains loops and nested loops, uses subshells and command substitutions, defines functions and calls said functions locally, and defines its own DEBUG and EXIT traps. Many sleep calls are included to check that the execution times and the commands are matching up properly.
gg() {
trap 'echo goodbye' EXIT
trap 'echo -n' DEBUG
echo 'start test'
printf '%s\n' 'loop 1 test'
for kk in $(sleep 1; seq 1 10); do
echo 'hi'
sleep 0.1s
echo $(echo bye; sleep 0.4s)
done
echo 'loop 1 test done'; echo 'loop 2 test'
for kk in {1..10}; do
if (( kk == ( ( kk / 2 ) * 2 ) )); then
echo even; sleep 0.2s
elif (( kk == ( ( kk / 3 ) * 3 ) )); then echo 'odd3'; sleep 0.3s
fi
echo
done
echo 'loop 2 test done'; echo 'loop 3 test (nested loops in subshell)'
(
for hh in {1..10}; do
for ii in $(seq 1 $gg); do
for jj in $(seq 1 $(( hh + ii ))); do
echo nope >/dev/null
sleep 0.01s
done
done
done
)
echo 'loop 3 test (nested loops in subshell) done';
echo 'loop 4 test (nested functions)'
ff() {
for kk in {1..10}; do
for ll in {1..5}; do
echo $(( $kk ** $ll ))
sleep 0.02s
done
done
}
ff
echo 'loop 4 test (nested functions) done';
echo 'loop test 5 (nested function in subshell)';
(
ff
)
echo 'loop test 5 (nested function in subshell) done';
echo 'test complete'
}
Running `gg` (without time profiling) takes just over
GitHub
timeprofile/timefunc.bash at main · jkool702/timeprofile
tool for measuring the execution time of all commands called by a shell noscript or shell function - jkool702/timeprofile
16.5 seconds
time gg
real 0m16.575s
user 0m0.355s
sys 0m0.200s
Running `timefunc gg` gives the following output:
timefunc gg >/dev/null
1.1: 0.001325 sec { (2x) "${@}"; (1x) ${noscriptFlag}; (2x) :; (1x) ff; }
1.3: 0.002744 sec { (1x) echo 'start test'; (10x) for kk in {1..10}; }
1.4: 0.002519 sec { (10x) :; (1x) printf '%s\n' 'loop 1 test'; }
1.6: 1.030974 sec { (50x) for ll in {1..5}; }
1.7: 0.015178 sec { (50x) echo $(( $kk ** $ll )); (10x) for kk in $(sleep 1; seq 1 10); }
1.8: 1.117169 sec { (10x) echo 'hi'; (50x) sleep 0.02s; }
1.9: 1.022775 sec { (10x) sleep 0.1s; }
1.10: 4.180431 sec { (10x) echo $(echo bye; sleep 0.4s); }
1.12: 0.000239 sec { (1x) echo 'loop 1 test done'; }
1.13: 0.000216 sec { (1x) echo 'loop 2 test (nested loops)'; }
1.16: 0.001946 sec { (10x) for kk in {1..10}; }
1.19: 0.010337 sec { (50x) for jj in {1..5}; }
1.20: 0.010285 sec { (50x) (( ( kk + jj ) == ( ( ( kk + jj )/ 2 ) * 2 ) )); }
1.21: 0.007913 sec { (25x) echo even; }
1.22: 5.058182 sec { (25x) sleep 0.2s; }
1.24: 0.005089 sec { (25x) (( ( kk + jj ) == ( ( ( kk + jj ) / 3 ) * 3 ) )); }
1.25: 0.001776 sec { (8x) echo 'odd3'; }
1.26: 2.418957 sec { (8x) sleep 0.3s; }
1.29: 0.012242 sec { (50x) echo; }
1.32: 0.000239 sec { (1x) echo 'loop 2 test (nested loops) done'; }
1.33: 0.000222 sec { (1x) echo 'loop 3 test (nested loops in subshell)'; }
1.48: 0.000253 sec { (1x) echo 'loop 3 test (nested loops in subshell) done'; }
1.49: 0.000234 sec { (1x) echo 'loop 4 test (nested functions)'; }
1.50: 0.000203 sec { (1x) ff; }
1.51: 0.000246 sec { (1x) echo 'loop 4 test (nested functions) done'; }
1.52: 0.000220 sec { (1x) echo 'loop test 5 (nested function in subshell)'; }
1.55: 0.000252 sec { (1x) echo 'loop test 5 (nested function in subshell) done'; }
1.56: 0.000326 sec { (1x) echo 'test complete'; }
TOTAL TIME TAKEN: 17.335308 seconds
SUBSHELL COMMANDS
2.1: 0.002224 sec { (1x) :); (10x) echo $(echo bye; sleep 0.4s)); (1x) ff; }
2.3: 0.006185 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); (9x) for kk in {1..10}; (1x) for kk in {1..10}); }
2.6: 1.015416 sec { (49x) for ll in {1..5}; (1x) for ll in {1..5}); (1x) seq 1 10); (1x) sleep 1; }
2.7: 0.014421 sec { (49x) echo $(( $kk ** $ll )); (1x) echo $(( $kk ** $ll ))); }
2.8: 1.119910 sec { (49x) sleep 0.02s; (1x) sleep 0.02s); }
2.9: 4.026090 sec { (10x) echo bye; (10x) sleep 0.4s); }
2.10: 0.001833 sec { (10x) echo $(echo bye; sleep 0.4s)); }
2.36: 0.002011 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); }
2.39: 0.002030 sec { (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); }
2.42: 0.013214 sec { (64x) for jj in $(seq 1 $(( hh + ii ))); (1x) for jj in $(seq 1 $(( hh + ii )))); }
2.43: 0.015589 sec { (64x) echo nope > /dev/null; (1x) echo nope > /dev/null); }
2.44: 0.789913 sec { (64x) sleep 0.01s; (1x) sleep 0.01s); }
2.54: 0.000217 sec { (1x) ff); }
time profile for gg has been saved to /root/timeprofile.gg
The overall execution time increased by ~760 ms to 17.335 seconds, an increase of ~4.6%. Its worth noting that this does not include the ~1 second needed at the end after the function has finished running to actually generate the time profile from the raw timing data. Its also, however, worth noting that the "total execution time" in the time profile report includes the time taken by the DEBUG trap to keep track of the cumulative time taken for each line, but the per-line totals do not (the DEBUG trap basically stops the timer by setting tStop, then figures out the time difference, then starts a
time gg
real 0m16.575s
user 0m0.355s
sys 0m0.200s
Running `timefunc gg` gives the following output:
timefunc gg >/dev/null
1.1: 0.001325 sec { (2x) "${@}"; (1x) ${noscriptFlag}; (2x) :; (1x) ff; }
1.3: 0.002744 sec { (1x) echo 'start test'; (10x) for kk in {1..10}; }
1.4: 0.002519 sec { (10x) :; (1x) printf '%s\n' 'loop 1 test'; }
1.6: 1.030974 sec { (50x) for ll in {1..5}; }
1.7: 0.015178 sec { (50x) echo $(( $kk ** $ll )); (10x) for kk in $(sleep 1; seq 1 10); }
1.8: 1.117169 sec { (10x) echo 'hi'; (50x) sleep 0.02s; }
1.9: 1.022775 sec { (10x) sleep 0.1s; }
1.10: 4.180431 sec { (10x) echo $(echo bye; sleep 0.4s); }
1.12: 0.000239 sec { (1x) echo 'loop 1 test done'; }
1.13: 0.000216 sec { (1x) echo 'loop 2 test (nested loops)'; }
1.16: 0.001946 sec { (10x) for kk in {1..10}; }
1.19: 0.010337 sec { (50x) for jj in {1..5}; }
1.20: 0.010285 sec { (50x) (( ( kk + jj ) == ( ( ( kk + jj )/ 2 ) * 2 ) )); }
1.21: 0.007913 sec { (25x) echo even; }
1.22: 5.058182 sec { (25x) sleep 0.2s; }
1.24: 0.005089 sec { (25x) (( ( kk + jj ) == ( ( ( kk + jj ) / 3 ) * 3 ) )); }
1.25: 0.001776 sec { (8x) echo 'odd3'; }
1.26: 2.418957 sec { (8x) sleep 0.3s; }
1.29: 0.012242 sec { (50x) echo; }
1.32: 0.000239 sec { (1x) echo 'loop 2 test (nested loops) done'; }
1.33: 0.000222 sec { (1x) echo 'loop 3 test (nested loops in subshell)'; }
1.48: 0.000253 sec { (1x) echo 'loop 3 test (nested loops in subshell) done'; }
1.49: 0.000234 sec { (1x) echo 'loop 4 test (nested functions)'; }
1.50: 0.000203 sec { (1x) ff; }
1.51: 0.000246 sec { (1x) echo 'loop 4 test (nested functions) done'; }
1.52: 0.000220 sec { (1x) echo 'loop test 5 (nested function in subshell)'; }
1.55: 0.000252 sec { (1x) echo 'loop test 5 (nested function in subshell) done'; }
1.56: 0.000326 sec { (1x) echo 'test complete'; }
TOTAL TIME TAKEN: 17.335308 seconds
SUBSHELL COMMANDS
2.1: 0.002224 sec { (1x) :); (10x) echo $(echo bye; sleep 0.4s)); (1x) ff; }
2.3: 0.006185 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); (9x) for kk in {1..10}; (1x) for kk in {1..10}); }
2.6: 1.015416 sec { (49x) for ll in {1..5}; (1x) for ll in {1..5}); (1x) seq 1 10); (1x) sleep 1; }
2.7: 0.014421 sec { (49x) echo $(( $kk ** $ll )); (1x) echo $(( $kk ** $ll ))); }
2.8: 1.119910 sec { (49x) sleep 0.02s; (1x) sleep 0.02s); }
2.9: 4.026090 sec { (10x) echo bye; (10x) sleep 0.4s); }
2.10: 0.001833 sec { (10x) echo $(echo bye; sleep 0.4s)); }
2.36: 0.002011 sec { (9x) for hh in {1..10}; (1x) for hh in {1..10}); }
2.39: 0.002030 sec { (9x) for ii in $(seq 1 $gg); (1x) for ii in $(seq 1 $gg)); }
2.42: 0.013214 sec { (64x) for jj in $(seq 1 $(( hh + ii ))); (1x) for jj in $(seq 1 $(( hh + ii )))); }
2.43: 0.015589 sec { (64x) echo nope > /dev/null; (1x) echo nope > /dev/null); }
2.44: 0.789913 sec { (64x) sleep 0.01s; (1x) sleep 0.01s); }
2.54: 0.000217 sec { (1x) ff); }
time profile for gg has been saved to /root/timeprofile.gg
The overall execution time increased by ~760 ms to 17.335 seconds, an increase of ~4.6%. Its worth noting that this does not include the ~1 second needed at the end after the function has finished running to actually generate the time profile from the raw timing data. Its also, however, worth noting that the "total execution time" in the time profile report includes the time taken by the DEBUG trap to keep track of the cumulative time taken for each line, but the per-line totals do not (the DEBUG trap basically stops the timer by setting tStop, then figures out the time difference, then starts a
new timer by setting tStart). This means that the per-line execution times shown should contain, at most, 1-2% overhead. i.e., they are *very* close to what the actual execution times are when running the function without `timefunc`.
***
KNOWN ISSUES
To have a command record its time it must trigger the DEBUG trap, which doesn't always happen (e.g., for commands run in a subshell via `( ... )`). Commands run in subshells are grouped based on what `${BASH_SUBSHELL}.${LINENO}` evaluates to when the exit trap for said subshell is called, which doesn't always separate out commands as logically as one might like. Thus, the time profile may not be as "line-by-line" separated for stuff run in subshells, and stuff run in subshells may be missing entirely from the main shell's time profile (lines starting with 1.x) and/or have its times listed in both the main shell and subshell time profiles (e.g., as with `echo $(sleep 1, echo hi)`.
That all said, I believe that all commands are accounted for somewhere in the time profile, and that for a given line in the time profile the listed time is *almost* always accurate for the set of commands listed. Its just that the cumulative time taken running a particular line (that was run in a subshell) may be merged with other subshell lines.
If anyone has ideas on how I might be able to better separate these let me know, but when I tried I could basically either do what I did or have every single subshell command on its own separate line and not combine any of them, making the generated time profile much longer and less useful (IMO).
https://redd.it/1641o96
@r_bash
***
KNOWN ISSUES
To have a command record its time it must trigger the DEBUG trap, which doesn't always happen (e.g., for commands run in a subshell via `( ... )`). Commands run in subshells are grouped based on what `${BASH_SUBSHELL}.${LINENO}` evaluates to when the exit trap for said subshell is called, which doesn't always separate out commands as logically as one might like. Thus, the time profile may not be as "line-by-line" separated for stuff run in subshells, and stuff run in subshells may be missing entirely from the main shell's time profile (lines starting with 1.x) and/or have its times listed in both the main shell and subshell time profiles (e.g., as with `echo $(sleep 1, echo hi)`.
That all said, I believe that all commands are accounted for somewhere in the time profile, and that for a given line in the time profile the listed time is *almost* always accurate for the set of commands listed. Its just that the cumulative time taken running a particular line (that was run in a subshell) may be merged with other subshell lines.
If anyone has ideas on how I might be able to better separate these let me know, but when I tried I could basically either do what I did or have every single subshell command on its own separate line and not combine any of them, making the generated time profile much longer and less useful (IMO).
https://redd.it/1641o96
@r_bash
Reddit
From the bash community on Reddit: timefunc: a function for creating a line-by-line execution time profile for bash code with very…
Explore this post and more from the bash community
Hijacking the new-tab command in gnome-terminal
I'm interesting in souping up my gnome-terminal (or any other terminal emulator that works with vte) so that, for example, if I'm inside a docker container and I use the new-tab command, the new tab opens up in the same docker container at the same working directory. I have control of the docker container, so I can put whatever I want into its bashrc to support that. As a starting point, I've been looking into how gnome-terminal knows to open new tabs at the same working directory when I'm _not_ in a container. I think I have a decent handle on that.
PROMPT_COMMAND gets set to a function __vte_prompt_command, which calls a function __vte_osc7. This second function uses escape characters to send a message to the terminal telling it the current working directory. At least, that seems to be the case.
The good news is that, in principle, this means one could send a working directory even from within a container, and it would still go back to the terminal (I have confirmed that this works). The problem, of course, is that when I try to open up a new tab, it would try to open it at that working directory on the host machine, not in the container. Assuming that location doesn't exist on the host machine, it will default to the home directory, and all of that will happen before it reaches my host machine's bashrc, where I could tell it to enter the docker container at a specified working directory.
So I'm wondering if there's any way to hijack the command that gets run when the user requests a new tab for gnome-terminal, so that I could access the working directory inside the docker container and use that information before it gets lost.
That's a rather long-winded question, I realize. Anyone have any thoughts? Thanks.
https://redd.it/16457zh
@r_bash
I'm interesting in souping up my gnome-terminal (or any other terminal emulator that works with vte) so that, for example, if I'm inside a docker container and I use the new-tab command, the new tab opens up in the same docker container at the same working directory. I have control of the docker container, so I can put whatever I want into its bashrc to support that. As a starting point, I've been looking into how gnome-terminal knows to open new tabs at the same working directory when I'm _not_ in a container. I think I have a decent handle on that.
PROMPT_COMMAND gets set to a function __vte_prompt_command, which calls a function __vte_osc7. This second function uses escape characters to send a message to the terminal telling it the current working directory. At least, that seems to be the case.
The good news is that, in principle, this means one could send a working directory even from within a container, and it would still go back to the terminal (I have confirmed that this works). The problem, of course, is that when I try to open up a new tab, it would try to open it at that working directory on the host machine, not in the container. Assuming that location doesn't exist on the host machine, it will default to the home directory, and all of that will happen before it reaches my host machine's bashrc, where I could tell it to enter the docker container at a specified working directory.
So I'm wondering if there's any way to hijack the command that gets run when the user requests a new tab for gnome-terminal, so that I could access the working directory inside the docker container and use that information before it gets lost.
That's a rather long-winded question, I realize. Anyone have any thoughts? Thanks.
https://redd.it/16457zh
@r_bash
Reddit
From the bash community on Reddit
Explore this post and more from the bash community