Capturing history of bash on exit

I have been using something the TRAP command in my .bashrc file for over 10 years to get a history of all commands run in the shell.

I have this line in my .bashrc:

trap $HOME/bin/save_history.sh EXIT

$ cat bin/save_history.sh

\#!/bin/bash

history > \~/.hist/`date +%d%b%y_%H%M_``echo $RANDOM`

This will capture the history into a file with the date, time and a random number so multiple files will not over write each other. A typical file name looks like

28Jun23_1047_25998

I started doing this so I had a log of whatever I did with git.

I am using git-bash on a win11 system. I have used this in my .bashrc on both linux systems and recently it has stopped working. The file is created, but it is empty. If I run that command from a command line in a live shell, it works. I get a file with 500 lines, the last 500 commands run in that shell.

Is there some reason that history output doesn't go to the file when run from a noscript?

​

https://redd.it/14lgdq3
@r_bash

Script to make a new python virtual env and install packages into it with pip

I wrote a quick noscript to make a new python virtual env and install packages into it with pip, but only if $2 $3 $4 are provided. The noscript accepts a -h flag to display a usage message (also if run without arguments). Here is the noscript

I tried to make it compatible with the Bourne shell and used the #!/bin/sh. On Arch and derivatives, sh is symlinked to bash, so it doesn't really affect my use case, but might help others.

## A few problems I want to ask about:

It works fine, but I couldn't:

find a way to `cd` into the new env directory and source the `env/bin/activate` from the noscript before exiting the noscript because of the noscript running in its own subshell, and I couldn't find any smart fixes for that. Although sourcing the python env from inside the noscript works as far as getting the env working and installing packages with pip inside the new env.
Another problem I faced was that using echo "$PWD"/"$1" would return "$PWD"/"$1"/"$1" like /home/wolandark/env/env which didn't even exist, and that confused me a bit.

Any feedback || tips are highly appreciated and thank you for your time.
Cheers!

## Edit:
After the tips I recieved under the post, I updated the noscript. Here it is.

https://redd.it/14kx6l8
@r_bash

How do I copy, merge and append multiple files taking them as user inputs?

So firstly what is the difference between these and how do I do them separately?

https://redd.it/14l6jmf
@r_bash

How should I pass "i" in the record object array in the for loop?

for i in {0..$IDs}
do
CSs=$(echo "$jsonData" | jq -r '.result.records.ChangeSetNamec')
echo "$CSs"
done

==============Below is the JSON===========

{
"status": 0,
"result": {
"records":
{
"attributes": {
"type": "agf__ADM_Work__c",
"url": "/services/data/v58.0/sobjects/agf__ADM_Work__c/a1F26000002EhXREA0"
},
"Id": "a1F26000002EhXREA0",
"Change_Set_Name__c": "W-001730"
},
{
"attributes": {
"type": "agf__ADM_Work__c",
"url": "/services/data/v58.0/sobjects/agf__ADM_Work__c/a1F26000002EwRyEAK"
},
"Id": "a1F26000002EwRyEAK",
"Change_Set_Name__c": "W-002024"
},
{
"attributes": {
"type": "agf__ADM_Work__c",
"url": "/services/data/v58.0/sobjects/agf__ADM_Work__c/a1F26000002ETgGEAW"
},
"Id": "a1F26000002ETgGEAW",
"Change_Set_Name__c": "W-001444"
}
,
"totalSize": 7,
"done": true
},
"warnings":
"The \"force data soql query\" command has been deprecated. Use \"data query\" instead.",
"The \"-u\" flag has been deprecated. Use \"--target-org | -o\" instead."

}



So I want to take the change_set of the respective ID and later then use the ID again in some SOQL query.

If i keep the statement as is i.e. "CSs=$(echo "$jsonData" | jq -r '.result.records[\].Change_Set_Name__c')" it returns all the IDs in one go, I want to iterate them one by one to perform some tasks on each ID and their changesets.

I tried using $i but seems like the '' are restricting it., it takes it as a past of the logic. Or is there any other way to iterate through the json

https://redd.it/14kehp3
@r_bash

Why choose these to test if the shell is broken

#!/bin/sh
#
# FFmpeg configure noscript
#
# Copyright (c) 2000-2002 Fabrice Bellard
# Copyright (c) 2005-2008 Diego Biurrun
# Copyright (c) 2005-2008 Mans Rullgard
#

# Prevent locale nonsense from breaking basic text processing.
LCALL=C
export LCALL

# make sure we are running under a compatible shell
# try to make this part work with most shells

tryexec(){
echo "Trying shell $1"
type "$1" > /dev/null 2>&1 && exec "$@"
}

unset foo
(: ${foo%%bar}) 2> /dev/null
E1="$?"

(: ${foo?}) 2> /dev/null
E2="$?"

if test "$E1" != 0 || test "$E2" = 0; then
echo "Broken shell detected. Trying alternatives."
export FFCONFEXEC
if test "0$FFCONFEXEC" -lt 1; then
FFCONFEXEC=1
tryexec bash "$0" "$@"
fi
if test "0$FFCONFEXEC" -lt 2; then
FFCONFEXEC=2
tryexec ksh "$0" "$@"
fi
if test "0$FFCONFEXEC" -lt 3; then
FFCONFEXEC=3
tryexec /usr/xpg4/bin/sh "$0" "$@"
fi
echo "No compatible shell noscript interpreter found."
echo "This configure noscript requires a POSIX-compatible shell"
echo "such as bash or ksh."
echo "THIS IS NOT A BUG IN FFMPEG, DO NOT REPORT IT AS SUCH."
echo "Instead, install a working POSIX-compatible shell."
echo "Disabling this configure test will create a broken FFmpeg."
if test "$BASHVERSION" = '2.04.0(1)-release'; then
echo "This bash version ($BASHVERSION) is broken on your platform."
echo "Upgrade to a later version if available."
fi
exit 1
fi


This is a code snippet from FFmpeg's comfigure scirpt.

It uses these to test whether the current shell is broken.

unset foo
(: ${foo%%bar}) 2> /dev/null
E1="$?"

(: ${foo?}) 2> /dev/null
E2="$?"

I understand the behaviour of these code. My question is why do they choose these? What is special about them?

https://redd.it/14k9rpz
@r_bash

Splitting file by unique/not-unique

I have a couple of files that look something like this:

chr1 94160888 94161133 GAGTGTCTCTCAGAGTGA 91
chr1 94160888 94161133 GTCACTGTGTCTCAGTGT 84
chr1 94160888 94161133 CACAGTCACACTCTCTCA 56
chr1 94160888 94161133 CACTCACTGAGTGTGTGA 51
chr1 94160888 94161133 GACTGAGACACTCACACT 22
chr1 94121054 94121299 CAGTCAGACAGTCAGACT 91
chr1 94121054 94121299 GACAGTGTCTCTCACTCT 76
chr1 94121054 94121299 GACAGTGTCTCTCACTCT 62


And I need to identify them by unique strings in column 4. I was hoping to split these into two separate files. However, the closest thing I've found to at least find unique values is sort -k4,4 -u, which would be fine, but it still prints the first instance of a duplicated string, which I do not want. Is there a way to print only the lines where a column 4 string appears once, and is there a way to print lines where one appears more than once?

​

EDIT: for file simplicity

https://redd.it/14jooud
@r_bash

Can't use wildcard * // only interpreted literally

Trying to write a back up noscript to deploy to my devices. I would like for it to prune logs as it prunes backups but am having trouble matching filenames with a wildcard.

[Whole noscript](https://hastebin.com/share/ociriwikep.bash) and [set -x output](https://hastebin.com/share/sogepemuwa.bash) available but, I 'll just paste the offending function below:

`# Remove logs of pruned backups`

`tidyLogs() {`
`mapfile -t removeList < <(grep -w "^Pruning archive" "$logFile" | awk '{ print $4 }' )`
`for ((i=0; i<${#removeList[@]}; i++)); do`
`# Slice off seconds from timestamp for matching filenames easily`
`# rm "${logFile%/"${logFile##*/}"}"/"${removeList[i]%:*}"*`

`# holder1="${logFile%/"${logFile##*/}"}"/"${removeList[i]%:*}"*`
`# eval holder2=$holder1# rm $holder2`

`set -- "${logFile%/"${logFile##*/}"}"/"${removeList[i]%:*}"*`
`rm "$*"`
`done`
`}`

It's most likely something simple but I've tried escaping and using different variations of double quotes. I moved on to the `eval` method and finally the `set` method, which I like but, it is being passed in literally no matter what I throw at it

Any advice would be appreciated

EDIT: For anyone that comes across this. It wasn't matching anything and as a last try was attempting the asterisk as literal. I missed that the backups had a hostname in them and the log files were named by timestamp only. Totally simply error.

https://redd.it/14j1ugx
@r_bash

Small collection of Bash noscripts to launch functionalities in folders
https://github.com/tanrax/bash-folders

https://redd.it/14io0q7
@r_bash

problem in my while loop: empty output files

Hello,

&#x200B;

I have this command:

awk -F'\t' '$3 ~ "Aspergillus fumigatus" {print}' $krakenfile > Aspergillusfumigatuslines.txt

It is working very fine, It just extracts every line in the file $krakenfile that contain the word "Aspergillus fumigatus" in its third column.

Now, I want to introduce another file ($fungalnames) that contains multiple lines, every line contains names of species and I want to apply the command on every line of the new file.

I tried this:

while IFS= read -r speciename
do
awk -F'\t' '$3 ~ "$speciename" {print}' $krakenfile > "${speciename}lines.txt
done < $fungalnames

Anyway, the output files are created (I had 8 species names in my $fungalnames file and there are 8 output files, every file is named to a line, but all the files are empty !!

I tried multiple times, and I changed syntaxes, but nothing is working.

I think I am escaping an important thing, can anyone help me please! Thank you in advance!

https://redd.it/14wtsdb
@r_bash

Automatically upload files to usb when usb is plugged in

Wondering if there is a way so that when we plug in a USB it takes files from a certain directory and automatically uploads them to the USB? Does anyone have any ideas? Thank you!

https://redd.it/14wto8h
@r_bash

How to repeat a command with parameters swapped?

If I run a command like:

specialcmd -ab Parm1 Parm2

to become #536 in the command history and later I want to run the same again with parameters swapped I can do:

!536:0-1 !536:3 !536:2

Is there a way to say combine the word designators without repeating the command number?

I did not find a viable delimiter or any documentation going that deep.

&#x200B;

&#x200B;

&#x200B;

https://redd.it/14iqv5q
@r_bash

Compile noscript for GCC 11.4.0, 12.3.0, & 13.1.0 Debian + Ubuntu

So I couldn't take waiting on gcc-13 on Debian Bookworm and Ubuntu Jammy so I went all out and made a noscript to compile the latest GCC 11, 12, and 13 versions.

I have tested it on Debian 12 and Ubuntu Jammy and it seems to work well for me.

Thought someone might appreciate this and want to give it a try as a strictly personal use or testing environment.

Be sure to read the build info and notes at the top of the noscript.

Also, there is a "debug" variable you can toggle "ON" or "OFF" to help troubleshoot any issues you might be having.

bash <(curl -sSL https://gcc.optimizethis.net)

GitHub Script

Cheers

https://redd.it/14iqn4z
@r_bash

Can someone help me understand why I'm getting this 'invalid bytes error"?

I have a noscript:

#!/bin/bash

echo FLASK_SECRET_KEY=\""$(< /dev/random tr -dc _A-Z-a-z-0-9 | head -c${1:-25}; echo;)"\" > .env

if [ "$1" = "rebuild" ]
then
if [ "$2" = "all" ]
then
docker compose down
docker image rm app1
docker image rm app2
docker image rm app3
else
for image in "${@:2}"
do
docker compose down
docker image rm "$image"
done
fi
else
echo Invalid argument.
exit
fi

docker compose up -d

Everytime i run it, it runs perfectly but it always shows this error at the start:

head: invalid number of bytes: ‘rebuild’

&#x200B;

https://redd.it/14i9196
@r_bash

Set PS1 variable information as a bar.

I was curious if there was a way to move all the usual information from the prompt to a line at the top of the terminal.

I have seen some wonderful and amazing terminal prompts but I’m at the point where just having a status line with the information I need seems more appealing.

Thanks for the help.

https://redd.it/14hdd49
@r_bash

Remove combinatorial match from a two column file

I've been running into a brick wall with a particular issue - there's a two column tsv file roughly in format of:

A B
A C
A H
B A
B C
B E
C B
C H

And I'm trying to remove duplicates regardless of order - so the outcome would be:

A C
A H
B E
C H

Since 'A B' = 'B A', 'B C' = 'C B' and so forth. I've tried shuffling the orders with awk, but of course it didn't work. Any lead would be appreciated.

Thank you!

https://redd.it/14gsupf
@r_bash

Combine values from a list

I am trying to use jq to put a list on one array of values.

&#x200B;

my input is this a newline separated inputs

name1

name2

name3

namex

&#x200B;

I would like to have this output

names : [name1, name2, name3, namex\]

Thanks for any guidance on how to tackle this, I am new to jq

&#x200B;

https://redd.it/14fkb6x
@r_bash

How to rsync to directory underneath mountpoint?

I have a normal directory: \~/.mozilla/firefox

I then mount something to \~/.mozilla/firefox

How can I rsync data to the original \~/.mozilla/firefox directory found underneath the mountpoint?

I'm using a tmpfs to reduce disk writes from firefox but would like the option to rsync the data while the tmpfs is mounted.

No, I don't want to use profile-sync-daemon

https://redd.it/14xbr7r
@r_bash

How to substring a variable using another variable as length?

Hi, I have this piece of code:



length_seq=${#sequence}
fold_without_mfe=${#fold_result:0:$length_seq}

&#x200B;

Where length_seq is the length of a genomic sequence, for example, 30. The fold_without_mfe variable is a simulation result, but it has a few more characters than I need, so I want to take only the firt 30 of this variable to match my sequence. I tried this but it throws me a bad substitution error, how can I approach this problem?

https://redd.it/14xlrwr
@r_bash

How can I get just part of the line in my text file?

I am running this command, trying to gather some data

grep -e "PROJECT" $(sudo find /path/to/dir/ -iname config) > projects.txt

and get a lot of output like this:

/path/to/dir/config/config-server.address:PROJECT="xyz";

All i want is the information between : and ;

I have tried using awk, but this gives me the same as no awk

grep -e "PROJECT" $(sudo find /path/to/dir/ -iname config | awk '{print $1}') > projects.txt

&#x200B;

https://redd.it/14xriut
@r_bash

Bash grievances

At this moment I have a love/hate relationship with Bash. I write in it for years and I can say that I write it, with modesty, well.

I understand that Bash isn’t like modern programming language like Golang.

But from time to time when I want to refractor code or more structured setup I felt more and more that the language is in the way.

Trying to translate something like the composite design pattern to Bash was not possible because it is so simple as it can be. That it doesn’t have support like inheritance. Return a type from a function and not by declaring or print it. Passing a function as a argument to a function is not possible.

All the things that could make life/noscripting a little bit easier, it doesn’t have any support for it.

And Bash has made progress in the last ten years. Arrays is something that I frequently use. But the speed of developments is way too irritating slow. Just as the speed of the shell it’s self.

And yes. I acknowledge the existence Python and others. But as someone who comes to different companies where there is no knowledge of these modern languages (even in 2023), I have to do it with Bash.

https://redd.it/14xwui5
@r_bash

How can I find files from a list without the extension?

I'm working on a personal project. I have 2 folders:


/JPG
/RAW


Each folder should contain files with identical filenames but with different extensions (they will always be either .jpg or .rw2)


For example:

/JPG/image1.jpg
/JPG/image2.jpg
/RAW/image1.rw2
/RAW/image2.rw2


Now there are some instances where a file only exists in one folder, but not the other.


I have identified these files and placed them into a list within a .txt file using the below command:

comm -3 <(print -lr JPG/*(:t:r)) <(print -lr RAW/*(:t:r)) > differences.txt


When I cat this file, I can see all the files which only belong to one of the folders. For example:

cat differences.txt
image7
image36
image49

&#x200B;

My next goal is to move these files into a new folder. But I'm not sure how. As you can see, the extensions were removed/excluded when they were outputted to "difference.txt"

I was thinking maybe I should use the find command and input a list but that doesn't seem to be possible. I've tried a few noscripts I found on stack overflow but to no avail.

Please could somebody point me in the right direction?

https://redd.it/14y0v6i
@r_bash