Confirming speed / bash practices

Long story short, I have a CSV file, 5 fields.

I read it into bash, and used IFS / while loop to read through the delimiters.

I'd say the CSV file in total is roughly 40MB. Pretty large for a CSV.

I went back and just decided to add a condition to check if the folder existed, because each row can be placed in different folders depending on the category.

When I added that check along with mkdir, I noticed that the speed dramatically went to hell. Without mkdir, the file completed in about 2 minutes. When I add mkdir -p, the speed to run the file went above 30 minutes, and then I just killed the noscript because it wasn't worth waiting anymore.

Is mkdir that heavy in a while loop? Or is their a better way to do this to ensure the sub-folders are created. Obviously I can't do what I was doing, because that just tanked the speed.

Just to give an idea, all I did was:

if test ! -d ${CAT}; then
    mkdir -p ${CAT}
fi

Which I guess I could just do mkdir -p and the check doesn't really matter. But I was just wondering what aspect is what caused the slow-down, using test or mkdir, and if there's a much faster set of commands when working inside a loop, or should I try to do as much as possible outside the loop and do the bare minimum inside.

I could see it adding a few minutes on, but I wasn't expecting such a dramatic change.

I'm sitting here running tests, and it appears if I do anything at all command based, it kills the speed. I tried tr just to manipulate a string, and that also did it.

https://redd.it/1gee908
@r_bash

cat match string / move to end of file

i've been over a few different websites reading up on this, but I feel like I'm missing something stupid.

I have a file, which contains a mix of ipv4 and ipv6 addresses. I'd like to use sed to match all ipv6 addresses in the file, cut them from their current position, and move them to the end of the file.

I've tried a few ways to do this, including using cat to read in the file, then using sed to do the action. It seems to be finding the right lines, but I read online that /d should be delete, and I'm trying to just get that to work before I even try to append to the end of the file.

cat iplist.txt | sed -n "/::/d"

I haven't even figured out the part of appending to the end of the file yet, I just wanted to get it to delete the right lines, and then add it back

cat iplist.txt | sed -n "/::/d" >> iplist.txt

https://redd.it/1gfgi10
@r_bash

M3U file list

I know I can create a file list with ls -1 > filename.txt, but I don't know how to prepend the directory path. I'm trying to create an m3u file list I can transfer to Musicolet on my phone. Can someone point me in the right direction?

https://redd.it/1gfk5jr
@r_bash

File names with spaces as arguments

I want to merge a bunch of PDF s. The file names have spaces : a 1.pdf, b 2.pdf, a 3.pdf. And they're a lot of them.

I tried this noscript:

merge $@

And called it with merge.sh *.pdf

The noscript got each separated character as an argument : a 1.pdf b 2.pdf a 3.pdf.

I there a way to feed these file names without having to enclose each in quotes?


https://redd.it/1gg0eh2
@r_bash

Help (Newbie)

if i gonna learning bash noscripting, where to start and how?. i know understand bash noscripting, but can'not make it myself

https://redd.it/1gge37v
@r_bash

6 Techniques I Use to Create a Great User Experience for Shell Scripts
https://nochlin.com/blog/6-techniques-i-use-to-create-a-great-user-experience-for-shell-noscripts

https://redd.it/1gi0z9o
@r_bash

Is there a CLI command to run default application against a file?

Example - you have an zingo file so you can enter

the-starter-command abc.zingo

and Zingo abc.zingo gets started - the-starter-command knows about the default application for different file types.

Real example - Libre Office for .doc and .odt files.

https://redd.it/1gjid8x
@r_bash

Archive of wiki.bash-hackers.org
https://github.com/flokoe/bash-hackers-wiki

https://redd.it/1gk3o6p
@r_bash

Color in prompt

I recently moved from powershell to bash and installed starship. My question is how can I make prompt to be not just white text:

https://preview.redd.it/h30nwaem7oxd1.png?width=918&format=png&auto=webp&s=83fe0e408da06de31a3e5d2f1fc6616f81ef45ce

but instead change color like in powershell?

https://preview.redd.it/01x6w2eq7oxd1.png?width=438&format=png&auto=webp&s=057aa79163fab217974bcda86b9af21360ba78b5



https://redd.it/1ger5b3
@r_bash

Issues when customizing LS_COLORS

Hello everyone,

I recently parametered my .bashrc file to customize my ls command colors. But some file types appear in two different colors, when I only put one in my .bashrc. Example with my .md files, which are supposed to be light blue but also appear hot pink :

https://preview.redd.it/71vj5kls6qxd1.png?width=569&format=png&auto=webp&s=c288ee9581663d3bd247184eebbc9a8410f3e667

Here are my parameters in my .bashrc :
`LS_COLORS="di=1;38;5;218:*.sh=1;38;5;213:*.tar=1;38;5;205:*.zip=1;38;5;205:*.gz=1;38;5;205:*.bz2=1;38;5;205:ln=1;38;5;218:*.docx=1;38;5;174:*.doc=1;38;5;174:*.pdf=1;38;5;174:*.jpg=1;38;5;174:*.png=1;38;5;174:*.jpeg=1;38;5;174:ex=1;38;5;198:*.md=1;38;5;153"`

I did not modify anything else in any other file. Is there anything I'm missing? How can I make my files the right color?

https://redd.it/1geziuf
@r_bash

Useful Shell Functions for Developers
https://2kabhishek.github.io/blog/useful-shell-functions/

https://redd.it/1gi078n
@r_bash

Pass delimited string variable-array directly into for loop?

I successfully followed instructions at this StackOverflow post to convert a string variable, var="a,b,c" to a 3 element array ignoring the commas:
arrIN=(${IN//,/ })

for i in "${arrIN@}"; do
echo "$i";
done

I would like to place command right after i in:
Neither of the following worked:

for i in "${(${IN//,/ })@}"; do
echo "$i";
done
Error: bash: ${(${IN//,/ })@}: bad substitution

Same error when I removed the the parentheses, ( ).

https://redd.it/1gh9ui8
@r_bash

Idea's needed

I was mildly annoyed at the one line update noscript for ubuntu. So I created a wonderfully bloated monstrosity of a bash noscript to accomplish the updates and some other stuff. I wish to bloat this thing further with more functionality. Any ideas would be appreciated.

https://github.com/ageorge224/Update-Script

https://redd.it/1gkkkee
@r_bash

Bash grep

I've tried numerous rules, but I can't seem to figure out the correct pattern.

I have numerous words in a file

abcd:1234:ef35
f24a:5fa3
ab12:34fa:cd42

I'm trying to write a grep that if a word contains more than 1 colon, the entire word is returned.

The issue is that the strings may be in a list, or they may be side-by-side, only separated by a space

abcd:1234:ef35 f24a:5fa3 ab12:34fa:cd42

I can't seem to pull the whole word, it always returns a single character.

https://redd.it/1gklxut
@r_bash

Who else has something like this in their .rc?

straight to prod \(\/s\)

https://redd.it/1gkahoz
@r_bash

Temporarily change terminal 16 color palette in a noscript?

What's the specific term to call/describe the 16 colors that's always being used by the terminal? (neofetch colored squares, etc.)

And is there a way to dynamically change them through a noscript?

Searching for solutions, not sure if the command I need is tput or dircolors or something else.

Why do I want to do this? One utility I'm using will only use the set of 16 colors used by the terminal. I'm looking for a workaround so that I can force it to use colors I specify (from the 256 color set) without changing the defaults of my terminal.

https://redd.it/1gh3xf1
@r_bash

Trying to understand why my search returns no results

Hi all,

Let me preface this by saying this is day one of my noscripting journey. I'll also add that I am using ChatGPT to try to cheat the hell out of it!

With that said, here is my problem. I am going step by step through the process of converting a PDF to text, parsing the text for info and then saving that info into a csv file.

I am on OSX so I started by using Shortcuts to "Get text from PDF" which i initially outputted to a text file; it worked fine. I then added a noscript (generated by ChatGPT) to search the clipboard (I changed the Shortcuts output to the clipboard) for the line "Grand Total" and output the line below (which had the amount) to a csv file. However the noscript can't find the line "Grand Total". Ive tried this initially with Applenoscript and now with a shell noscript, neither work.

Here is the code I'm using:

#!/bin/bash

echo "pbpaste version: $(pbpaste)"

# Get the clipboard contents
input_text=$(pbpaste)

# Convert the clipboard text into an array of lines
mapfile -t lines <<< "$input_text"

# Initialize variables
grand_total_found=false
grand_total_value=""

# Loop through each line
for ((i = 0; i < ${#lines[@]}; i++)); do
# Normalize the line by trimming spaces and converting to lowercase
current_line=$(echo "${lines[i]}" | sed 's/^[ \t]*//;s/[ \t]*$//' | tr '[:upper:]' '[:lower:]')

# Check if line contains "grand total" (case-insensitive)
if [[ "$current_line" == *"grand total"* ]]; then
# Get the next line for the total amount and trim whitespace
grand_total_value=$(echo "${lines[i+1]}" | sed 's/^[ \t]*//;s/[ \t]*$//')
grand_total_found=true
break
fi
done

# Check if "Grand Total" was found
if [ "$grand_total_found" = true ]; then
# Prompt for output file location
echo "Enter the path to save the CSV file (e.g., /path/to/output.csv):"
read -r output_file

# Write "Grand Total" and value to the CSV file
echo "Grand Total,$grand_total_value" > "$output_file"
echo "Grand Total saved to CSV successfully at $output_file"
else
echo "No 'Grand Total' found in the clipboard text."
fi


And here is the output from that currrently:

> pbpaste version: Remittance Statement
>
> 1.00
>
> To:
>
> REDACTED
>
> Date: 31/10/2024
>
> Ref: TR16246
>
> (Property) REDACTED
>
> Date Main Tenant Denoscription VAT
>
> Charge
>
> (inc VAT)
>
> Payment
>
> (inc VAT)
>
> 29/11/2022 Not Applicable 392208 7,850.29
>
> 0.00
>
> 7,850.29
>
> 0.00
>
> Grand Total
>
> 7,850.29
>
> REDACTED
>
> E-mail: REDACTED
>
> VAT No: REDACTED
>
> Page 1 of 2
>
> Income and Expenditure
>
> Type VAT
>
> Charges
>
> (inc VAT)
>
> c000- Contractor Charge 7,850.29
>
> Payments
>
> (inc VAT)
>
> Totals
>
> 7,850.29
>
> Total Remitted 7,850.29
>
> REDACTED
>
> Page 2 of 2
>
> No 'Grand Total' found in the clipboard text.


I added the echo just to review the text it was taking from the clipboard was correct.

Any help at this basic stage much appreciated as this is going to get more complicated (I'll eventually need to output multiple lines). Also, what are the best places to look for documentation onn this sort of stuff?

Thanks all.

https://redd.it/1gghmnl
@r_bash

Simple bash noscript help

Looking to create a very simple noscript to start a few services at once just for ease. My issue is it only wants to run one or the other. I'm assuming because they're both trying to run in the same shell? Right now I just have

cd ~/path/to/file &
./run.sh &
sudo npm run dev

As it sits, it just starts up the npm server. If I delete that line, it runs the initial bash noscript fine. How do I make it run the first noscript, then open a new shell and start the npm server?

https://redd.it/1gl3ox2
@r_bash

Learning more practical automation

Can anyone point me to where I can learn more real world noscripting. More so applying updates to things or monitoring system health, so far all of the “courses” don’t really help more than understanding simple concepts.

https://redd.it/1gm1gfm
@r_bash

ImageMagick6: ¿how change save 75 compr.(default) to 95 compr.?

Hi, this ask is about ImageMagic 6: Do you know how I change the compression for save by default is 75 and I'd like to set compression 95 (so change 75 for 95 by default).

Thank you and Regards!

https://redd.it/1gm5myp
@r_bash

When a process is killed because it exhausted free memory, I'd prefer bash says "Killed: out of memory" instead of just "Killed"

I see in siglist.c the internationalized string:

sys_siglist[SIGKILL] = _("Killed");

But I'm wondering if we can use anything that the kernel does around https://github.com/torvalds/linux/blob/master/mm/oom_kill.c#L947 to tell the user that the reason was low memory?

https://redd.it/1gmfdxb
@r_bash