What is the best and faster tool for counting lines in a file that matches a specific pattern. The text file is quite a large one about 4GB



https://redd.it/1dva3ph
@r_bash

Add command into an existing variable (curl+torsocks usage)

I have an existing variable

PREVIEW=$(curl -Ls $URL)

if the output of the variable $PREVIEW results empty (maybe because api limit is reached), I want to add torsocks before curl and then retry

what is the correct way to launch torsocks curl -Ls $URL? I've tried to eval $PREVIEW without success.

Thanks in advance.



https://redd.it/1dvffrr
@r_bash

Help with color formatting / redirection in bash wrapper function?

**TD;LR - This one is probably more involved. I have a `wrapper` function ([pastebin](https://pastebin.com/skxQJpQ2)) that works perfectly for capturing `stdout` but seems to blow up when I attempt the same tricks with `stderr`. I'm assuming I'm doing *something* wrong but have no idea what.**

A little over a week ago, I had asked a question about [redirection](https://old.reddit.com/r/bash/comments/1dnxehx/question_about_stream_redirection_file_denoscriptors/) and got some excellent answers from you guys that really helped. Since then, I've been trying to adapt what I learned there to create a more flexible wrapper function capable of the following:

- wrapping a call to some passed application + its args (e.g. `curl`, `traceroute`, some other bash function, etc)
- capturing stderr, stdout, and return code of the passed call to local variables (with the intention of being able to export these to named variables that are passed to the wrapper function - I have done this in other functions and am not worried about this part, so that's out of scope in the examples below): Solved
- allow selectively printing stderr / stdout in real time so that certain commands like `traceroute reddit.com` (progress on stdout) / `curl --no-clobber -L -A "${userAgent}" "${url}" -O "${fileName}"` (progress on stderr) / etc can still be displayed while the command is still running: Solved - mostly based on adapting [this](https://stackoverflow.com/questions/12451278/capture-stdout-to-a-variable-but-still-display-it-in-the-console)
- Preserve colors in captured variables: Solved
- Preserve colors in realtime output: partially solved (works for stdout but not for stderr)

Using u/Ulfnic 's excellent [suggestion](https://old.reddit.com/r/bash/comments/1dnxehx/question_about_stream_redirection_file_denoscriptors/la6gcuu/) as a base, I've almost got everything I want but I'm stumped by the color output I'm getting. I've been over this a dozen times and I'm not seeing anything that sticks out... but obviously it is not behaving as desired.

I'm (currently) on Fedora 39 which has

$ bash --version | head -1
GNU bash, version 5.2.26(1)-release (x86_64-redhat-linux-gnu)

The functions I am using are defined [here](https://pastebin.com/skxQJpQ2) which I have saved as `funcs.sh` and am loading using `. funcs.sh`.

The expected usages:

A) running the wrapper function with no options and passing it a command (plus args) to be executed, it will capture stderr, stdout, and return code to separate internal variables which can be acted on later. This works perfectly and its output looks like this

https://files.catbox.moe/rk02vz.png

B) running the wrapper function with the `-O` option will print stdout in realtime so commands like `traceroute` can give progress updates without waiting for the app to finish running before output is displayed. Should still do all the same things as (A) but additionally print stdout in realtime, while preserving color. This also works perfectly and its output looks like this

https://files.catbox.moe/8a7iq0.png

C) running the wrapper function with the `-E` option will print stderr in realtime so commands like `curl` can give progress updates without waiting for the app to finish running before output is displayed. Should still do all the same things as (A) but additionally print stderr in realtime, while preserving color.

This one is broken but I don't even understand why the code isn't working as expected. Its output looks like this

https://files.catbox.moe/obryvu.png

Functionally, it has a few issues:

1. It is incorrectly capturing stderr output to the local variable `outstr`.
2. The realtime printing of stderr loses all color for some reason, even though AFAICT the handling for stdout and stderr is identical
3. The local variable `errstr` loses all color formatting, despite the incorrectly assigned `outstr` preserving it.

When I run `wrapper -E realTimeStderrTest` (e.g. the un-colorized version of the same test), it works perfectly (issue \#1 does

not happen but issues \#2 and \#3 aren't applicable in black and white mode) so I am assuming it is something related to colors that it doesn't like but I have no clue what exactly. That output is [here](https://files.catbox.moe/zqnae9.png)

https://redd.it/1dvmij1
@r_bash

why are mode bits represented in bash in both octally? AND symbolically? or alphabetically?

so i understand that file permissions in mode bits are represented alphabetically, or symbolically?

r = read

w = write

x = execute

now if i want to change this with chmod, i need to do this with octals?

x = 1

w = w

r = 4

can someone explain this to me? why are there two systems?

https://redd.it/1dw0hgq
@r_bash

Displaying stdout from continuously running program and run command if string present

Hi, I have a noscript that runs in a terminal window, and I need to see the displayed stdout from a program that it launches, which continues running. But I also need to monitor the program's stdout and run a command if a string eventually appears in the output. Once that condition is met then I don't want to see the terminal anymore so I kill the terminal, but the program keeps running until I exit its window. I would prefer to not have to write the stdout to a file for parsing. This is as close as I can get, but it doesn't show the program's output. Any tips? Thanks!

#!/bin/bash
thisPID="$(echo $$)"
docker container start Something
nohup xfreerdp /v:localhost |
grep --line-buffered 'PDUTYPEDATA' |
while read; do
wmctrl -c 'FreeRDP' -b toggle,maximizedvert,maximizedhorz;
kill $thisPID
done

https://redd.it/1dw291c
@r_bash

Text pasted to a file not parsed correctly (invisible characters)

I have a noscript which parses text on a line-by-line basis. Works as expected, except for text pasted from a particular webpage that involves both English text and foreign characters (cannot share more details than that), where it seems to parse every *other* line by the shell. However, if I `echo` *instead* of running `yt-dlp` on what's parsed, it still prints expected values of `$noscript` and `$url`. I was told there are invisible characters involved like `$'\r'` byte (0x0d) which bash does not consider as whitespaces to ignore, but how can I see if this is the case on a text editor (I use Neovim) and/or fix the cause of the issue? If I append a character manually to each line that was pasted, the noscript then parses each line as you would expect and is a workaround for the issue.


file="$2"

while IFS= read -r line || [[ -n "$line" ]]; do
noscript="${line#* }"
url="${line%% *}"

if [[ -n "$noscript" ]]; then
default_template="$noscript - %(uploader).12B (%(upload_date)s) %(id)s.%(ext)s"
else
default_template="%(noscript).140B - %(uploader).12B (%(upload_date)s) %(id)s.%(ext)s"
fi

yt-dlp --output "$default_template" "$url"
# echo "$url"

done < <( awk NF "$file" ) # Delete blank lines from $file

exit

https://redd.it/1dw8l6e
@r_bash

Is there any sense in quoting special vars like $? and $# ?

I mean, bash and other shells are aware $? and $# cant contain any spaces or patterns, so I guess they treat $? and "$?" the same? Or do they still try to perform word splitting on $? ?

https://redd.it/1dwq3no
@r_bash

Trying to send multiple flags to rsync

So I use rsync over ssh to move files over my local network. I'm not worried about security too much, but use rsync over ssh so I can do it over internet sporadically.

This is what works:

export DEN=username@den.local
export USER=/home/kitchen
rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -rptuv --delete --progress $DEN:/home/username/Music/English/A/ $USER/Music/Music/A/

I am trying to put all the flags in a variable.

The following variable doesn't work

export RSYNCFLAGS="-e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -rptuv --delete --progress"
rsync $RSYNCFLAGS $DEN:/home/username/Music/English/A/ $USER/Music/Music/A

I also tried using a variable array, but that didn't work as expected:

export RSYNCFLAGS=(-e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -rptuv --delete --progress)
rsync ${RSYNCFLAGS*} $DEN:/home/username/Music/English/A/ $USER/Music/Music/A

They both have problems with the -e at the beginning (it doesn't add it to the variable at all). When I move that later on, it still gives a problem. Can anyone help me out?

https://redd.it/1dwquoi
@r_bash

Experience customizing the colors on "ohmybash"?

I was wondering if anyone here has the experience of altering or modifying the provided themes in "Ohmybash" I'm trying to change the powerline and text color on my "agnoster" theme but no luck thus far.

https://redd.it/1dwyh4m
@r_bash

Parameter Substitution and Pattern Matching in bash

Hi. I may have misread the documentation, but why doesn't this work?

Suppose var="ciaomamma0comestai"
I'd like to print until the 0 (included)

I tried echo ${var%%[:alpha:\]} but it doesn't work

According to the Parameter Expansion doc

>${parameter%%word}
The word is expanded to produce a pattern and matched according to the rules described below (see Pattern Matching).

But Patter Matching doc clearly says

>Within ‘[’ and ‘\]’, character classes can be specified using the syntax [:class:\], where class is one of the following classes defined in the POSIX standard:

alnum alpha ascii blank cntrl digit graph lower print punct space upper word xdigit

So that above command should work...

I know there are other solutions, like {var%%0*} but it's not as elegant and does not cover cases where there could be other numbers instead of 0

https://redd.it/1dxix8c
@r_bash

a serialized dictionary argument parser for Bash (pip-installable)

Hey all, I built a serialized dictionary argument parser for Bash, that is pip-installable,

pip install blue_options

then add this line to your `~/.bash_profile` or `~/.bashrc`,

source $(python -m blue_options locate)/.bash/blue_options.sh

it can parse a serialized dictionary as an argument; for example,

area=<vancouver>,~batch,count=<-1>,dryrun,gif,model=<model-id>,~process,publish,~upload

like this,

function func() {
local options=$1

local var=$(abcli_options "$options" var default)
local key=$(abcli_options_int "$options" key 0)

[[ "$key" == 1 ]] &&
echo "var=$var"
}

more: [https://github.com/kamangir/blue-options](https://github.com/kamangir/blue-options) + [https://pypi.org/project/blue-options/](https://pypi.org/project/blue-options/)

https://redd.it/1dxmewr
@r_bash

Help customizing "OhMyBash"?

How can I get the color #55c369 as the color for my prompts background on the agnoster theme , It seems like "OhMyBash" uses the 'ANSI' color code--So how would I get the color translated to ANSI if that possible? Currently my prompt is displaying the opposite color way I want

What I currently have\^

What I would like to have\^

https://redd.it/1dxnvzg
@r_bash

Print missing sequence of files

I download files from filehosting websites and they are multi-volume archived files with the following naming scheme (note the suffix .part0..<ext>, not sure if this is the correct regex notation):

sampleA.XXXXX.part1.rar
sampleA.XXXXX.part2.rar
sampleA.XXXXX.part3.rar # empty file (result when file is still downloading)
sampleA.XXXXX.part5.rar
sampleB.XX.part03.rar
sampleC.part11.rar
sampleD.part002.rar
sampleE.part1.rar
sampleE.part2.rar # part2 is smaller size than its part1 file
sampleF.part1.rar
sampleF.part2.rar # part2 is same size as its part1 file



I would like a noscript whose output is this:

sampleA.XXXXX
- downloading: 3
- missing: 4
sampleB.XX
- missing: 01, 02
sampleC
- missing: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10
sampleD
- missing: 001
sampleE completed
sampleF
- likely requires: 3

I implemented this but it doesn't handle 1) partN naming scheme where there's variable amount of prepended 0's (mine doesn't support any prepended 0's) and 2) also assumes part1 of a volume must exist. This is what I have. I'm sure there's a simpler way to implement the above and don't think it's worth adjusting it to support these limitations (e.g. simpler to probably compare find outputs with expected outputs to find the intersectionso I'm only posting it for reference.

Any ideas much appreciated.

https://redd.it/1dxn80x
@r_bash

stdin read question

ls -1 | while IFS= read -r line; do
echo "$line"
read -p "Press Enter to continue..."
done

Why does this not prompt after every ls line?

It should pause after every line, cause thats how stdin & read works?

And what would be a workaround to make this work as i exect?

https://redd.it/1dxlqb1
@r_bash

How do I handle window decoration offsets when positing windows with xdotool or wmctrl?

Here's two examples of positing windows to X=40 Y=40 and changing their size to 1000x600:

wmctrl -r :ACTIVE: -e '0,40,40,1000,600'

xdotool getactivewindow windowmove %@ 20 20 windowsize %@ 1000 600

Depending on the program both wmctrl and xdotool will calculate that 40 X/Y position BEFORE or AFTER window decorations. So some programs will always be placed at X Y where others will always be placed at X+DEC_WIDTH_LEFT Y+DEC_HEIGHT_TOP.

I don't mind adding my own decoration offsets but I have no way of telling programmatically which windows should have that offest applied.

So far i've tried the following but neither gives me a property that distinguishes between them:


xdotool getactivewindow getwindowname getwindowgeometry --shell

xprop -id "$(xdotool getactivewindow)"

xwininfo -id "$(xdotool getactivewindow)"

Here's a list of programs based on how they're positioned realtive to decorations:

# Placement BEFORE decorations
krita
mpv
libreoffice {writer,calc,math,ect}
brave-browser

# Placement AFTER decorations
gimp
xarchiver
smplayer
alacritty
xfce4-terminal
thunar
chromium
firefox (with Title Bar enabled)

Glad for any insight, thank you.

https://redd.it/1dy6c7v
@r_bash

noscript no work halp

while sleep 0.5; do find . -name '.c' -o -name '.h' | entr -d make; done &
sh -c 'while sleep 0.5; do find . -name '.c' -o -name '.h' | entr -d make; done' &

Why does this not keep running in a background?
Its a closed while loop, but it seems to exit in a first iteration of the loop.

but without the & it works as expected

also entr, its just a simplified version of inotify

http://eradman.com/entrproject/

https://redd.it/1dy6jvr
@r_bash

.bash_history format

In bash, running the `history` command prints in a beautifully formatted output:

5625 [2024-06-22 12:22:38] F libdisplay-info
5626 [2024-06-22 12:22:50] p -Ssq libdisplay-info
5627 [2024-06-22 12:23:02] p -Fl libdisplay-info
5628 [2024-06-22 20:35:24] p -Flq libdisplay-info
5629 [2024-06-22 20:36:02] Q libdisplay-info

However, the .bash_history file looks like crap in comparison:

#1719084158
F libdisplay-info
#1719084170
p -Ssq libdisplay-info
#1719084182
p -Fl libdisplay-info
#1719113724
p -Flq libdisplay-info
#1719113762
Q libdisplay-info

I've hacked together an ugly, fragile bit of code to write a duplicate the first example to `"${HOME}"/.bash_history_dated`.

sed 'N;s/\n/ /' < "${HOME}"/.bash_history \
| cut -c2- \
| awk '{$1 = strftime("%F %r", substr($1,1,10))} 1 {print "["$1"] ",$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20}' \
> "${HOME}"/.bash_history_dated

This runs whenever I exit the shell via `trap "/home/jeff/bin/bash-history-timestamp" 0`, and the results (if there's not more than two lines input per command).
This is great, and I didn't want the command numbers included in this.

[2024-06-22 12:22:38 PM] F libdisplay-info
[2024-06-22 12:22:50 PM] p -Ssq libdisplay-info
[2024-06-22 12:23:02 PM] p -Fl libdisplay-info
[2024-06-22 08:35:24 PM] p -Flq libdisplay-info
[2024-06-22 08:36:02 PM] Q libdisplay-info

My questions are:

1) Why doesn't this work in place of the ugly code `history | cut -c 8- > "${HOME}"/.BASH_HIST_DATED`. This works in the shell, but only creates an empty file when ran in the noscript.

2) How to improve my ugly code to be cleaner and more robust to work with multiple cli input lines if it's the only solution.

https://redd.it/1dyl2ln
@r_bash

Running a remote command using EOF but only getting a fraction of the output before the noscript completes. How do I keep stdout open until the command completes? Is there a way to inject a "wait" command into EOF?

I'm running a command on a remote system. The remote system is not linux and I can't send piped/chained commands, so I have to use EOF (unless there's another alternative I'm not aware of.

The below command outputs \~12k lines when I run it on the console directly, but when I run it from my unix host, it gets a couple hundred before it logs out.

{
ssh user@host << EOF
\environment no more
\show mobile-gateway bearer-context | match 9/10
\logout
EOF
} > $output


I see the logout command being executed, so the remote system completed the show command, but the output isn't being captured on my side.

https://redd.it/1dz3jmk
@r_bash

how do I use mkdir -p for mkdir a and a/b and a/c and a/d? Why using -p?

Hi, Id like to learn how I should use mkdir -p for mkdir a a/b a/c a/d

I use actually mkdir -p a a/b a/c a/d

but what is the advantage of using the flag -p?

I can use the command mkdir a a/b a/c a/d without -p and get the same tree...

Thank you and regards!

https://redd.it/1dzckiu
@r_bash

Strange problem with [ -e ] on network share

Dear all

     I've a Bash noscript that checks for the existence of a file-on-a-network. The file path starts /run/user/1000/gvfs/smb-share:server=. When the file does exist, the following happens.

An initial -e check gives a false negative, i.e., it reports (by evaluating as false) that the file does not exist when it does.
A second check, carried out immediately after the first check, gives the correct result, i.e., it reports (by evaluating as true) that the file exists.

     Can anyone shed light?
     I am on 6.5 kernel, on Linux Mint Cinnamon 21.3 (which is based on Ubuntu 22.04), with Bash version '5.1.16(1)-release'.

https://redd.it/1dzzrcq
@r_bash