Two first lines of csv files using bash + general explanation

I want to find the first two lines of a csv file using bash in a jupyter notebook.

This is my code for the moment, what I should put in the X in order to obtain the first two lines of the csv file?

​

%%sh

head -X data/earthquakes.csv

​

I'm learning a bit of how to using bash with jupyter notebooks, in fact in don't understand a why is bash used in data science. Could somebody also explain this to me and why %%sh does not work in windows?

​

Thank you very much!

https://redd.it/x7e8a8
@r_bash

Not having success moving a file at the end of my noscript after transcoding completes

Hello! I have a noscript that I borrowed to help me post process recordings from Plex to A) remove commercials with an app called Comcut and B) transcode it with Handbrake CLI.

The problem I am running into is that once the noscript removes the commercials and then transcodes the file, it is not successfully replacing the newly transcoded file at the end of the noscript...

I even tried to embed the users password and run the "mv -f" command with sudo, but that did not seem to work either. Here is the noscript, and the command to move the newly transcoded file is the 7th line to the end, marketed with "# \*\*\*\* OVERWRITE HERE \*\*\*\*"

Any suggestions would be greatly appreciated! Thank you!

#! /bin/bash
#
# Plex DVR Postprocessing
# Version 0.0.1
# twitter.com/thatvirtualboy
# www.thatvirtualboy.com
#

# FIRST, RUN COMCUT TO REMOVE COMMERCIALS, THEN TRANSCODE AND COMPRESS

lockFile='/tmp/dvrProcessing.lock'
inFile="$1"
tmpFile="$1.mp4"
dvrPostLog='/tmp/dvrProcessing.log'
time=`date '+%Y-%m-%d %H:%M:%S'`
handbrake=/opt/homebrew/Cellar/handbrake/1.5.1_1/bin/HandBrakeCLI
cut=/opt/homebrew/Cellar/comskip/bin/comcut

echo "'$time' Plex DVR Postprocessing noscript started" | tee $dvrPostLog

# Check if post processing is already running
while [ -f $lockFile ]
do
echo "'$time' $lockFile' exists, sleeping processing of '$inFile'" | tee -a $dvrPostLog
sleep 10
done

# Create lock file to prevent other post-processing from running simultaneously
echo "'$time' Creating lock file for processing '$inFile'" | tee -a $dvrPostLog
touch $lockFile

# Run comcut
echo "'$time' Comcut started on '$inFile'" | tee -a $dvrPostLog
$cut "$inFile"

# Encode file to MP4 with handbrake-cli
echo "'$time' Transcoding started on '$inFile'" | tee -a $dvrPostLog
$handbrake -i "$inFile" -o "$tmpFile" --preset="Apple 1080p30 Surround" --encoder-preset="veryfast" -O

# **** OVERWRITE HERE ****
# Overwrite original ts file with the transcoded file
echo "'$time' File '$tmpFile' move started" | tee -a $dvrPostLog
echo "REDACTED" | sudo -S mv -f "$tmpFile" "$inFile"

#Remove lock file
echo "'$time' All done! Removing lock for '$inFile'" | tee -a $dvrPostLog
rm $lockFile

exit 0

https://redd.it/x7dtah
@r_bash

How to use sed command to replace values in a list & dictionary in a yaml file


I want to change the value of the seeds here.


seed_provider:

\- class_name: org.apache.cassandra.locator.SimpleSeedProvider

parameters:

\- seeds: "127.0.0.1:7000"


And want to change value of this :

data_file_directories:
\- /etc/test/config.yaml

https://redd.it/x7176q
@r_bash

Parameter Expansion for building cmd in for loop

Regarding building the source and target variables why does this work:

declare -a elemens=(
# hostname user filetotransfer file_suffix
"hostname1 username1 /path/to/file/on/remote suffix1"
"hostname2 username2 /path/to/file/on/remote suffix2"
)

for elem in "${elemens[@]}"
do
read -a strarr <<< "$elem"

source="${strarr[1]}@${strarr[0]}.hostname:${strarr[2]}"
target="./filename_${strarr[3]}"

scp -p $source $target
done


...and not just building them in the same line as scp as so:

scp -p "${strarr[1]}@${strarr[0]}.hostname:${strarr[2]} ./filename_${strarr[3]}"

With the later I just get the scp help message.
I realize it has something to do with variable expansion, but I do not understand why.

https://redd.it/x78gny
@r_bash

Count totals and correlate to $1

Hi all, I'm stumped by a problem and would love if I could get some help. I have a txt with lines and lines of data like this:

xxx.xxx.xx.xxx ftp ssh
yyy.yyy.yy.yyy ssh
zzz.zzz.zz.zzz smtp ftp

I need to count and correlate each service to the IP address, so the output would be similar to:

ftp count: 2
xxx.xxx.xx.xxx
zzz.zzz.zz.zzz

ssh count: 2
xxx.xxx.xx.xxx
yyy.yyy.yy.yyy

smtp count: 1
zzz.zzz.zz.zzz

I've been trying tons of stuff with awk but I'm getting nowhere and am afraid I'm deep down a rabbit hole. I think I need someone else's perspective on this one.

Anything you could give me to point me in the right direction would be awesome! Thanks!

https://redd.it/x6l7t5
@r_bash

Grepping for whitespace and double quotes as part of an if-then statement

I am attempting to create a noscript that will eventually run as a cron to check a conf file, and if it does not have a critical setting within it, to execute another noscript that regenerates the conf file.

The critical setting is: SecRuleEngine "On"

I can get the grep on my bash prompt just fine:

user@host ~# grep --color -h 'SecRuleEngine "On"' /path/to/conf-file.conf
SecRuleEngine "On"

And what I was wanting to do was to have a simple 1 for yes, 0 for no, like:

user@host ~# grep -c -h 'SecRuleEngine "On"' /path/to/conf-file.conf
1

Which I was hoping to feed into an if then statement in the noscript to test it, like:

if grep -c -h 'SecRuleEngine "On"' /path/to/conf-file.conf = "1" ; then
echo "all good"
fi

But on the line that starts with "if" I get:

: too many arguments

What I was hoping to do was get the noscript to see the grep as intended, then reverse it by putting a `!` after the `if` and then issuing the command to run the configuration noscript. What corrections do I need to make to my syntax?

[https://redd.it/xdmqhu
@r_bash

How do you export a mongo db into a JSON?

mongoexport --uri="mongodb://localhost:27017/" --username admin --password root --authenticationDatabase "admin" --db products --collection shoes --out dump.json

What's wrong with this command?

2022-09-02T18:00:24.492-0400 could not connect to server: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.

There shouldn't be any authentication. Looking inside Studio 3T, I see Authentication Mode: None.

https://redd.it/x6zz1k
@r_bash

Parsing a csv with Unix time and an integer on each line.

Hello,

My csv looks something like this:

timestamp,Value

1662353808788,5

1662354108786,12

.

.

.

Hundreds of lines, ordered by time.

My goal is for the noscript to return the biggest value recorded in the last 24hours and ideally if given a parameter should be also able to return the biggest value for a specific day. I imagined something like placing a 0 as a parameter for today, 1 for the day before, 2 for the day before that, but not set on that.

Any help appreciated, even if you could just tell me what I will need to accomplish it, some algorithm, pseudo code, commands I should look at. Never worked with unix time before.

Thank you

https://redd.it/x6cacz
@r_bash

printf vs echo - Showing last command

Hey all!

I've been writing a pretty basic noscript to take care of some file backup tasks for me. I'm trying to be good by commenting and producing log output. I'm having an issue getting the last command run to display in a function I've declared.

Here are the function declarations:

# Declare conditional logging
function success()
{
echo -e ${date} "$? ran successfully!" >> ${dockerDir}/shared/archive.log
}

function fail()
{
echo -e ${date} "\xF0\x9F\x92\x80" "$? failed..." "\xF0\x9F\x92\x80" >> ${dockerDir}/shared/archive.log
}

Pretty straightforward, right?

Here is one example of calling the functions with nested if statements:

#Tar 'em up
tar --create --bzip2 --file ${dockerDir}/shared/${archiveTar} ${dockerDir}${bookstack}*bookstackFiles.tar ${dockerDir}${bookstack_db}*bookstack_db.sql
if [ $? -eq 0 ]; then
success
else
fail
if [[ -e ${dockerDir}${bookstack}*bookstackFiles.tar || ${dockerDir}${bookstack_db}*bookstack_db.sql ]]
then
rm ${dockerDir}${bookstack}*bookstackFiles.tar
rm ${dockerDir}${bookstack}*bookstack_db.sql
EXIT
fi
fi

Now, here's the output from archive.log:

2022/09/13 20:54:29 0 ran successfully!
✅ 2022/09/13 20:54:29 \ Bookstack files tarred and bzipped successfully! 202209132054-bookstackArchive.tar.bz2 ✅
2022/09/13 20:54:29 0 ran successfully!
✅ 2022/09/13 20:54:29 \ Bookstack files encrypted successfully! 202209132054-bookstackArchive.tar.bz2.gpg ✅

The issue I'm trying to figure out is why it's not showing me in the log the last command that was run. In the first line, it should show something like "tar ran successfully!" (I'm not positive if it's going to include the command args or not). The third line should show something like "gpg ran successfully!".

Instead I'm just getting "0". I've tried different permutations of this using printf, echo -e, echo. I've tried different ways of referencing the previous command like $?, !:0, referencing a different variable I declare earlier in a trap statement.

Nothing is giving me the expected output. Thoughts?

https://redd.it/xds4z6
@r_bash

File Metadata "Comment"

How to read the metadata "comment" of a file?

https://redd.it/xdw131
@r_bash

[OC] shortbashpwd: prompt for bash with abbreviated directory names like in fishgi
https://github.com/NikitaIvanovV/shortbashpwd

https://redd.it/x60spv
@r_bash

What does this command do in .zsh file

I am using the docker-compose plugin in zsh and want to use docker compose v2 commands. so when exploring the plugins I found this line

(( ${+commands[docker-compose]} )) && dccmd='docker-compose' || dccmd='docker compose'

can someone explain what does this do?

https://redd.it/xdxqvd
@r_bash

Inserting variables under the third column in a textfile.

So i have two columns and would like add text to the Value3 column.

i've tried code below which works for the first two columns. But not for the third.

Help or suggestions is appreciated.

echo -e "" "val1 $numb1" "\n" "val2 $numb2" | paste -d "\t" /files/test.txt -

Columns

Value1 Value2 Value3
val1 34 val1 55
val2 34 val2 55

https://redd.it/x5y5wa
@r_bash

Linux Tech eBook Bundle by No Starch Press
https://www.humblebundle.com/books/linux-no-starch-press-books?partner=indiekings&charity=1542725

https://redd.it/xe09me
@r_bash

can someone what did I overlook

what is the purpose of the noscript: unlocking the dataset using the curl command

the code that works:

curl "`https://$host/api/v2.0/pool/dataset/unlock`" -k -X POST -H "accept: */*" -H "Content-Type: application/json" -H "Authorization: Bearer 1-$API_TOKEN" -d '{"id": "the_dataset_hardcoded","unlock_options": {"key_file": false,"recursive": false,"toggle_attachments": true,"datasets": [{"name" : "the_dataset_hardcoded" , "passphrase" : "the_password_hardcoded"}]}}'

now this one works flawlessly, but the moment i try to replace the "the_dataset_hardcoded" and/or the "the_password_hardcoded" as an environmental variable the noscript stops working

my next try is to create the part as a string that i can echo out for debugging so i create:

curl_builder3="'{\"id\": \"$pool\",\"unlock_options\": {\"key_file\": false,\"recursive\": false,\"toggle_attachments\": true,\"datasets\": [{\"name\" : \"$pool\" , \"passphrase\" : \"$pass\"}]}}'"

and run it like this:

curl "`https://$host/api/v2.0/pool/dataset/unlock`" -k -X POST -H "accept: */*" -H "Content-Type: application/json" -H "Authorization: Bearer 1-$API_TOKEN" -d $curl_builder3

but what i get for this is:

curl: (3) unmatched brace in URL position 1:

{"key_file":

^

bash: line 3: Server: command not found

&#x200B;

but when I echo the created env variable i get the correct string that if i copy and run in terminal it does what is supposed to do

&#x200B;

now what did I miss

&#x200B;

&#x200B;

edit:

the curl command is now fixed and it looks like this:

curl "`https://$host/api/v2.0/pool/dataset/unlock`" -k -X POST -H "accept: */*" -H "Content-Type: application/json" -H "Authorization: Bearer 1-$API_TOKEN" -d "{\"id\": \"$pool\",\"unlock_options\": {\"key_file\": false,\"recursive\": false,\"toggle_attachments\": true,\"datasets\": [{\"name\" : \"$pool\" , \"passphrase\" : \"$pass\"}]}}"

&#x200B;

https://redd.it/x5v910
@r_bash

Launch an app from a noscript

Hi all,

&#x200B;

I'm making a noscript in which i need to check if an app is launched and if it's not, launch it from the noscript.

&#x200B;

The app is a .sh file, i can use the bash command to launch it but if i do the noscript doesn't stop until the app is shut down which is not what i'm looking for.

&#x200B;

Basically i would like to run this noscript every 5 minutes or so, that's why i need a command that just launches the app and not runs it in the noscript. Hope i'm making sense here.

&#x200B;

Any help is appreciated i'm still new to noscripting, thanks for reading !

https://redd.it/xe5egk
@r_bash

Need help with loop in a bash file

I have a bash noscript which needs to do the following:

change to folder /repos
loop through all the subfolders (git repos) and 
    run command "git config --bool core.bare true"

This is what I have so far:
for dir in ~/repos/*; do (cd "$dir" && git config --bool core.bare true); done

https://redd.it/x5g916
@r_bash

Need help improving my first Bash Script

&#x200B;

What I wanted to do

* Have this scrip start every 10 minutes
* Exit the noscript if it is already running
* Copy new files to a network folder based on file creation date
* Verify the copy was successful
* Email me on completion and either a message of no errors or a list of files that did not copy correctly
* Unmount the cards

What I have so far

* A noscript that when run with the SD cards in it identifies the type of camera by card name
* Copies all the video files from camera 1's memory card to a network share
* Copies all the video files from camera 2's memory card to a network share
* Copies all the GPS data from camera 2's memory card to a network share.
* Unmount the cards

&#x200B;

I should have 6 hours of time available to copy before I have to take the cards and put them back in the camera.

My code is below.

&#x200B;

#!/bin/bash
clear
echo "Checking for SD Cards"
sleep 2
echo "checking for Lamnando Card"
sleep 5
if [ -d "/media/dashcam/6310-5AE7/rearVideo1" ]; then
echo "SD Card for Lanmando is inserted"
echo " Copying Files Front Camera Files"
rsync -a -P /media/dashcam/6310-5AE7/frontVideo0/* /synology-nas/shared/DashCam/
echo "Complete"
echo " Copying Rear Camera Files"
rsync -a -P /media/dashcam/6310-5AE7/rearVideo1/* /synology-nas/shared/DashCam/
echo "Complete"



sleep 2
else
echo "Lanmando Card Not Found"
fi
echo "Checking for VaVa Card"
if [ -d "/media/dashcam/VA-VD009/VAVA" ]; then
echo "SD Card for VaVa is inserted"
echo "Copying Video Files"
rsync -a -P /media/dashcam/VA-VD009/VAVA/MOVIE/* /synology-nas/shared/DashCam
echo "Complete"
echo "Copying GPS data"
rsync -a -P /media/dashcam/VA-VD009/VAVA/GPSDATA/* /synology-nas/shared/DashCam
else
echo "VaVa Card Not Found"
fi
echo "unmounting cards"
sleep 3
if [ -d "/media/dashcam/6310-5AE7/rearVideo1" ]; then
umount /media/dashcam/6310-5AE7
fi

if [ -d "/media/dashcam/VA-VD009/VAVA" ]; then
umount /media/dashcam/VA-VD009
fi
sleep 3
echo "All cards unmounted you can now safely remove them"

https://redd.it/x444uu
@r_bash

https://github.com/dzove855/Bash-Timer - Simple Bash timer

A simple Bash Timer which allows the show the execution and return code of each executed command.

https://redd.it/x3wc19
@r_bash

Script to unpack rar archives in multiple sub folders using bsdtar or p7zip?

I'm using a filezilla docker to download folders that each contain a set of rar files. I had a noscript that used unrar that filezilla would run when the download finished.

Unfortunately the docker is based on alpine linux and no longer has unrar in its repos. So my two options are bsdtar or p7zip and I haven't quite figured out how to get them to replace my previous noscript.

With unrar the line I used was unrar x -r -o- *.rar and it would go through all the folder and unpack all the archives.

7zip has a recurse option, but i can't get it to work. 7z x -r -aou *.rar is what I've tried.

Any suggestions?

https://redd.it/xeg082
@r_bash

How can I write a noscript that replaces all occurrences that match a pattern in all files within a directory and sub directory?

I am a frontend developer and we were using react-bootstrap 3 which had export a component Clearfix.
We have been importing that using

import {Clearfix} from "react-bootstrap" or import {Clearfix, (some other components separated by comma)} from "react-bootstrap"

We upgraded to react-bootstrap 4 but it doesn't export that component anymore and we have built a custom component for that. but because we have used the one from bootstrap in hundreds of files I don't wanna go and replace all one by one.
Is there a way to write a noscript that it will replace

import {Clearfix} from "react-bootstrap"

with

import {Clearfix} from "my/custom/path"

and if it finds

import {Clearfix, (some other components separated by comma)} from "react-bootstrap"

it just removes Clearfix from here and adds a new line below it with

import {Clearfix} from "my/custom/path"

https://redd.it/x3wrm3
@r_bash