mothballing a compressed file... "max xz tarballs... xz" extended

In a prior note.. I showed a simple one line to compress, directories.

but what happens when you need to make the compressed file not obviously a tarball..   ie. moving the file from system to system triggers huge delays due to virus scanning trying to open the file up, but eventually giving up due to timeouts..  it's a waste of time and the files were scanned on the origional machine prior to compression and will be scanned at the final machine as the tarball contents are written out the the drive again.

the answer is simple..   pipe it through base64

XZ_OPT=-9e tar cJf tarfile.tar.xz directory 

becomes

XZ_OPT=-9e tar cJf directory | base64 - > /tarfile.tar.xz.base64

In my case, the tar and .xz are giveaways for the scanner as well, so rename the contents as approprate...

References:

http://strayink.blogspot.com/2016/06/max-xz-tarballs-xz.html

visudo with nano

export EDITOR=nano && sudo -E visudo

 from 

http://ubuntuforums.org/showthread.php?t=790286 

max xz tarballs... xz

XZ_OPT=-9e tar cJf tarfile.tar.xz directory

from

http://unix.stackexchange.com/questions/28976/how-to-xz-a-directory-with-tar-using-maximum-compression

list all shell variables

( set -o posix ; set ) | less

From :

https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values/356973

Disable rdp printer mapping

HKLM\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services

RegDWORD  fDisableCpm   1

From
https://blogs.technet.microsoft.com/rmilne/2014/06/02/how-to-disable-remote-desktop-printer-mapping/

List logged in users on Windows

query user /server:$SERVER

From

http://stackoverflow.com/q/23219718/4611335

Bash obfuscation

I need to place info in scripts that shouldn't be readily obvious.  This isn't encryption, it's just obfuscation.   Some of the inspiration for this came from random readings of bash manuals and discussions and one part directly from http://www.commandlinefu.com/commands/view/6066/convert-ascii-string-to-hex.  Because of a personal preference to awk, I converted the commands.

To Hide Varables

Run the following command with the command or info you want obscured.


1 ) encode the STRINGTOHIDE

echo "STRINGTOHIDE" | base64 -w 0

U1RSSU5HVE9ISURFCg==

2)  encode the varable

echo -n 'HIDDENVARABLE="`echo U1RSSU5HVE9ISURFCg== | base64 -di`"' | xxd -ps |  awk '{gsub(/[[:xdigit:]]{2}/,"\\x&");print}'

\x48\x49\x44\x44\x45\x4e\x56\x41\x52\x41\x42\x4c\x45\x3d\x22\x60\x65\x63\x68\x6f\x20\x55\x31\x52\x53\x53\x55\x35\x48\x56
\x45\x39\x49\x53\x55\x52\x46\x43\x67\x3d\x3d\x20\x7c\x20\x62\x61\x73\x65\x36\x34\x20\x2d\x64\x69\x60\x22

3)  check it

eval $'\x48\x49\x44\x44\x45\x4e\x56\x41\x52\x41\x42\x4c\x45\x3d\x22\x60\x65\x63\x68\x6f\x20\x55\x31\x52\x53\x53\x55\x35\x48\x56\x45\x39\x49\x53\x55\x52\x46\x43\x67\x3d\x3d\x20\x7c\x20\x62\x61\x73\x65\x36\x34\x20\x2d\x64\x69\x60\x22'

echo $'HIDDENVARABLE'
HIDDENVARABLE


To hide a command,

1)
echo -n 'HIDDENCOMMAND' | xxd -ps | awk '{gsub(/[[:xdigit:]]{2}/,"\\x&");print}'

 echo -n 'ls -lah' | xxd -ps | sed 's/[[:xdigit:]]\{2\}/\\x&/g'
\x6c\x73\x20\x2d\x6c\x61\x68

eval $'\x6c\x73\x20\x2d\x6c\x61\x68'

eval $'hex string'

Simple shell based smtp client... really simple..

I first found email as part of cygwin.   This is a quick and simple tool for sending email.  I use it to send status messages and attaching scheduled reports from shell scripts.  it just works.  no fuss, no local mta setup.

I've installed, it into Ubuntu server, and Raspbian and had it work, as well as using it regularly in Cygwin.  The man page is really thorough.  There are optional config files, but I typically specify everything on the command call, just personal preference.

My quick script for installing dean jone's email program, based on his site. (ok, i added a line to get the parts to compile, otherwise it's his commands to pull and build.  Being that this blog serves as my personal annotated bookmarks, I hope he doesn't mind)

#!/bin/bash sudo apt-get install git gcc make git clone --recursive https://github.com/deanproxy/eMail.git cd eMail ./configure make sudo make install
In my scripts, I put the following at the top

email_from_address=some@email.address
email_from_name="someones name"
email_server=address or ip
email_port=a port
email_recipients=someoneelse@email.address,different@server.elsewhere

and then where I want to send email,

echo "
 message here 

white space is sent so layout your message as you want it
to look

great!
" | email -n $email_from_name -f $email_from_address -s "subject" -r
$email_server -p $email_port $email_recipients


that's it..

If bcc is needed, do --bcc before the final recipients list and and a list of recipients

My NoIP2 init script and user setup

I recently had to setup a NoIP2 client on a system for which no package exists.  As it's a Debian based distro, I felt pretty safe using the instructions for Ubuntu 12.04 LTS.(about part way down the page).  In short, it's a locally compiled process.  This is great for getting a running client, but the scripts I could find for automatically starting the client at reboot weren't quite what I thought acceptable.  When I tried to finish the setup with update-rc, there were errors about missing lsb tags and the client was running under an id with more privileges than it should ever have.

I created a user named noip, limited its access to the two files necessary to run and then wrote a lsb compliant init script and installed it.


Download complete script from here.


#!/bin/bash # # Copyright (C) <2014> <john W. Gill> # # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), # to deal in the Software without restriction, including without limitation # the rights to use, copy, modify, merge, publish, distribute, sublicense, # and/or sell copies of the Software, and to permit persons to whom the # Software is furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE. # commandname=noip2 initscriptlocation=/etc/init.d/$commandname commandlocation=/usr/local/bin/$commandname commanduser=noip # add a user for this process useradd $commanduser echo '#!/bin/sh ### BEGIN INIT INFO # Provides: no-ip2 # Required-Start: $all # Required-Stop: # Should-Start: # Should-Stop: # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: no-ip2 rcscript ### END INIT INFO name=noip2 user=noip command=/usr/local/bin/$name config=/usr/local/etc/no-ip2.conf start() { pid=` ps axo pid,command | grep $command | grep -v grep | cut -d " " -f2 ` if [ -z "$pid" ]; then echo "Starting $name." chown $user $command chmod 711 $command chown $user $config chmod 111 $command $command -c $config echo else echo "$name already running." fi } stop() { pid=` ps axo pid,command | grep $command | grep -v grep | cut -d " " -f2 ` if [ -n "$pid" ]; then echo "Stopping $name." $command -K $pid else echo "$name was not running." fi } status() { pid=` ps axo pid,command | grep $command | grep -v grep | cut -d " " -f2 ` if [ -n "$pid" ]; then echo "$name running." else echo "$name stopped." fi } case "$1" in start) start && status ;; stop) stop && status ;; restart|reload|try-restart|force-reload) stop && status && start && status ;; status) status ;; *) echo $"$name Control script. Usage: $0 {start|stop|restart|reload}" exit 1 esac exit 0 ' > $initscriptlocation chmod 711 $initscriptlocation update-rc.d $commandname defaults

The not so great thing about standards.

It has been my personal experience that standards are great! They define things, simplify interactions, and reduce learning time. Except when they don't. A simple standard is easy to understand. If it's well written, it can even work. But as standards become more complex it is not so easy to make sure that all situations are accounted for.

Every big standard has “wiggle” room in it. Don't believe me? They why are there so many different implementations of individual standards. It's this “wiggle” room that is so appealing to so many of the companies and organizations that implement a standard. It provides for a perceived competitive advantage, regardless of the reality.

Standards are all around us, but we've become so accustomed to them that we seldom take notice. For instance, I own two DVD players. Each remote is different. The simple standard like a “play/pause” button is there, but they are not in the same place or even laid out the same. On one remote the play is a little button and the navigation buttons are arranged in a circular pattern with the “select/ok” in the middle. On the other remote the “play, stop, forward, rewind” buttons are arranged in a circular pattern and the navigation buttons are buried down in the middle of other nondescript buttons. Of course the simple solution would be to buy two universal remotes and stop thinking about it, but that's too easy and covers the symptoms.

So what's the solution? Unfortunately there isn't an all encompassing one. As we go about out lives we make choices that tend to limit what “wiggle” room we as consumers will take. This tends to promote those good implementations, though not always. Sometimes a good marketing campaign can overcome weaknesses in implementation, generating enough consumer desire that a bad implementation will dominate the market. This results in a slew of “me-to” re-implementations. The world hiccups a bit, and then goes on with life.

Standards are great! Really. There are so many to choose from and so much “wiggle” room in them that any two implementations of a tool (ie. dvd remote, cellphone charger, etc.) accomplish the same task, while being significantly different. It's all in what makes life more effective and convenient to us the consumer/user.

Oh, it's that "wiggle" room that keeps most of us employed. Trying to make all the different standards and nonstandard implementations work together is expensive. So standards are great, except that they aren't all that standard, and don't always solve the problem that they were devised to solve.

BTW: Today I picked on DVD players, mostly because picking on other things like cars or computers didn't interest me at the moment. It would be just as easy to make the same points about them, but I'll let you think about it instead.

My OpenELEC & MythTv (mythbuntu12.04 LTS) setup

The real key is step 6 the other steps were needed, but it was step 6 that makes the difference!

I'm now running Raspberry Pi frontends with a MythTV backend.  In short it works. But you need the decoder licenses and willingness to seriously overclock it. 

WARNING I VOIDED THE WARRANTY ON MY Pi TO MAKE THIS WORK. I'M NOT RESPONSIBLE FOR ANY CONSEQUENCES OR EVENTUAL REPERCUSSIONS.

Beyond the obvious, "Can I do it" question, my other driver was to cut power. Ok it's not a great savings, the power cost of a 500+ watt computer is small compared to a refrigerator or a air conditioner.  In contrast, when comparing the aforementioned computer's power demands and heat generated, a Raspberry Pi is negligible, even when running as hard as I'm pushing them.  And not having fan noise, or extra heat in the media cabinet was a plus as well.

Now, go get the decoder licenses. It took about 5 hours for them to come, so ordering them early isn't bad. Also I'm using a powered usb hub that is capable of powering the pi and serving three usb ports for processing you will need this. I'm using this sd card, some people have problems, so get two and keep a hot backup image, just in case.  And finally, for a remote control I'm using StreamZap. I did try a Logitech mini diNovo but for now wanted the look of a typical remote. Extra stuff: male to male audio cable, for music playback without using the tv; usb extension cables to get the usb Wi-Fi dongle away from the tv; Tenda 300Mbps Dual Band Wireless N USB Adapter (W522U), heat sinks.

(all the above links are to the actual items I used, except for the audio cable. I bought mine at a local parts store years ago.)

Steps

1. Install OpenELEC. 
2. Fix the double press issue with the remote.
1. Windows, 
1. open an explorer window to the pi and go to the Configfiles folder.
2. create a folder for udev.rules.d.
3. create a file named remote-control-lirc.rules and put the following in it as one line. SUBSYSTEM=="rc", ATTRS{protocols}=="*rc-5-sz*" RUN+="/bin/sh -c 'echo rc-5-sz > /sys$env{DEVPATH}/protocols'"
2. SSH
1. cd or mkdir to /storage/.config/udev.rules.d
2. follow step 3 above
3. Change shutdown button to reboot. 
1.  Windows
1. explorer...  Userdata\keymaps
2. create a file called keymaps.xml and add:
   <keymap>
     <global>
       <remote>
         <power>XBMC.Reboot()</power>
       </remote>
     </global>
   </keymap>
2. SSH
1. /storage/.xbmc/userdata/keymaps
2. step 2
4. Overclock
1. your on your own, but here's my settings
   gpu_mem=100
   gpu_mem_512=128
   arm_freq=1000
   arm_freq_min=900
   core_freq=500
   core_freq_min=450
   gpu_freq=600
   sdram_freq=600
   over_voltage=7
   over_voltage_min=7
   current_limit_override=0x5A000020
   force_turbo=1
   hdmi_force_hotplug=1
   hdmi_group=1
   hdmi_mode=5
   hdmi_ignore_cec_init=1
   
5. Adjust the XBMC advanced settings
1. Windows
1. Userdata
2. advancedsettings.xml
   <advancedsettings>
   
     <network>
       <cachemembuffersize>134217728</cachemembuffersize>
     </network>
   
   </advancedsettings>
   
2. SSH
1. /storage/.xbmc/userdata/advancedsettings.xml
6. Adjust the Network Buffer Size! (CRITICAL)
1. Windows
1. Configfiles\sysctl.conf
   
   net.core.wmem_max=67108864
   net.core.rmem_max=67108864
   net.core.rmem_default=67108864
   net.core.wmem_default=67108864
   
   net.ipv4.tcp_rmem= 67108864 67108864 67108864
   net.ipv4.tcp_wmem= 67108864 67108864 67108864
   
    
2. SSH
1. /storage/.config/sysctl.conf
7. config the cmyth pvr plugin.

Firefox sync server on a Raspberry Pi running Raspbian: Update. One step install

This is my one step install for weave-minimal.  Originally for Raspbian, it seams to work on kubuntu as well.

In my previous post I documented a way to get a Firefox sync server running on a Raspberry Pi.  After some re-working of things I started that system over and discovered that I'd made a few mistakes in my documentation.  Rather then make the same mistakes again, I re-documented in the a form that guarantees, that all the steps are tested and working.  Here's my bash install script that I used to re-install the weave-minimal server on my clean copy of Raspbian .


Download complete script from here:

#!/bin/bash # # Copyright (C) <2013> <john W. Gill> # # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), # to deal in the Software without restriction, including without limitation # the rights to use, copy, modify, merge, publish, distribute, sublicense, # and/or sell copies of the Software, and to permit persons to whom the # Software is furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in # all copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE. # initscriptlocation=/etc/init.d/weave-minimal apt-get install python-setuptools easy_install -U weave-minimal # init information from readme :: # https://github.com/posativ/weave-minimal#setup-and-configuration # # WARNING! # accept from all hosts.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! # Understand, that there may be good reasons not to allow this. In my case, the # server isn't actually hosted on the internet, just internally. # YOYO applies useradd www echo '#!/bin/sh ### BEGIN INIT INFO # Provides: weave-minimal # Required-Start: $all # Required-Stop: # Should-Start: # Should-Stop: # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: weave-minimal firefoxsync service ### END INIT INFO name=weave-minimal user=www command=/usr/local/bin/weave-minimal host=--host=0.0.0.0 logfile=/var/log/$name.log pid=/var/run/$name.pid port=8080 dbPath=/var/lib/weave-minimal/ start() { pidnumb=`cat $pid` if [ -n $pidnumb ]; then pidstatus=`ps aux | grep $pidnumb | grep $command` else pidstatus= fi if [ -z "$pidstatus" ]; then echo "Starting $name." touch $pid chown $user $pid touch $logfile chown $user $logfile if [ ! -d $dbPath ]; then mkdir -p $dbPath fi touch $dbPath chown $user $dbPath start-stop-daemon --start --background --pidfile $pid \ --chuid $user --make-pidfile --exec $command -- --data-dir=$dbPath \ $host --port=$port --enable-registration --log-file=$logfile else echo "$name already running." fi } stop() { echo "Stopping $name." start-stop-daemon --stop --retry TERM/5/TERM/5/TERM/5/KILL/5 --pidfile $pid } status() { pidnumb=`cat $pid` if [ -n $pidnumb ]; then pidstatus=`ps aux | grep $pidnumb | grep $command` else pidstatus= fi if [ -n "$pidstatus" ]; then echo "$name running." else echo "$name stopped." fi } if [ ! -d $DBPATH ]; then mkdir /var/lib/weave-minimal chown $USER /var/lib/weave-minimal fi case "$1" in start) start && status ;; stop) stop && status ;; restart|reload|try-restart|force-reload) stop && status && start && status ;; status) status ;; *) echo $"$name Control script. Usage: $0 {start|stop|restart|reload}" exit 1 esac exit 0 ' > /etc/init.d/weave-minimal chmod 711 /etc/init.d/weave-minimal update-rc.d weave-minimal defaults
After installing, on the Pi,

 I went to Tools/Sync and entered the following:


(Incidently, the server is the ip address of the my Pi.  I use dhcp address reservation on to assign it the same address every time.)

after clicking next I was rewarded with