Parallel compression – decompression of files on multi-core-multi-threaded cpu’s

Abstract

Using multiple cores and processors simultaneously to achieve faster compression and decompression rates is possible nowadays with the new generation of multi-core cpu’s. Using the following methods to create compressed backups of your files will be less time consuming. Continue reading “Parallel compression – decompression of files on multi-core-multi-threaded cpu’s”

SHC: How to encrypt and convert a shell script into a binary executable

Abstract

SHC is a free software (GPL v2) that takes a shell script and produces C source code. The generated source code is then compiled and linked to produce a stripped binary executable.

Introduction

There are some uncomfortable times when you will be asked to not distribute the source code of a shell script (eg. automation script, deployment script, maintenance etc ) that you wrote for a job you have taken as a freelancer or as a sysadmin for a private institution. This is where SCH comes in rescue.

SHC creates a stripped binary executable version of the script specified with -f on the command line. If you supply an expiration date with the -e option the compiled binary will refuse to run after the date specified. The message “Please contact your provider” will be displayed instead. This message can be changed with the -m option.

Materials and Methods

You can use the SHC once you install some dependencies and run a make and install

Install Required Packages

Install required packages for SHC compiler. For Debian/Ubuntu

sudo apt-get install libc6-dev 

Download and Build

Download the latest source code of SHC compiler:

Extract the zip and compile the SHC source code on your system and install it using the following commands.

cd where-the-extracted-is
make 
sudo make install

As of Debian 8.0+ (Jessie) and Ubuntu 15.04+ (Vivid) SHC (version 3.8.7-2) is available from the repositories. So if you are on any of this and you dont mind any bugs available on this older version (see Version bug fixes) you can simply install it by running:

sudo apt install shc

Usage

Use the following command to create a binary file of your script.sh

shc -T -f script.sh

OPTIONS

The command line options that are aalso available are:

-e date: Expiration date in dd/mm/yyyy format [default:none]

-m message: message to display upon expiration [default: “Please contact your provider”]

-f script_name: File name of the script to compile

-i inline_option: Inline option for the shell interpreter i.e: -e

-x comand: exec command, as a printf format i.e: exec(‘%s’,@ARGV);

-l last_option: Last shell option i.e: —

-r Relax security. Make a redistributable binary which executes on different systems running the same operating system.

-v Verbose compilation -D Switch on debug exec calls

-T Allow binary to be traceable (using strace, ptrace, truss, etc.)

-C Display license and exit

-A Display abstract and exit

-h Display help and exit

Results and Discussion

SHC itself is not a compiler such as cc, it rather encodes and encrypts a shell script and generates C source code with the added expiration capability. It then uses the system compiler to compile a stripped binary which behaves exactly like the original script. Upon execution, the compiled binary will decrypt and execute the code with the shell -c option. Unfortunatelly, it will not give you any speed improvement as a real C program would.

The compiled binary will still be dependent on the shell specified in the first line of the shell code (i.e. #!/bin/sh), thus shc does not create completely independent binaries.

SHC’s main purpose is to protect your shell scripts from modification or inspection. You can use it if you wish to distribute your scripts but don’t want them to be easily readable by other people.

References

SHC is a tool writen by Francisco Rosales Garcia and distributed under the terms of GNU GPL version 2

  1. Author: Francisco Rosales Garcia

Introduction to DIY Home Server

Abstract

A home server can be extremely useful for managing your personal data like backing up your files, streaming your music, photos and videos and accessing locally installed services like webmail, webeditor, calendar, notes, contacts, finances and many more.

This is an introductory article for a series of articles for building, deploying, securing, maintaining, troubleshooting and administering your home server or any server in general.

Introduction

A server, either external (hosted on 3rd party providers) or internal (in your home) is a computer dedicated to run an instance of an application (software) capable of accepting requests from the client (your PC, smartphone/tablet, TV etc.) and giving responses accordingly. Servers can run on any computer including dedicated computers, which individually are also often referred to as “the server”. Note that in theory, any computerized process that shares a resource to one or more client processes is a server.

Materials and Methods

Before we get started, I would suggest that you should get a member of your family (kids, brother/sister, grandma? ) or a friend involved to the DIY Home Server project because it will be more fun and entertaining than to work alino on the project.

To be able to build a home server we need to understand some terms:

  • Hardware
  • Network
  • Operating System
  • Clients

Hardware

The computer that will take the role of a home server could be any spare computer lying around. Unfortunately though this would mean that the this computer will have to be 24/7 powered on, could be loud and the consumption of electricity would make it not viable for the role. That is why I suggest avoid using common computers for home server role but instead buying a credit card-sized single-board computer like:

This type of computers are low on power consumption, silent and generally provide more “value for money”. The total return of investment exceeds the one that will be provided by a common computer.

Network

The network is the “glue” that connects our server with the various clients (as mentioned earlier).

The network is consisted of

  • Router
  • Cables
  • IP assignments

Our router is actually a mini internet/dhcp server and its role is to provide Internet connection, IP addresses and data/requests deliverance, among other things through the cables to the the clients and vice versa.

Operating System

There are specialized free (as in beer and freedom of speech) operating systems for the server role that are compatible with the single-board computers that I mentioned earlier. Most popular among them are:

  • Ubuntu
  • Debian
  • Arch Linux
  • openSUSE

Personally I prefer Ubuntu because of their predefined development cycle and the 5 years of support on software updates.

You should choose the one that you are comfortable administering as this will be installed on the server for years to come.

Clients

Clients are any devices that connect and exchange information with our home server. The kind of exchange depends on the types of services that the server provides and the capabilities of the device that request those services. A home server usually provides:

  • Centralized storage
  • Media serving
  • Information (calendar, contacts etc.)
  • E-mail
  • Security monitoring
  • P2P file sharing

Results and discussion

Now that we’ve discussed the core terms of building a home intranet, we should be able to create a draft plan of the parts that we will need to succeed in our project.

I should mention that building a home server not only pays off but it is fun and educative. So stay tuned for the next article either by subscribing to newsletter or following me at social nets.

References

  1. Server definition : wikipedia
  2. Home Server : wikipedia
  3. Intranet definition: wikipedia

Create a backup or a bundle of your git repository

Introduction

Keeping a backup or moving around a local repository while retaining its entire versioning information can be achieved with the current utilities of git.

Bundles are a great way to backup entire Git repositories. They also let you share changes without a network connection.

Materials and Methods

We will use the git bundle utility inside our local repository and also git merge to deploy our bundled repo.

git bundle turns a repository into a single file that retains the versioning information of the entire project.

Here are a list of options that are available:

git bundle create <name-of-the-bundle> <git-rev-list-args>
git bundle verify <file>
git bundle unbundle <file> [<refname>...]

Create a bundle

While you are inside a repo run:

git bundle create ../myrepo.bundle master

It should print out something like this:

Counting objects: 43, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (39/39), done.
Writing objects: 100% (43/43), 54.44 KiB | 0 bytes/s, done.
Total 43 (delta 8), reused 0 (delta 0)

This will create the bundle outside (one folder above) the repo and name itmyrepo.bundle. When we created the file It was like we where just pushing our master branch to a remote, except it’s contained in a file instead of a remote repository.

Verify the bundle

Before we move our bundle its a good idea to verify its content and check that the bundle file is valid and relevant to the current repository.

While you are inside a repo run:

git bundle verify ../myrepo.bundle

It should print out something like this:

The bundle contains this ref:
230b04bdd3367b2db73 refs/heads/master
The bundle records a complete history.
../myrepo.bundle is okay

Deploy the bundle

Assuming that you have the myrepo.bundle file in /home/user and you have created a folder newrepo and initialized a git repository inside it. Here is a generic procedure:

mkdir newrepo
cd newrepo
git init
git bundle unbundle ../myrepo.bundle

the last command will inform you with the HEAD of you repo

f7243ba54eb7de4b76a0 HEAD

Now you can merge the contents of myrepo.bubdle file with the one that you initialized inside the newrepo folder:

git merge f7243b

Alternative procedures are also available, so please follow the links provided on theReferences section

Results and discussion

As you can see once you create the bundle file you can easily move it to another folder on the same or an other machine. Assume you want to transfer the history from a repository R1 on machine A to another repository R2 on machine B. For whatever reason, direct connection between A and B is not allowed, but we can move data from A to B via some mechanism (USB flash drive, email, etc..). This way we can update R2 with development made on the branch master in R1.

References

  1. Ryan Hodson, Git Tips & Tricks : rypress.com
  2. Scott Chacon and Ben Straub, Pro Git : git-scm.com

Ask – Find the best answer to your questions related to Ubuntu in just 1 second !!!

Ah, it’s almost 9:00 and I am still awake. I’ve been trying to figure out how great it’s the howdoi and what if I could modify the script to work along with AskUbuntu.com

How it works

Just type your question (query or just a phrase) or anything like being in Google, but in terminal.

The script will search using Google to find the appropriate AskUbuntu links and throw away the rest of them. Then, it will pick up the first one (use the -n <number> parameter to specify how many answers do you want to see), browse into the code tab, grab the first top voted-reply and throw away the garbage. If you want just the links, use the -l parameter.

[button color=”green” link=”https://github.com/drpaneas/ask/archive/master.zip”%5DDownload Ask[/button][button color=”blue” link=”https://github.com/drpaneas/ask”%5DFork it Github[/button]