Boolean World – Boolean World

Arrays in C Programming with Examples

Tue, 19 Nov 2019 15:54:45 +0000

Arrays in C allow you to store multiple items of the same data type, such as a list of integers. Arrays form the basis for many data structures and allow you to build advanced programs.

In this article, we are going to discuss what an array is and how you can use them, along with examples. We will also see the concept of “strings”, which is closely related to the topic of arrays.

What is an array in C and why should you use them?

In C, an array is a way to store a fixed number of items of the same data type under a single name. Each data item of the array can be accessed by using a number called an “index” or “subscript”.

You might think, why do we need arrays to store multiple data types, when you can just declare normal variables? Let us take an example — suppose, you have been asked to write a program that takes in the temperature for the last 90 days and perform some processing on that data.

With enough time and patience, you can declare 90 integer variables (such as a1, a2, a3 and so on) to store this information. However, processing the information in these 90 variables is nearly impossible. What if you wanted to know the average temperature between a range of days (such as the 10th and 25th day), where this range would be given by the user?

While you can’t do this with normal variables, you can use arrays to make such a program. You can store all the temperature values under a single variable like a, and then extract any set of values from it by using the index.

Declaring and using an array in C

To declare an array, you simply need to specify the data type of the array elements, the variable name and the array size. For example, if you want to declare an integer array with four elements, you’d use:

int a[4];

This statement allocates a contiguous block of memory for four integers and initializes all the values to 0. This is how it is laid out in memory:

Array indexes start from zero and end with (array size - 1). So for the above array, you can use the first element with a[0], second element with a[1], third element with a[2] and fourth (last) element with a[3].

You can use the indexes to set or get specific values from the array. Here are a few examples:

a[0] = 10;
a[1] = 20;
a[2] = a[1] / a[0]; // a[2] will be set to 20/10 = 2
a[3] = a[1] - 2; // a[3] will be set to 20-2 = 18

After these changes, here is how the array will look like in memory:

Apart from the square brackets to indicate the index, array elements behave like normal variables. So, for example, you can print them by using:

printf("%d %d %d %d\n", a[0], a[1], a[2], a[3]);

You can see the full program in action below, or download it here.

One important thing to note is that C does not enforce any array bounds checks, and accessing elements outside the maximum index will lead to “undefined behaviour”. So, in the above example, trying to get or set a value like a[5] can cause your program to crash or behave abnormally.

Alternative ways to initialize arrays

Previously, we’ve seen how to declare an array and set its elements. However, if you know the elements of the array, then there is an easier way to declare the array. For example, you want to declare an integer array with the values 10, 20, 30, 40, you can use the “initializer list” syntax:

int a[4] = {10, 20, 30, 40};

This statement will automatically create an array of size 4, and initialize a[0] to 10, a[1] to 20 and so on. To confirm that this works, you can print the variables, just like we did in our previous program.

You can also make an array that is bigger than the initializer list, like so:

int a[6] = {10, 20, 30, 40};

In this case, the rest of the elements are initialized with zero. In our above example, elements from a[0] to a[3] will be initialized, whereas a[4] and a[5] will be set to zero. Again, you can easily verify this by writing a program:

You can also skip the array size when using initializer lists, like in the example below:

int a[] = {10, 20, 30};

The C compiler automatically guesses the size of the array from the size of the initializer list. So, the above example is the same as writing int a[3] = {10, 20, 30};

However, you cannot skip both the size and the initializer list, and write int a[];. Arrays in C are of a fixed size, and so their size must be known when you create the array. Therefore, if you skip both of them, C cannot create the array, and this will lead to a compile-time error.

Looping/iterating over an array in C

Previously, we have learnt how you can use the individual elements of an array. Since the array indexes are integers starting from 0 to (array size - 1), you can use loops to visit all the elements of the array. This process of visiting all the elements of an array is also called “traversing an array”.

Here is a very simple example. It initializes an array and prints each array element and the index:

In the for loop, the value of i starts with 0 (because we set i = 0) and ends at 3 (because the loop continues till i < 4). Inside the for loop, we print the value of a[i]. So, in the first iteration, we print the value of a[0]. In the second iteration, we print a[1]. In this way, we print all the elements of the array.

Array program examples in C

Now that we know the basics of an array, we will look at some basic programs that use arrays in C.

Reading user-entered numbers into an array

Let us begin with a simple program that reads five numbers into an array, and then prints them out. Here is the source code for the program, and you can download it here. (In this coding playground, you can change the input numbers via the “Input” tab.)

We begin by initializing an array of five elements and, a variable i which we’ll use for the for loop.

In the first loop, we read numbers from the user and set it in the a[i] element. The elements of an array simply behave like regular variables. If you had an int variable named p and wanted to set the value of p from user input, you’d use scanf("%d", &p). Similarly, here, to set the value of a[i] from user input, we used scanf("%d", &a[i]).

The second loop simply prints the variables one by one using printf().

Linear search in an array

Sometimes, you may need to search for an element in an array. For example, given an array {10, 20, 30, 40}, you may want to know if 30 is present in the array.

Linear search is a simple technique to search for an element. We iterate over every element of the array to check if it matches with the number we’re looking for.

Here is the source code for the program, and you can download it here.

In this program, we have declared an array a, the loop variable i, the element to search for search. We have also declared another variable pos, which keeps track of the array index where we found the element we were searching. We’ve initialized pos to -1; the reason for doing so will become clear later on.

Next, we read the elements of the array as well as the number to search for.

In the final loop, we’ve implemented the linear search logic. We iterate through each element of the array, and check if a[i] is equal to the value of search. If it is equal, we set the pos variable to the current index i, since we found the element at index i. Then, we break out of the loop, since we’ve found our element and we don’t need to process any further.

Then, we compare the value of pos. The minimum value of an array index can be 0. So, if we find the value of pos to be -1, this means there was no match and we’ll print a message like “30 was not found”. Otherwise, we’ll print a message such as “30 was found at position 2”.

Two-dimensional (2D) arrays in C

So far, we’ve looked at arrays where the element stores a simple data type like int. These arrays are sometimes called one-dimensional (1D) arrays.

Just as int or float are data types, an array is also a data type. Therefore, you can build an array who’s individual elements are 1D arrays. These kinds of arrays are called two-dimensional (2D) arrays.

To declare a 2D array, you need:

the basic data type
the variable name
the size of the 1D arrays
the number of 1D arrays, which combined together make up the 2D array.

Here is how you can declare a 2D array:

int a[2][4];

This statement allocates a contiguous block of memory. For our example, two 1D arrays combine together to form a 2D array, as you can see in the diagram below. Each 1D array can hold four integers. Also, just like 1D arrays, all the elements of the array are initialized to zero.

Now, let us see how we can access the elements of a 2D array. Since a 2D array consists of multiple 1D arrays, you need two indexes. This is because, you need to specify the index of the 1D array, and the specific element inside that 1D array.

So, if you want to access the first 1D array’s second element, you should use a[0][1]. Similarly, to access the second 1D array’s third element, you should use a[1][2] and so on.

A very common approach is to visualize 2D arrays like a table, where the first index is the row number and the second index is the column number. So, you can represent the above array as:

This allows us to talk about 2D arrays in a much easier way. To use the element at the first row, third column, you can use a[0][2]; to use the element at the second row, second column, use a[1][1] and so on.

Initializing, using and looping over 2D arrays

Apart from using two indexes, using 2D arrays is the same as using 1D arrays. A basic example of using the arrays is as follows:

int a[2][2];

a[0][0] = 10;
a[0][1] = a[0][0] * 10; // a[0][1] will be set to 100
a[1][0] = a[0][1] / 5; // a[1][0] will be set to 20
a[1][1] = a[0][1] + a[1][0]; // a[1][1] will be set to 120

You can find the full program below, or download it here.

If you know the elements beforehand, you can also use the initializer list syntax that we discussed previously. For example, if you want to make the following array:

You can write it as follows:

int a[2][2] = {20, 10, 40, 60};

For easier understanding, you can also group the elements of the initializer list by using braces. For example, to initialize a 2D array with two rows and three columns, you can write:

int a[2][3] = {{10, 20, 30}, {40, 50, 60}};

You can also loop over, print, or ask for input using scanf() as you would with 1D arrays. Here is an example of initializing an array, and then using for loops to print the elements. You can download the code here.

In the above program, we have used nested for loops to print the elements of the array. This is because a 2D array has two indexes, which means that the first and second index has to be changed individually. Since the printf() call uses a[i][j], so, the outer loop with variable i changes the first index, i.e. the row number. The inner loop with variable j changes the column number.

To understand this better, let us step through the loops. The outer loop starts with i = 0, and we get to the inner loop. The inner loop starts with j = 0 and ends at j = 2 (since j < 3), so we print a[0][0], a[0][1] and a[0][2]. After this, the inner loop value increases to 1, and similarly, we print a[1][0], a[1][1] and a[1][2].

In this way, we print all the elements of the 2D array.

2D array program examples in C

In this section, we’re going to look at some basic programs involving 2D arrays in C. Since 2D arrays can be visualized in the form of a table or matrix, all of our examples will revolve around the concept of using them for matrix operations.

Reading user-entered numbers into a 2D array

Before we look at more complex examples, let us first look at how to read numbers from the user into an array. Here’s the source code of the program, and you can download it here.

In the program, we begin by initializing an array of 10 rows and 10 columns, along with loop variables as well as variables that store the number of rows and columns requested by the user. Next, we ask the user to enter the number of rows and columns, and then we check if the number of rows and columns will fit into the 2D array. If it doesn’t fit, we print an error and exit the program.

Then, in the first nested for loop, we ask the user to enter the elements with scanf(). When printing the message “Enter row X, column Y”, we have used i + 1 and j + 1 ; this is because row/column numbers start from 1, but array indexes start from 0.

In the second nested for loop, we simply print the elements as we’ve seen in our previous example.

Finding the transpose of a matrix

When you “transpose” a matrix, you take every element at the ith row and jth column of the matrix, and put it in the jth row and ith column. The formula for transpose is as follows:

\begin{bmatrix} a_{00} & a_{01} & a_{02} \\ a_{10} & a_{11} & a_{12} \\ \end{bmatrix}^T = \begin{bmatrix} a_{00} & a_{10} \\ a_{01} & a_{11} \\ a_{02} & a_{12} \\ \end{bmatrix}

As you can see in this example, the size of a matrix also changes when you “transpose” it. A matrix of size $m \times n$ changes to a matrix of size $n \times m$ .

We can translate the above program into code quite easily. Suppose, the input array is a and the result array is b. Elements in a[i][j] will go into a b[j][i]. Here’s the source code for the program, and you can download it here:

Just like our previous example, we declare the arrays are a and b respectively and have 10 rows and 10 columns to hold the input and result matrix. Next, we ask the user to enter the rows and columns. If it exceeds the size of the array, we print an error and exit.

Then, we ask the user to enter the numbers one by one and print the input matrix. After that, we iterate through the matrix using a nested for loop, and implement the logic where a[i][j] goes to b[j][i].

As we discussed, the matrix size changes when you transpose it. In all the other loops, we were comparing i < rows and j < cols, but now since the size has changed, we should use i < cols and j < rows. So, in the last loop, we use this idea and print all the elements of the matrix.

Adding two matrices

In order to add two matrices, both must have the same number of rows and columns. The process of adding two matrices involves taking numbers from the same position of the matrix, adding them, and putting the result in the same position. The formula for matrix addition is as follows:

\begin{bmatrix} a_{00} & a_{01} & a_{02} \\ a_{10} & a_{11} & a_{12} \\ \end{bmatrix} + \begin{bmatrix} b_{00} & b_{01} & b_{02} \\ b_{10} & b_{11} & b_{12} \\ \end{bmatrix} = \begin{bmatrix} a_{00} + b_{00} & a_{01} + b_{01} & a_{02} + b_{02} \\ a_{10} + b_{10} & a_{11} + b_{11} & a_{12} + b_{12} \\ \end{bmatrix}

So, if we have the two input matrices a and b and the result matrix c, then you can use c[i][j] = a[i][j] + b[i][j] in a loop to add the matrices. Here’s the source code for the program, and you can download it here:

In this program, we have two input matrices a and b and one result matrix c, and each of them are of 10x10 size. First, we ask the user for the size of the matrices, and if they’re bigger than the arrays, we print an error and exit.

Otherwise, we ask the user to enter the values in the input matrix. Then, using a nested for loop, we implement the process of matrix addition, c[i][j] = a[i][j] + b[i][j]. Finally, we print all the three matrices.

Multiplying two matrices

Let us assume we have two matrices. One is of size $m \times n$ and the other one is $p \times q$ . These matrices can be only multiplied if $n = p$ . After multiplying, the result matrix will have a size of $m \times q$ .

Now, to calculate the result in the ith row and jth column of the result matrix, take all the elements of the ith row of the first matrix, and multiply it with the corresponding value in the jth column. Then, take all these products and fill the sum in the result matrix. To see what we mean, let us take an example of how you can multiply two 2x2 matrices:

\begin{bmatrix} a_{00} & a_{01} \\ a_{10} & a_{11} \\ \end{bmatrix} \times \begin{bmatrix} b_{00} & b_{01} \\ b_{10} & b_{11} \\ \end{bmatrix} = \begin{bmatrix} a_{00} b_{00} + a_{01} b_{10} & a_{00} b_{01} + a_{01} b_{11} \\ a_{10} b_{00} + a_{11} b_{10} & a_{10} b_{01} + a_{11} b_{11} \\ \end{bmatrix}

The program that implements this logic is below. You can download it here.

In the program, we first declare 10x10 input and result matrices, along with some loop variables. Just like our previous programs, we ask the user for the sizes of the two matrices, and check if they are bigger than the 10x10 size. In addition, we check if the number of columns in the first matrix equals the number of rows in the second matrix. If any of these conditions fail, we print an error and exit the program.

Then, we ask the user to enter the numbers for both matrices. Once we’ve read all the numbers, we calculate the product with a nested loop. The two outer loops with variables i and j are used to move across the different elements of the result matrix. Initially, we set c[i][j] = 0. Now, we need to pick all the elements from the ith row of the first matrix and jth column from the second matrix. So, we use another loop with the variable k that helps us with this and multiplies and adds all the elements.

Finally, we display the input and result matrices.

Multidimensional arrays in C

In the previous sections, we have seen 1D and 2D arrays. Any array with more than one dimension is a multidimensional array, and the concept can be extended to any number of dimensions, with the basics staying the same.

For example, a three dimensional (3D) array is a combination of multiple 2D arrays, and you can initialize a three dimensional array by using something like:

int a[3][2][2];

This statement creates a 3D array which is a combination of three 2x2 2D arrays. As with all arrays in C, memory allocation is contiguous and all the elements is initialized to zero. You can think of this array as follows:

To use any element of the array, you need to use three indexes. For example, writing a[0][1][1] will access the first 2D array’s second row and second column.

You can also use initializer lists with 3D arrays. Here is an example:

int a[3][2][2] = {
 {
 {10, 20},
 {30, 40},
 },
 {
 {1, 2},
 {4, 5}
 },
 {
 {3, 5},
 {7, 11}
 }
};

In the above example, the first 2D array’s first row is initialized to 10, 20, and the second row is initialized to 30, 40. Similarly, the second 2D array’s first row is initialized with 1, 2; the second row is initialized to 4, 5, and so on.

To iterate over a 3D array, you need to nest three for loops. The outer loop to change the outermost index. The loop inside it changes the middle index, and the innermost loop changes the last index. To see what we mean, take a look at the example below which initializes the array and prints all its elements. You can download it here.

Similarly, for a four dimensional array, you need to nest four for loops, and access it with four indexes, like a[0][1][4][2].

Strings in C: Arrays of Characters

In C, strings are simply an array of characters followed by the “null terminating character”. The null terminating character is a character with the integer value zero, and it is used to represent the end of a string.

Let us see how you can define a string. If you want to create a string with the content “world”, you can use the “initializer string” syntax. It looks like this:

char s[] = "world";

The array s is six characters in length, because, apart from the characters “w”, “o”, “r”, “l”, “d”, there is also a null terminating character in the array.

You can also use the “initializer list” syntax to declare strings. However, in this case, the C compiler won’t put in the null terminating character for you, so you have to manually put it in. In C, '\0' is used to represent this character, so you would have to use:

char string[] = {'w', 'o', 'r', 'l', 'd', '\0'};

With the initializer list/string syntax, you can also set the array size manually if you wish to, but this isn’t necessary:

char string[6] = {'w', 'o', 'r', 'l', 'd', '\0'};
char string[6] = "world"; // same as above

However, there is one special case. If you write something like this:

char string[5] = "world";

The C compiler will declare an array of five characters. However, since there is no space to fit the null terminating character, it will not add it to the string. However, for this example, you can’t put a size less than 5, and C compilers will throw an error.

char string[4] = "world"; // wrong!

C provides multiple string manipulation functions in the header file string.h. However, this is outside the scope of arrays, so we will not discuss it here.

How to Add, Remove and Modify Users in Linux

Mon, 04 Nov 2019 15:12:16 +0000

As a Linux administrator, it is essential to know how to add users, modify users and delete users in a Linux system. It is a good practice to have different accounts for different users and set permissions, for security purposes.

In this article, we will see how to manage users in Linux by adding new users, modifying existing users and delete user accounts which are not required.

To execute most of the commands in this article, you need root access. The distribution we’re using to show these examples supports sudo, so we’ll add sudo before the command to execute the command as root. If you don’t have sudo installed, then you may either log in as the root user into your system, or you may run su and enter the root password to gain root access.

The commands we’ve discussed here work across all Linux distributions, including Debian, Ubuntu, CentOS and RHEL.

Add a user in Linux

To add users, run the useradd command, like so:

sudo useradd -m <name of the user>

For example, if you want to add the user named john, then the command will be like:

sudo useradd -m john

By default, useradd creates a user without creating a home directory. So, to make useradd create a home folder, we’ve used the -m switch.

If the command is successful, it won’t have any output, like so:

Behind the scenes, it automatically creates the user john by assigning a unique user ID for the user, and adding the user’s details to the /etc/passwd file. It also creates a home directory for the user under /home (so the full path is /home/john).

At this point, the user has been created, but they don’t have a password and can’t log in. So, to assign a password to the newly created user, run the passwd command like so:

sudo passwd <username>

The command will ask for the new password, and ask you to confirm it:

This command adds the user’s password in /etc/shadow in an encrypted format. After running this command, the new user should be able to login as usual.

You can view the new user’s ID by using id -u <username>. In our case, john was given an ID of 1001:

User Groups in Linux

Before we move on creating and managing “groups”, let us review what a group is.

In Linux, groups are a collection of users. A group can have zero or more users assigned to it. Just like users, each group has its own “group name” and a unique “group ID”. Groups are used to assign users to a set of permissions, access or privileges.

There are two types of groups:

Primary Group: When a Linux user is created, it is automatically assigned to a single, default group, known as the “primary group”. Usually, the name of the primary group is the same as the user’s username, although you can change this if you want.
Supplementary Group: Apart from the primary group, you can add a user to other groups. These other groups to which a user belongs are called supplementary groups.

Let us understand this with an example. Previously, when we created the user john, a group named john was automatically created as well. This group is the primary group, and the user john was added to the group john.

Later, you can add john to another group, such as sales. Now, with respect to the user john, sales is a supplementary group.

Information about all groups on your system is stored in /etc/group. Groups can also have passwords, although they’re rarely used because it requires everyone in the group to know a common password. If a group has a password, it is stored in /etc/gshadow.

Create a new Group

To create a new group in Linux, run the groupadd command, like so:

sudo groupadd <name of the group>

For example, to add a group named sales to your system, use the command:

sudo groupadd sales

Just like the useradd command groupadd also doesn’t show any output if the user was successfully created:

If you want to verify that the group was indeed created, you can view the file /etc/groups.

Previously, we’ve mentioned that group passwords are rarely ever used. However, if you do want to assign a group password, you can use the gpasswd command:

sudo gpasswd sales

Enter the group password and confirm it by typing it again. The group password is set once you complete this process.

View a user’s groups and user ID

To see a user’s information, such as a user’s ID and the groups they belong to, you can use the id command. To see your own user’s information, simply type:

id

The output shows your user’s ID ( uid) and primary group’s ID ( gid), as well as a list of primary and supplementary groups you belong to. For example, in the output below, the user booleanworld belongs to the groups booleanworld and wheel.

On the other hand, if you want to view the information for a different user, use the following command:

id <username>

It lists the user ID, primary group ID, as well as the associated names of the groups and their IDs:

Otherwise, if you only want to see the groups to which a user belongs, you can use the groups command. It’s similar to id, and by default it lists your own groups by default. For example, once again we can see that the current user, booleanworld belongs to the groups booleanworld and wheel.

If you want to see the group of another user, use:

groups <username>

In the example below, we’re using it to list the groups john belongs to:

Add a User to a Group

Now that you have an idea about groups, we can now modify a user and assign them to groups. To add a user to a group, use the following command:

sudo usermod -a -G <group name> <user name>

For example, to assign the user John to the group sales, you should run:

sudo usermod -a -G sales john

Here, the -a flag “adds” the user to the group and the -G signifies that we’re adding them to a supplementary group (as opposed to changing their primary group).

If you want to change the user’s primary group instead, you can use the -g flag like so:

sudo usermod -g <primary group name> <username>

In the above command, notice that we aren’t using the -a append flag. This is because, we want to simply change the primary group of the user, and we’re not adding the user to a group. By its very definition, a primary group can only have one user.

Change Password of a User

Previously, when we created a new user, we’ve used the passwd command to assign a password to the new user. You can also use this to change passwords. If you want to change your own password, simply run:

passwd

When you change your own password, it’ll ask you for your current password. Once you enter it correctly, you will be asked to enter your new password twice.

You can also use it to change a different by providing the username, although you need to be root to change passwords for others. The syntax for changing another user’s password is:

sudo passwd <username>

When you are the root user, passwd doesn’t ask you for your current password — it’ll just ask you for the new password.

You can also use passwd to prevent a user from logging in (aka “locking out the user”), using the -l switch. For example, if you want to prevent john from logging in, you can use:

sudo passwd -l john

Grant Sudo Permissions to Users

sudo is a utility to allow users to execute commands as another user, usually the root user. In most distributions, only a certain set of users can execute sudo.

For example, in Debian and its derivatives (like Ubuntu), users in the sudo group can use the sudo command. Similarly, CentOS or RHEL has a wheel group that does the same thing.

If you want a user (say john) to be able to use sudo, you can use usermod to add them to the sudo group like so:

sudo usermod -a -G sudo john

For CentOS or RHEL use:

sudo usermod -a -G wheel john

What if you don’t use something based on Debian or CentOS? Although the default sudo configuration can vary a lot between distributions, the steps below should help you get started.

First, you should create your own group, such as sysadmins and add users to it, just like we’ve done previously. Then, you can edit the file /etc/sudoers as a root user to allow anyone who belongs to sysadmins to have sudo access. To edit the file, you can use an editor like nano or vi by running:

sudo nano /etc/sudoders # if you have 'nano' installed
sudo vi /etc/sudoers # if you have 'vi' installed

Now, go to the end of the file, and add the following text on its own line. This will allow anyone belonging to sysadmins to use sudo:

%sysadmins  ALL=(ALL)       ALL

Save the file and exit the editor. After this, any users in sysadmins would be able to use sudo to run commands.

Delete a User in Linux

To delete a user in Linux, you can use userdel command, like so:

sudo userdel <username>

By default, this command preserves the home directory and some other special files, such as the user’s list of cron jobs. If you want to delete these files as well, you should use the --remove-all-files flag.

For example, if you want to delete the user john along with the home directory of john, use the following command:

sudo userdel -r john

Building Recursive Descent Parsers: The Definitive Guide

Mon, 28 Oct 2019 11:12:08 +0000

Parsers are programs that help process text. Compilers and interpreters use parsers to analyze programs before processing them further, and parsers for formats like JSON and YAML process text into the programming language’s native data structures.

In this article, we’re going to look at how to build “recursive descent parsers”. Recursive descent parsers are a simple but powerful way of building parsers — for each “entity” in the text that you want to process, you define a function.

First, we are going to look at some of the theory underlying parsing. Then, we using Python, we’re going to build a calculator and a JSON parser that supports comments, trailing commas and unquoted strings. Finally, we’re going to discuss how you can build parsers for other kinds of languages that behave differently than the aforementioned examples.

If you’re already familiar with the theory, you can directly skip to the parts where we build the parser.

How does parsing work?

In order to process a piece of text, a program performs three tasks. In a process called “lexing” or “tokenization”, the program goes over characters in the text, and extracts logical groups called “tokens”. For example, in an expression such as “3 + 5 - 10”, a program to process arithmetic expressions could extract the following tokens:

[{ "Type": "Number" , "Text": "3" },
 { "Type": "Operator", "Text": "+" },
 { "Type": "Number" , "Text": "5" },
 { "Type": "Operator", "Text": "-" },
 { "Type": "Number" , "Text": "10" }]

The program then uses rules to identify meaningful sequences of tokens, and this is called “parsing”. The rules used to make this happen are called “grammars” — much in the same way that words of a natural language like English are strung together into meaningful sequences using English grammar. Individual rules of a grammar are called “productions”.

Imagine we’re building a program to process math expressions, and we’re interested only in addition and subtraction. An expression must have a number (such as “3”), followed by any number of operators and numbers (such as “+ 5” ). The parsing rule can be visualized like so:

Expression \rightarrow Number (Operator Number)*

The $*$ indicates that the $(Operator Number)*$ group may repeat any number of times (including zero).

Finally, the program takes a set of actions for a grammar rule. For example, after parsing, our math program would need to calculate the value of the expression.

Although we’ve described these as separate steps, a program can perform all of these at once. Doing all the steps at once carries a few advantages, and this is the approach we’re going to take in this article.

Writing production rules

Through the rest of this article, we’re going to use production rules to visualize grammars. Therefore, we’ve explained how we write the production rules throughout this article. If you’ve previously gone through a textbook on parsing, the notation we’ve used here is a bit different.

Earlier, we’ve seen an example of a simple production. When the grammar rule contains a token’s name, and the token is fairly simple, it’s common to write the token’s text in the grammar. If we consider the previous example, $Operator$ yields a + or a -, so we could rewrite the rule like so:

Expression \rightarrow Number ( ("+" | "-" ) Number )*

We’ve also seen how we can use $*$ to indicate that something repeats any number of times. Similarly, a $+$ indicates that something repeats, but it should occur at least once. A $?$ indicates something that’s optional.

As an example, suppose we’re building a parser to process a list of conditions, like “x > 10 and y > 30”. Assume that the and is optional, so we could rewrite this condition as “x > 10 y > 30”. You could design a grammar for the above like so:

Expression \rightarrow Condition ( "and" ? Condition )*

Condition \rightarrow Variable Operator Number

When there are multiple alternatives, the order of alternatives matter. For a rule like $Operator \rightarrow "+" | "-"$ , we should try matching a “+” first, and then a “-”. Although in this trivial example it might seem that order does not matter, we’re going to see examples later on where order becomes important.

Finally, a production rule only focuses on tokens and other grammar symbols — it ignores whitespaces and comments.

Considerations when building parsers

Previously, we’ve seen a few simple grammars. However, when trying to parse something more complex, a number of issues can come up. We’re going to discuss them here.

Handling precedence in grammars

Previously, we’ve seen a grammar that can handle a math expression consisting of additions and subtractions. Unfortunately, you can’t extend the same grammar for multiplications (and divisions), since those operations have higher precedence than additions (and subtractions).

In order to handle this scenario, we must design our grammar so that the parser defers any additions, but performs multiplications as soon as it is encountered. To achieve this, we split the grammar into two separate rules, as shown below:

Expression \rightarrow Term ( ( "+" | "-" ) Term )*

Term \rightarrow Number ( ( "*" | "/" ) Number )*

Let us take an expression (such as “3 + 2 * 5”) to ensure that this really works. We’ll assume that once our parser has finished matching a grammar rule, it automatically calculates the expression. We’ll also use underlined text to indicate what the parser is looking for. In addition, we’ve also omitted the quotes for clarity.

Expression \Rightarrow \underline{Term} + Term \Rightarrow 3 + \underline{Term} \Rightarrow 3 + \underline{Number} * Number \Rightarrow 3 + 2 * \underline{Number} \Rightarrow 3 + 2 * 5

When 2 * 5 is parsed, the parser immediately returns the result of 10. This result is received at the $Term "+" Term$ step, and the addition is performed at the end, yielding 13 as the result.

In summary, you should arrange your rules so that operators with the lowest precedence are at the top, while the ones with highest precedence are at the bottom.

Avoiding left recursion

Earlier, we have mentioned that a recursive descent parser consists of functions that process “entities” in the text. Now that we know about tokens and production rules, let us redefine this a bit more formally: each function in the parser extracts a token, or implements the logic of a production rule.

Let us take a simple grammar, such as $Expression \rightarrow Number "+" Number$ to see what this really means. If you wrote the pseudocode for this grammar, it would look something like this:

def expression():
 first_number = number()
 read('+')
 second_number = number()
 # process these two numbers, e.g. adding them

Next, consider a grammar that parses a list of comma separated numbers. You’d generally write it as:

List \rightarrow Number ("," Number)*

Now, imagine that for some reason, you wanted to avoid the $*$ wildcard and rewrite it like so:

List \rightarrow List "," Number | Number

This grammar is theoretically correct — it either picks a single number from the text, or it expands to a list with a number at the end. The second list, in turn, can expand to another number or a list, until parser finishes through the text.

However, there’s a problem with this approach — you’d enter an infinite recursion! If you try calling the list() function once, you end up calling it again, and so on. You might think that rewriting the grammar as $Number | List "," Number$ might help. However, if you tried writing a function, you’d read a single number and quit parsing. The number you read might be part of a list like “2, 4, 6” and you won’t be able to read the other numbers.

When dealing with parsing, this is called left recursion. The case that we saw above involves “direct left recursion”, but there’s also “indirect left recursion”, where A calls B, B calls A and so on. In either case, you should rewrite the grammar in another way to avoid this issue.

Avoiding backtracking

Suppose, you’re trying to build a parser that parses an operation of two numbers, such as “2 + 4”, and wrote a grammar in this way:

Expression \rightarrow Addition | Subtraction \\ Addition \rightarrow Number "+" Number \\ Subtraction \rightarrow Number "-" Number

This grammar is correct, and it will also behave in the way you expect and produce correct results. However, this grammar is not what you’d want to use.

To see why this is the case, consider the input string “5 - 2”. We’ll first go with the addition rule, and parse a number. Then, we’d expect a “+” but when reading the string, we’ll get a “-” which means we have to backtrack and apply the subtraction rule. In doing so, we’ve wasted time and CPU cycles on parsing a number, only to discard it and parse it again as part of the subtraction rule.

Additionally, when building parsers that directly work off of data streams, like a network socket, rewinding the stream is often not an option. In this article, we’ll only deal with parsers that have all of the text in memory. Nevertheless, it’s a good practice to avoid backtracking and to do so, you should factor out the common parts from the left. This is how our rule would look after factoring:

Expression \rightarrow Number ( "+" Number | "-" Number )

The factoring step is complete and will avoid backtracking. However, for aesthetic reasons, we’ll just write it as:

Expression \rightarrow Number ( "+" | "-" ) Number

Building a base for the Recursive Descent Parser in Python

Now that we’ve discussed the various considerations for parsing, we’ll build out the calculator and JSON parser. However, before we do that, we’ll write a base “Parser” class that has some methods for common functions, such as recognizing characters and handling errors.

This section might seem a bit too long, but it’s well worth the patience. Once you complete this section, you’ll have all the tooling to build complex parsers with very little code required to recognize tokens and implement production rules.

The exception class

Since this is the easiest task by far, so we’ll tackle this first. We’ll define our own exception class named ParseError like so:

class ParseError(Exception):
 def __init__(self, pos, msg, *args):
 self.pos = pos
 self.msg = msg
 self.args = args

 def __str__(self):
 return '%s at position %s' % (self.msg % self.args, self.pos)

The class accepts the text position where the error occurred, an error message (as a format string), and the arguments to the format string. Let’s see of how you might use this exception type:

e = ParseError(13, 'Expected "{" but found "%s"', "[")

When you try printing the exception, you’ll get a formatted message like this:

Expected "{" but found "[" at position 13

Notice that we haven’t used a % symbol within the format string and its arguments (such as '"Expected "{" but found "%s"' % "["). This is handled in the __str__ method of the class, where we’ve formatted self.msg with elements in self.pos.

Now, you might ask, why do we want to format it in the __str__ method, instead of writing it with a %? It turns out that using the % operator to format a string takes up quite a bit of time. When parsing a string, there’ll be many exceptions raised as the parser tries to apply various rules. Hence, we’ve deferred the string formatting to when it’s really needed.

The base Parser class

Now that we have an error class in place, the next step is to write the parser class. We’ll define it as follows:

class Parser:
 def __init__(self):
 self.cache = {}

 def parse(self, text):
 self.text = text
 self.pos = -1
 self.len = len(text) - 1
 rv = self.start()
 self.assert_end()
 return rv

Here, you’ll notice a cache member in the __init__ method — we’ll get back to why this is needed when we discuss recognizing characters.

The parse method is fairly simple — first, it stores the string. Since we haven’t scanned any part of the string yet, we’ll set the position to -1. Also, we’ll store the maximum index of the string in self.len. (The maximum index is always one less than the length of the string.)

Next, we’ll start parsing the string and check if we’ve reached the end of the string. This is to ensure that our parser was able to process the entire text without anything left at the end. After this, we return the result.

The assert_end method is simple: we’ll check if the current position is less than the maximum index. Bear in mind that these methods are inside the class, although we’ve omitted the indentation for simplicity.

def assert_end(self):
 if self.pos < self.len:
 raise ParseError(
 self.pos + 1,
 'Expected end of string but got %s',
 self.text[self.pos + 1]
 )

Throughout our parser, we’ll be looking at the character that is one more than the current index ( self.pos). To see why this is the case, consider that we start out with self.pos = -1, so we’d be looking at the index 0. Once we’ve recognized a character successfully, self.pos goes to 0 and we’d look at the character at index 1, and so on. This is why the error uses self.pos + 1.

With most languages, whitespace between tokens can be safely discarded. Therefore, it seems logical to include a method that checks if the next character is a whitespace character, and if so, “discards” it by advancing to the next position.

def eat_whitespace(self):
 while self.pos < self.len and self.text[self.pos + 1] in " \f\v\r\t\n":
 self.pos += 1

Processing character ranges

Now, we’ll write a function to help us recognize characters in the text. Before we do that though, we’ll consider our function from a user’s perspective — we’ll give it a set of characters to match from the text. For example, if you wanted to recognize an alphabet or underscore, you might write something like:

self.char('A-Za-z_')

We’ll give the char method containing a list of characters or ranges (such as A-Z in the above example). Ranges are denoted through two characters separated by a -. The first character has a numerically smaller value than the one on the right.

Now, if the character at self.text[self.pos + 1] matches something in the argument of self.char, we’ll return it. Otherwise, we’ll raise an exception.

Internally, we’ll process the string into something that’s easier to handle — a list with separate characters or ranges such as ['A-Z', 'a-z', '_']. So, we’ll write a function to split the ranges:

def split_char_ranges(self, chars):
 try:
 return self.cache[chars]
 except KeyError:
 pass

 rv = []
 index = 0
 length = len(chars)

 while index < length:
 if index + 2 < length and chars[index + 1] == '-':
 if chars[index] >= chars[index + 2]:
 raise ValueError('Bad character range')

 rv.append(chars[index:index + 3])
 index += 3
 else:
 rv.append(chars[index])
 index += 1

 self.cache[chars] = rv
 return rv

Here, we loop through the chars string, looking for a - followed by another character. If this condition matches and the first character is smaller than the last, we add the entire range (such as A-Z) to the list rv. Otherwise, we add the character at chars[index] to rv.

Then, we add the list in rv it to the cache. If we see this string a second time, we’ll return it from the cache using the try ... except KeyError: ... block at the top.

Admittedly, we could have just provided a list like ['A-Z', 'a-z', '_'] to the char method. However, in our opinion, doing it in this way makes calls to char() look a bit cleaner.

Extracting characters from the text

Now that we have a method that gives a list containing ranges and characters, we can write our function to extract a character from the text. Here’s the code for the char method:

def char(self, chars=None):
 if self.pos >= self.len:
 raise ParseError(
 self.pos + 1,
 'Expected %s but got end of string',
 'character' if chars is None else '[%s]' % chars
 )

 next_char = self.text[self.pos + 1]
 if chars == None:
 self.pos += 1
 return next_char

 for char_range in self.split_char_ranges(chars):
 if len(char_range) == 1:
 if next_char == char_range:
 self.pos += 1
 return next_char
 elif char_range[0] <= next_char <= char_range[2]:
 self.pos += 1
 return next_char

 raise ParseError(
 self.pos + 1,
 'Expected %s but got %s',
 'character' if chars is None else '[%s]' % chars,
 next_char
 )

Let us first focus on the argument chars=None. This allows you to call self.char() without giving it a set of characters. This is useful when you want to simply extract a character without restricting it to a specific set. Otherwise, you can call it with a range of characters, as we’ve seen in the previous section.

This function is fairly straightforward — first, we check if we’ve run out of text. Then, we pick the next character and check if chars is None, in which case we can increment the position and return the character immediately.

However, if there are a set of characters in chars, we split it into a list of individual characters and ranges, such as ['A-Z', 'a-z', '_']. In this list, a string of length 1 is a character, and anything else is a range. It checks if the next character matches the character or the range, and if it does, we return it. If we failed to match it against anything, it exits the loop which raises ParseError, stating that we couldn’t get the expected character.

The 'character' if chars is None else '[%s]' % chars is just a way of providing a more readable exception. If chars is None, the exception’s message would read Expected character but got ..., but if char was set to something like A-Za-z_, we’d get Expected [A-Za-z_] but got ....

Later on, we’ll see how to use this function to recognize tokens.

Extracting keywords and symbols

Apart from extracting individual characters, recognizing keywords is a common task when building a parser. We’re using “keywords” to loosely refer to any contiguous string that is it’s “own entity”, and may have whitespace before and after it. For example, in JSON { could be a keyword, and in a programming language if, else could be a keyword, and so on.

This is the code for recognizing keywords:

def keyword(self, *keywords):
 self.eat_whitespace()
 if self.pos >= self.len:
 raise ParseError(
 self.pos + 1,
 'Expected %s but got end of string',
 ','.join(keywords)
 )

 for keyword in keywords:
 low = self.pos + 1
 high = low + len(keyword)

 if self.text[low:high] == keyword:
 self.pos += len(keyword)
 self.eat_whitespace()
 return keyword

 raise ParseError(
 self.pos + 1,
 'Expected %s but got %s',
 ','.join(keywords),
 self.text[self.pos + 1],
 )

This method takes keywords, such as self.keyword('if', 'and', 'or'). The method strips whitespace and then checks if we’ve run out of text to parse. It then iterates through each keyword, checking if the keyword exists in the text. If it does find something, we’ll strip the whitespace following the keyword, and then return the keyword.

However, if it doesn’t find something, it exits the loop which raises ParseError, stating that the keywords couldn’t be found.

A helper for matching multiple productions

In the previous sections, you’ve noticed that we use exceptions to report errors. We also know that to write a recursive descent parser, you write functions that recognize a token or a production rule. Now, imagine that you wanted to parse a simple text, where you expect a number or a word. The production rule might look like:

Item \rightarrow Number | Word

We’ll also assume that the text contains a word. When the parser tries to look for a digit (perhaps by using char() ), it won’t find it, and this causes it to raise an exception. In addition, you have to strip the whitespace before and after matching a production rule. So, the code for item() might look like this:

def item(self):
 self.eat_whitespace()
 try:
 rv = self.number()
 except ParseError:
 rv = self.word()
 self.eat_whitespace()
 return rv

This already looks like a lot to implement a simple rule! Imagine what it would be like to implement a complex rule — you’d have to use try...except blocks all over the place.

So, we’ll write a match() function that’ll strip the whitespace and try to match multiple rules. The function is as follows:

def match(self, *rules):
 self.eat_whitespace()
 last_error_pos = -1
 last_exception = None
 last_error_rules = []

 for rule in rules:
 initial_pos = self.pos
 try:
 rv = getattr(self, rule)()
 self.eat_whitespace()
 return rv
 except ParseError as e:
 self.pos = initial_pos

 if e.pos > last_error_pos:
 last_exception = e
 last_error_pos = e.pos
 last_error_rules.clear()
 last_error_rules.append(rule)
 elif e.pos == last_error_pos:
 last_error_rules.append(rule)

 if len(last_error_rules) == 1:
 raise last_exception
 else:
 raise ParseError(
 last_error_pos,
 'Expected %s but got %s',
 ','.join(last_error_rules),
 self.text[last_error_pos]
 )

Before we discuss how it works, let’s see how simple it is to rewrite our previous example using match():

def item(self):
 return self.match('number', 'word')

So, how does that work? match() takes a method name to run (such as number or word in the above example). First, it gets rid of the whitespace at the beginning. Then, it iterates over all the method names and fetches each method using its name via getattr(). It then tries to invoke the method, and if all goes well, it’ll strip whitespace after the text as well. Finally, it returns the value that it received by calling the method

However, if there’s an error, it resets self.pos to the value that was there before trying to match a rule. Then, it tries to select a good error message by using the following rules:

When matching multiple rules, many of them might generate an error. The error that was generated after parsing most of the text is probably the error message we want.

To see why this is the case, consider the string “abc1”. Trying to call number() would fail at position 0, whereas word() would fail at position 2. Looking at the string, it’s quite likely that the user wanted to enter a word, but made a typo.

If two or more erroring rules end up in a “tie”, we prefer to tell the user about all the rules which failed.

The last_error_pos and last_exception keeps track of the last error’s position and exception respectively. In addition, we maintain a list last_error_rules so that in case of a tie, we can tell the user about all the rules we tried to match.

In the except ParseError block, we compare the last error’s position and the current error. If the current error is equal, we consider it a tie and append to the list. Otherwise, we update the error position, exception and list of errored rules.

In the end, we check how many rules failed. If there’s only one, last_exception would have the best error message. Otherwise, it’s a tie, and we’ll let the user know with a message like Expected number,word but found ....

Looking ahead at what’s next - helper functions

At this point, we have all the necessary things in our parser, and all failures throw exceptions. However, sometimes we want to only match a character, keyword or rule if it’s there, without the inconvenience of handling exceptions.

We’ll introduce three small helpers to do just that. In the case of an exception when looking for stuff, these will return None:

def maybe_char(self, chars=None):
 try:
 return self.char(chars)
 except ParseError:
 return None

def maybe_match(self, *rules):
 try:
 return self.match(*rules)
 except ParseError:
 return None

def maybe_keyword(self, *keywords):
 try:
 return self.keyword(*keywords)
 except ParseError:
 return None

Using these functions are easy. This is how you might use them:

operator = self.maybe_keyword('+', '-'):
if operator == '+':
 # add two numbers
elif operator == '-':
 # subtract two numbers
else: # operator is None
 # do something else

First parser example: a calculator

Now that we’ve built the foundation for writing a parser, we’ll build our first parser. It’ll be able to parse mathematical expressions with additions, subtractions, multiplications, divisions, and also handle parenthesized expressions like “(2 + 3) * 5”.

We will begin by visualizing the grammar in the form of production rules.

Production rules for the calculator grammar

We’ve previously seen a grammar to handle everything except for parenthesized expressions:

Expression \rightarrow Term ( ( "+" | "-" ) Term )*

Term \rightarrow Number ( ( "*" | "/" ) Number )*

Now, let’s think about how parenthesized expressions fit in to this grammar. When evaluating “(2 + 3) * 5”, we’d have to calculate “(2 + 3)” and reduce it to a number. Which means that in the above grammar, the term “Number” can either refer to a parenthesized expression, or something that’s really a number, like “5”.

So, we’ll tweak our rules to accommodate both. In the rule for “Term”, we’ll replace “Number” for a more appropriate term like “Factor”. “Factor” can then either refer to a parenthesized expression, or a “Number”. The modified grammar looks like this:

Expression \rightarrow Term ( ( "+" | "-" ) Term )*

Term \rightarrow Factor ( ( "*" | "/" ) Factor )*

Factor \rightarrow "(" Expression ")" | Number

With that out of the way, let’s implement the production rules!

Implementing the parser

We’ll implement our Calculator parser by extending the base Parser class. We begin defining the class like so:

class CalcParser(Parser):
 def start(self):
 return self.expression()

Previously, when implementing the base parser class, we had a start() method to start parsing. We’ll simply call the expression() method here, which we’ll define as below:

def expression(self):
 rv = self.match('term')
 while True:
 op = self.maybe_keyword('+', '-')
 if op is None:
 break

 term = self.match('term')
 if op == '+':
 rv += term
 else:
 rv -= term

 return rv

The grammar rule for $Expression \rightarrow Term ( ( "+" | "-" ) Term )*$ . So, we start by reading a term from the text. We then set up an infinite loop, and look for a “+” or “-”. If we don’t find either, this means that we’ve finished reading through the list of additional terms as given by $( "+" | "-" ) Term )*$ , so we can break out of the loop.

Otherwise, we read another term from the text. Then, depending upon the operator, we either add or subtract it from the initial value. We continue with this process until we fail to get a “+” or “-”, and return the value at the end.

We’ll implement term() in the same way:

def term(self):
 rv = self.match('factor')
 while True:
 op = self.maybe_keyword('*', '/')
 if op is None:
 break

 term = self.match('factor')
 if op == '*':
 rv *= term
 else:
 rv /= term

 return rv

Next, we’ll implement “Factor”. We’ll try to read a left parenthesis first, which means that there’s a parenthesized expression. If we do find a parenthesis, we read an expression and the closing parenthesis, and return the value of the expression. Otherwise, we simply read and return a number.

def factor(self):
 if self.maybe_keyword('('):
 rv = self.match('expression')
 self.keyword(')')

 return rv

 return self.match('number')

In the next section, we’re going to make our parser recognize a number.

Recognizing numbers

In order to recognize a number, we need to look at the individual characters in our text. Numbers consist of one or more digits, optionally followed by a decimal part. The decimal part has a period (.), and followed by at least one digit. In addition, numbers may have a sign such as “+” or “-” before them, such as “-124.33”.

We’ll implement the number() method like so:

def number(self):
 chars = []

 sign = self.maybe_keyword('+', '-')
 if sign is not None:
 chars.append(sign)

 chars.append(self.char('0-9'))

 while True:
 char = self.maybe_char('0-9')
 if char is None:
 break

 chars.append(char)

 if self.maybe_char('.'):
 chars.append('.')
 chars.append(self.char('0-9'))

 while True:
 char = self.maybe_char('0-9')
 if char is None:
 break

 chars.append(char)

 rv = float(''.join(chars))
 return rv

Although the function is long, it’s fairly simple. We have a chars list in which we put the characters of the number. First, we look at any “+” or “-” symbols that may be present before the number. If it is present, we add it to the list. Then, we look for the first mandatory digit of the number and continue looking for more digits using the maybe_char() method. Once we get None through maybe_char, we know that we’re past the set of digits, and terminate the loop.

Similarly, we look for the decimal part and its digits. Finally, we convert all the characters into a string, which in turn, we convert to a floating point number.

An interface for our parser

We’re done building our parser. As a final step, we’ll add a little bit of code in the global scope that’ll allow us to enter expressions and see results:

if __name__ == '__main__':
 parser = CalcParser()

 while True:
 try:
 print(parser.parse(input('> ')))
 except KeyboardInterrupt:
 print()
 except (EOFError, SystemExit):
 print()
 break
 except (ParseError, ZeroDivisionError) as e:
 print('Error: %s' % e)

If you’ve followed along so far, congratulations! You have built your first recursive descent parser!

If you’re following along in a local IDE, you can run your code at this point. Alternatively, you can use the playground below to run the parser. You can enter expressions through the “Input” tab below:

You can also find the full code here.

Second parser example: an “extended” JSON parser

JSON is a data interchange format that supports basic data structures like numbers, strings and lists. It’s a very widely used format, although it’s simplicity means that it’s lacking things like comments and strict about the things it accepts. For example, you can’t have a trailing comma in a list or have an explicit “+” in front of a number to indicate its sign.

Since Python already has a JSON module, we’re going to aim for a little higher and build a JSON parser that supports comments, trailing commas and unquoted strings.

Implementing this parser will teach you how to handle comments. It’ll also illustrate how making an easy to use language can make parsing complicated, and how you can deal with such situations.

Before we implement our grammar, let us get a feel for our extended JSON format:

{
 # Comments begin with a '#' and continue till the end of line.

 # You can skip quotes on strings if they don't have hashes,
 # brackets or commas.
 Size: 1.5x,

 # Trailing commas are allowed on lists and maps.
 Things to buy: {
 Eggs : 6,
 Bread: 4,
 Meat : 2,
 },

 # And of course, plain JSON stuff is supported too!
 "Names": ["John", "Mary"],
 "Is the sky blue?": true
}

Production rules for the JSON grammar

Our extended JSON sticks to the same data types that JSON supports. JSON has four primitive data types — null, booleans, numbers and strings, and two complex data types — lists (or arrays), and maps (also called hashes or dictionaries). Lists and hashes look similar to the way they do in Python.

Since JSON data would consist of one of these types, you might write the production rules like so:

Start \rightarrow AnyType

AnyType \rightarrow ComplexType | PrimitiveType

Then, you can break up these two types into JSON’s types:

ComplexType \rightarrow List | Map

PrimitiveType \rightarrow Null | Boolean | Number | String

This grammar is fine for parsing regular JSON but needs a bit of tweaking for our case. Since we’re going to support unquoted strings, which means “1.5” (without the quotes) is a number, but “1.5x” (again, without the quotes), is a string. Our current grammar would read the “1.5” from “1.5x”, and then cause an error because “x” wasn’t expected after a number.

This means that first, we’d have to get the full set of unquoted characters. Next, we’ll analyze the characters and determine if it’s a number or a string. So, we’ll combine numbers and unquoted strings into a single category, “Unquoted”. The quoted strings are fine, so we’ll separate them into another category, “QuotedString”. Our modified production rule for “PrimitiveType” now looks like this:

PrimitiveType \rightarrow Null | Boolean | QuotedString | Unquoted

In addition, the order of the rules is important. Since we have unquoted keys, we must first try parsing the text as a null or a boolean. Otherwise, we might end up recognizing “null” or “true” as an unquoted string.

Implementing the parser and dealing comments

To implement the JSON parser, we’ll begin by extending the base parser class like so:

class JSONParser(Parser):
 # ...

We will first tackle the problem of dealing with comments. If you think about it, comments are really similar to whitespace — they occur in the same places where whitespaces can occur, and they can be discarded just like whitespace. So, we’ll reimplement eat_whitespace() to deal with the comments, like so:

def eat_whitespace(self):
 is_processing_comment = False

 while self.pos < self.len:
 char = self.text[self.pos + 1]
 if is_processing_comment:
 if char == '\n':
 is_processing_comment = False
 else:
 if char == '#':
 is_processing_comment = True
 elif char not in ' \f\v\r\t\n':
 break

 self.pos += 1

Here, we need to keep track of whether we’re processing whitespaces or a comment. We loop over the text character-by-character, checking if we have whitespaces or a #. When there’s a #, we update is_processing_comment to True and in the next iterations of the while loop, we can safely discard all characters until we reach the end of the line. However, when processing whitespace characters, we must stop once a non-whitespace character appears.

Next, we’ll implement the production rules and the start() method. The input text will have a JSON type contained in it, so we’ll simply call any_type() in the start() method:

def start(self):
 return self.match('any_type')

def any_type(self):
 return self.match('complex_type', 'primitive_type')

def primitive_type(self):
 return self.match('null', 'boolean', 'quoted_string', 'unquoted')

def complex_type(self):
 return self.match('list', 'map')

Parsing lists and maps

Previously, we’ve seen how to parse comma separated lists. JSON’s lists are just a comma separated list with square brackets tacked onto them, and the list may have zero or more elements. Let’s see how you would implement parsing a list:

def list(self):
 rv = []

 self.keyword('[')
 while True:
 item = self.maybe_match('any_type')
 if item is None:
 break

 rv.append(item)

 if not self.maybe_keyword(','):
 break

 self.keyword(']')
 return rv

We begin by reading off the initial square bracket followed by an item from the list using self.maybe_match('any_type'). If we failed to get an item, this indicates that we’re probably done going through all the items, so we break out of the loop. Otherwise, we add the item to the list. We then try to read a comma from the list, and the absence of a comma also indicates that we’re done with the list.

Similarly, maps are just comma separated lists of “pairs” with braces, where a pair is a string key followed by a colon(:) and a value. Unlike Python’s dict s which can have any “hashable” type as a key (which includes int s and tuple s), JSON only supports string keys.

This is how you’d implement the rules for maps and pairs:

def map(self):
 rv = {}

 self.keyword('{')
 while True:
 item = self.maybe_match('pair')
 if item is None:
 break

 rv[item[0]] = item[1]

 if not self.maybe_keyword(','):
 break

 self.keyword('}')
 return rv

def pair(self):
 key = self.match('quoted_string', 'unquoted')

 if type(key) is not str:
 raise ParseError(
 self.pos + 1,
 'Expected string but got number',
 self.text[self.pos + 1]
 )

 self.keyword(':')
 value = self.match('any_type')

 return key, value

In pair(), we try to read a “QuotedString” or “Unquoted” for the key. As we’ve mentioned previously, “Unquoted” can either return a number or a string, so we explicitly check if the key that we’ve read is a string. pair() then returns a tuple with a key and value, and map() calls pair() and adds these to a Python dict.

Recognizing null and boolean

Now that the main parts of our parser are implemented, let us begin by recognizing simple tokens such as null and booleans:

def null(self):
 self.keyword('null')
 return None

def boolean(self):
 boolean = self.keyword('true', 'false')
 return boolean[0] == 't'

Python’s None is the closest analogue to JSON’s null. In the case of booleans, the first characters is sufficient to tell whether it’s true or false, and we return the result of the comparison.

Recognizing unquoted strings and numbers

Before moving on to recognize unquoted strings, let us first define the set of acceptable characters. We’ll leave out anything that’s considered special in such as braces, quotes, colons, hashes (since they are used in comments) and backslashes (because they’re used for escape sequences). We’ll also include spaces and tabs in the acceptable set of characters so that you can write strings like “Things to buy” without using quotes.

So, what are the characters that we want to accept? We can use Python to figure this out:

>>> import string
>>> ''.join(sorted(set(string.printable) - set('{}[]:#"\'\f\v\r\n')))
'\t !$%&()*+,-./0123456789;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ\\^_`abcdefghijklmnopqrstuvwxyz|~'

The next question is, how do you figure out if the text you read is a number or a string? The answer is — we cheat! Since Python’s int() and float() can take a string and return a number, we’ll use them and if those result in a ValueError, we’ll return a string. As for when to use int() or float(), we’ll use a simple rule. If the text contains “E”, “e” or “.”, (for example, “12.3” or “2e10”), we’ll call float(); otherwise, we’ll use int().

Here’s the code:

def unquoted(self):
 acceptable_chars = '0-9A-Za-z \t!$%&()*+./;<=>?^_`|~-'
 number_type = int

 chars = [self.char(acceptable_chars)]

 while True:
 char = self.maybe_char(acceptable_chars)
 if char is None:
 break

 if char in 'Ee.':
 number_type = float

 chars.append(char)

 rv = ''.join(chars).rstrip(' \t')
 try:
 return number_type(rv)
 except ValueError:
 return rv

Since matching rules are handled by match(), this takes care of stripping any initial whitespace before unquoted() can run. In this way, we can be sure that unquoted() won’t return a string consisting of only whitespaces. Any whitespace at the end is stripped off before we parse it as a number or return the string itself.

Recognizing quoted strings

Quoted strings are fairly simple to implement. Most programming languages (including Python) have single and double quotes that behave in the same way, and we’ll implement both of them.

We’ll support the following escape sequences — \b (backspace), \f (line feed), \n (newline), \r (carriage return), \t (tab) and \u(four hexadecimal digits) where those digits are used to represent an Unicode “code point”. For anything else that has a backslash followed by a character, we’ll ignore the backslash. This handles cases like using the backslash to escape itself ( \\) or escaping quotes ( \").

Here’s the code for it:

def quoted_string(self):
 quote = self.char('"\'')
 chars = []

 escape_sequences = {
 'b': '\b',
 'f': '\f',
 'n': '\n',
 'r': '\r',
 't': '\t'
 }

 while True:
 char = self.char()
 if char == quote:
 break
 elif char == '\\':
 escape = self.char()
 if escape == 'u':
 code_point = []
 for i in range(4):
 code_point.append(self.char('0-9a-fA-F'))

 chars.append(chr(int(''.join(code_point), 16)))
 else:
 chars.append(escape_sequences.get(char, char))
 else:
 chars.append(char)

 return ''.join(chars)

We first read a quote and then read additional characters in a loop. If this character is the same type of quote as the first character we read, we can stop reading further and return a string by joining the elements of chars.

Otherwise, we check if it’s an escape sequence and handle it in the aforementioned way, and add it to chars. Finally, if it matches neither of them, we treat it as a regular character and add it to our list of characters.

An interface for our JSON parser

To make sure we can interact with our parser, we’ll add a few lines of code that accepts input and parses it as JSON. Again, this code will be in global scope:

if __name__ == '__main__':
 import sys
 from pprint import pprint

 parser = JSONParser()

 try:
 pprint(parser.parse(sys.stdin.read()))
 except ParseError as e:
 print('Error: '+ str(e))

To ensure we can read multiple lines, we have used sys.stdin.read(). If you’re going to run this locally, you can enter text and use Ctrl+D to stop the program from asking for more input. Otherwise, you can use this runnable snippet:

You can find the complete code here.

Building other kinds of parsers

Now that we’ve gone through building two parsers, you might have a fairly good idea of how to roll your own for the task at hand. While every language is unique (and therefore, their parsers), there are some common situations that we haven’t covered. For the sake of brevity, we won’t cover any code in these sections.

Whitespace sensitive languages

Before we begin, let us clarify that we’re not referring to languages that use whitespace-based indentation — we’ll cover them in the next section. Here, we’ll discuss languages where the meaning of things change based on the whitespace you put between elements.

An example of such a language would be where “a=10” is different than “a = 10”. With bash and some other shells, “a=10” sets an environment variable, whereas “a = 10” runs the program “a” with “=” and “10” as command-line arguments. You can even combine the two! Consider this:

a=10 b=20 c = 30

This sets the environment variables “a” and “b” only for the program “c”. The only way to parse such a language is to handle the whitespaces manually, and you’d have to remove all the eat_whitespace() calls in keyword() and match(). This is how you might write the production rules:

Command \rightarrow Variables ProgramArguments? \\ Variables \rightarrow VariableDeclaration ( Whitespace VariableDeclaration )* \\ VariableDeclaration \rightarrow Name "=" VariableValue

Whitespace based indentation

Languages like C and Javascript use curly braces to indicate the body of loops and if-statements. However, Python uses indentation for the same purpose, which makes parsing tricky.

One of the methods to handle this is to introduce a term such as “INDENT” to keep track of indented statements. To see how this would work, consider the following production rule:

WhileLoop \rightarrow "while" Condition ( INDENT Statement )*

After matching a condition, our parser would look for INDENT. Since this is the first time it’s trying to match INDENT, it takes whatever whitespace (except for newlines) that appears along with a non-whitespace character (since that would be part of a statement).

In subsequent iterations, it looks for the same whitespace in the text. If it encounters a different whitespace, it can raise an error. However, if there’s no whitespace at all, it indicates that the “while” loop is finished.

For nested indentations, you’d need to maintain a list of all indentations that you’re currently handling. When the parser handles a new indented block, it should add the new indentation string on the list. You can tell if you’re done parsing the block, by checking that you were able to retrieve less whitespace than you did previously. At this point, you should pop off the last indentation off the list.

How to Set up SSH Keys on a Linux/Unix Server

Thu, 24 Jan 2019 15:45:03 +0000

SSH (Secure SHell) is a network protocol that allows you to securely execute commands on and transfer files to a remote server. Typically, most servers running SSH are configured to use password-based authentication; however, this is vulnerable to brute-force attacks. SSH key-based authentication is a more secure and convenient way of logging onto your server. It uses a pair of keys to authenticate users and does not require a password to log in.

In this article, we will see how to configure a server to use SSH keys for authentication. To follow these steps, you will need a Linux/Unix server and a computer that has OpenSSH installed. MacOS and most Linux distributions have it installed by default. Windows users can use the “Bash on Windows” feature or use msys2.

How do SSH keys work?

When you connect to a server using SSH, the client and server establish an encrypted connection to exchange data. Once this connection is established, there is an authentication phase to check the user’s credentials. With password-based authentication, the client sends the password to the server over the encrypted channel. However, this is vulnerable to brute-force attacks — an automated system can try common passwords, or various combinations of letters, words and names against your server.

With key-based authentication, you use “asymmetric encryption” and “challenge-response authentication” to authenticate users. Simply put, you generate two mathematically related keys, called “public” and “private” keys. Then, you copy the public key to the server, but you keep the private key on your local machine, safely guarded from others. Due to how these keys work, you can encrypt data only with the public key, and decrypt data with the private key.

When you authenticate, the server uses the public key to encrypt a random number and sends it to the client. The client then “proves” that it has the private key by decrypting the number and sending it back to the server. The server then matches the numbers, and the authentication is completed. Because the client needs to prove itself in this way, this method is secure against any brute-force attacks.

SSH also uses asymmetric encryption to establish the encrypted connection, but this process is separate from key-based authentication. The server uses a separate public and private key to establish this connection. Also, as SSH key pairs are used to authenticate users on a server, each user will have their own public and private keys for this purpose.

Generating a SSH key pair

OpenSSH comes with a tool called ssh-keygen to generate key pairs. By default, it uses 2048-bit RSA keys, although this can be changed (more on that later). We will generate our first key pair with the command:

ssh-keygen

When you run this command, it will ask you where you want to save the key. By default, the private key is saved to ~/.ssh/id_rsa and the public key to ~/.ssh/id_rsa.pub. If you want to change the location, you can enter a custom path.

In addition, to better protect your private key, it will also ask for a passphrase. If you use a passphrase, it will be used to encrypt the generated private key. In this way, even if someone managed to steal your private key, you would be safe as they would need to decrypt the private key with the passphrase, to use it. However, for the purposes of this article, we won’t use a passphrase by leaving the field blank.

Now, you can verify that these files have been created, by running:

ls ~/.ssh

This lists out the two files:

You can also save the key pair to a custom location with the -f switch. If you run:

ssh-keygen -f ~/my-key

The private key would be in ~/my-key and the public key would be ~/my-key.pub.

(Optional) Choosing the SSH key type and size

In the previous section, we have seen that ssh-keygen generates 2048-bit RSA keys. This key format strikes a balance — it is compatible with most systems, and it is also secure enough for most purposes. However, you may want to choose a different key type or key length, depending on your use case.

OpenSSH supports several types of keys — DSA-, RSA-, ECDSA- and Ed25519-based keys. The main advantage of using ECDSA/Ed25519 keys over DSA/RSA keys is that they can provide the same level of security with much smaller keys. However, older versions of OpenSSH do not support ECDSA/Ed25519 keys. In addition, DSA has been deprecated in modern OpenSSH versions due to vulnerabilities, and various concerns exist about the security of ECDSA. This only leaves RSA and Ed25519 as practical choices.

You can change the key type using the -t option. For example, if you want to create a Ed25519 key, you would simply use:

ssh-keygen -t ed25519

By default, these keys will be written to ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub, although you can change the filename to something else.

Similarly, you can explicitly state that you want a RSA key by using:

ssh-keygen -t rsa

If you want a different key size, you can specify it using the -b option. RSA keys can vary from 1024 to 16384 bits, and Ed25519 keys can vary from 256 to 16384 bits. As an example, if you want a 4096-bit RSA key, you should use:

ssh-keygen -b 4096

Copying the public key to the server

Now that you have generated the key pair, you should copy the public key over to the server. Because public keys are used to authenticate a given user, the key lives in ~/.ssh/authorized_keys, for that user. In addition, this file should have the permission bits 400, so that other users aren’t able to read the file.

Most versions of OpenSSH come with the ssh-copy-id command, which can be used to automate this process. Let us assume that you’ve generated the public key in the default path. Now, if you want to copy the public key for the user booleanworld to the server 192.0.2.16, you should use:

ssh-copy-id booleanworld@192.0.2.16

However, if you have the key saved in a different location, you can use the -i switch to manually set the path:

ssh-copy-id -i ~/my-key.pub booleanworld@192.0.2.16

If you’re using a very old version of OpenSSH which doesn’t have this command, you can manually run this process. First, create the ~/.ssh directory for the user and set the permissions:

ssh booleanworld@192.0.2.16 'mkdir -p ~/.ssh; chmod 700 ~/.ssh'

Next, copy the public key into ~/.ssh/authorized_keys with scp. We will assume that you have the key in ~/.ssh/id_rsa.pub, which is the default location. So, to copy the file, run:

scp ~/.ssh/id_rsa.pub booleanworld@192.0.2.16:~/.ssh/authorized_keys

Finally, you can set the permissions of the key with another SSH command:

ssh booleanworld@192.0.2.16 'chmod 400 ~/.ssh/authorized_keys'

Using the SSH key to log in

Now that you have installed the key on the server, it’s time to log in. If you have the key in the default location, you simply need to log in as usual:

ssh booleanworld@192.0.2.16

On the other hand, if you have the private key in a custom location, you should set the path with the -i switch:

ssh -i ~/my-key booleanworld@192.0.2.16

If everything worked well, you should be able to log in without a password. However, if you opted for a password-protected private key, ssh will ask you for a password to decrypt the key, like so:

With the setup we have so far, you will be able to log in to your user account, either using a password or the private key. This doesn’t help with security since automated systems can still run brute-force attacks. Thus, in the next section, we will disable password authentication in the SSH server.

Before you continue any further, you should ensure all users who want to log in to the server have a public key configured on the server.

Disabling password-based authentication

To disable password authentication, you should edit the /etc/ssh/sshd_config file on the server. Here we’ll be using the nano editor, although you can use any other editor of your choice. If you are logged in to the server as a root user, run:

nano /etc/ssh/sshd_config

Otherwise, most systems have sudo to enable you to run commands as the root user, so you can use:

sudo nano /etc/ssh/sshd_config

However, if you do not have sudo installed, you can get a root shell by running su and entering the root user’s password. Then, you can use the commands for the root user as above.

Once you open the file, find the line containing the “PasswordAuthentication” directive. This line will typically look like:

#PasswordAuthentication yes

If there is a # in front of the “PasswordAuthentication” line, you should remove it. Then, change the yes to a no, and then save the file and exit the editor. However, if there is no such line, you can simply add the following line to the end of /etc/ssh/sshd_config:

PasswordAuthentication no

Once you’ve saved the file, you should restart the SSH server with the following command:

sudo service sshd restart # For older sysvinit/upstart systems
sudo systemctl restart sshd # For modern systemd-based systems

Now, we will check if password-based authentication still works. To do so, you should tell SSH to only use password authentication; otherwise, SSH always tries to use the keys (if you have them in the default location):

ssh -o PreferredAuthentications=password booleanworld@192.0.2.16

The server should reject the authentication attempt with an error similar to:

Managing multiple SSH keys

If you have multiple SSH keys for multiple servers, you have to remember the server and its corresponding key, which can become a hassle. You can specify a list of servers and keys in the ~/.ssh/config file on your local system.

As an example, if you want to configure SSH to use ~/ssh-keys/server1 when logging into 192.0.2.16, and ~/ssh-keys/server2 when logging into 192.0.2.34, you should use:

Host 192.0.2.16
 IdentitiesOnly yes
 IdentityFile ~/ssh-keys/server1

Host 192.0.2.34
 IdentitiesOnly yes
 IdentityFile ~/ssh-keys/server2

Now, when you try to log in to any of these two servers with the ssh <username>@<host> command, SSH will automatically use the key defined in the configuration file. For any other server, it will fall back to the default keys or password authentication (depending on server support and the default key in use).

Changing the passphrase of a SSH key

Previously, we’ve seen how you can generate public and private keys. If you want to add a passphrase to an unencrypted private key, or you want to change the passphrase for an encrypted private key, you can do so by using the -p flag in ssh-keygen like so:

ssh-keygen -pf <path to SSH private key>

If you run this on an unencrypted key, ssh-keygen should ask for the new password (like the example below). Likewise, if you have an encrypted key, ssh-keygen should ask for the old and new passphrases.

Once you’ve added or changed the passphrase on a key, you can log in with the usual ssh command.

If you have OpenSSH version 6.5 or later installed on your local system, you can opt to encrypt the key with a more secure password encryption format by using the -o flag. You can confirm the OpenSSH version on your system using:

ssh -V

If you have the required OpenSSH version, you can use the -o flag to encrypt the key as follows:

ssh-keygen -opf ~/.ssh/id_rsa

Remembering private key passphrases with ssh-agent

If you have a lot of passphrase-protected keys, having to type in the passphrase every time you want to log in to a server is a bit of a pain. Fortunately, OpenSSH comes with a program called ssh-agent, which can temporarily remember passphrases for you.

Before you can ask ssh-agent to remember passphrases for your keys, you should first start it up. MacOS and many Linux distributions already do this for you. A simple way to confirm if your OS does this is to look for the ssh-agent process using:

pgrep -u $USER ssh-agent

If this command doesn’t print anything, this means that your OS doesn’t start ssh-agent by default. So, you should add the following snippet at the end of your ~/.bashrc file:

if ! pgrep -u "$USER" ssh-agent > /dev/null; then
 ssh-agent | grep -Ev '^echo ' > ~/.ssh-agent-vars
fi

if [[ ! -z "$SSH_AUTH_SOCK" ]]; then
 . ~/.ssh-agent-vars
fi

This will start ssh-agent whenever you launch a terminal, if it isn’t already running in the background. However, this change will work only for any new terminal windows that you open.

Now, suppose you have a key in ~/.ssh/id_rsa, and you would like ssh-agent to remember the passphrase for it. Type in the following command to do so:

ssh-add ~/.ssh/id_rsa

ssh-add will ask for the passphrase for the key. Once you enter the passphrase and hit enter, ssh-agent will remember it till you shut down your computer. Now, you can try logging in as usual with ssh <user>@<host> and you won’t be prompted for the passphrase for the key.

However, this process leaves a lot to be desired. Every time you turn on your computer and want to log in to a new server, you have to use ssh-add so that ssh-agent “remembers” the passphrase for the key. If you forget this step, ssh will ask for the passphrase later as well.

OpenSSH 7.2 and later versions make this process seamless — you can configure it so that there’s no need to separately run ssh-add. If you’re using ssh and use a key that ssh-agent doesn’t know about, you need to enter the passphrase as usual. Once you enter the passphrase, ssh-agent will automatically save it and ssh won’t ask you about the passphrase later on.

If you have OpenSSH 7.2 or later, you can add this snippet to the ~/.ssh/config file to enable this feature:

Host *
 AddKeysToAgent yes

Conclusion

SSH key-based authentication provides a secure way of authenticating to a server. It significantly improves the security of your server by preventing brute-force attacks. Further, you can use a passphrase to encrypt the private key, so that the key can’t be used even if it is leaked to third-parties.

Although we have discussed everything with regards to the ssh command, the same steps are also applicable for other commands in the OpenSSH suite, such as scp and sftp.

cURL Command Tutorial with Examples

Mon, 17 Dec 2018 18:08:22 +0000

cURL is a command line tool and a library which can be used to receive and send data between a client and a server or any two machines connected over the internet. It supports a wide range of protocols like HTTP, FTP, IMAP, LDAP, POP3, SMTP and many more.

Due to its versatile nature, cURL is used in many applications and for many use cases. For example, the command line tool can be used to download files, testing APIs and debugging network problems. In this article, we shall look at how you can use the cURL command line tool to perform various tasks.

Install cURL

Linux

Most Linux distributions have cURL installed by default. To check whether it is installed on your system or not, type curl in your terminal window and press enter. If it isn’t installed, it will show a “command not found” error. Use the commands below to install it on your system.

For Ubuntu/Debian based systems use:

sudo apt update
sudo apt install curl

For CentOS/RHEL systems, use:

sudo yum install curl

On the other hand, for Fedora systems, you can use the command:

sudo dnf install curl

MacOS

MacOS comes with cURL preinstalled, and it receives updates whenever Apple releases updates for the OS. However, in case you want to install the most recent version of cURL, you can install the curl Homebrew package. Once you install Homebrew, you can install it with:

brew install curl

Windows

For Windows 10 version 1803 and above, cURL now ships by default in the Command Prompt, so you can use it directly from there. For older versions of Windows, the cURL project has Windows binaries. Once you download the ZIP file and extract it, you will find a folder named curl-<version number>-mingw. Move this folder into a directory of your choice. In this article, we will assume our folder is named curl-7.62.0-win64-mingw, and we have moved it under C:\.

Next, you should add cURL’s bin directory to the Windows PATH environment variable, so that Windows can find it when you type curl in the command prompt. For this to work, you need to follow these steps:

Open the “Advanced System Properties” dialog by running systempropertiesadvanced from the Windows Run dialog (Windows key + R).
Click on the “Environment Variables” button.
Double-click on “Path” from the “System variables” section, and add the path C:\curl-7.62.0-win64-mingw\bin. For Windows 10, you can do this with the “New” button on the right. On older versions of Windows, you can type in ;C:\curl-7.62.0-win64-mingw\bin (notice the semicolon at the beginning) at the end of the “Value” text box.

Once you complete the above steps, you can type curl to check if this is working. If everything went well, you should see the following output:

C:\Users\Administrator>curl
curl: try 'curl --help' or 'curl --manual' for more information

cURL basic usage

The basic syntax of using cURL is simply:

curl <url>

This fetches the content available at the given URL, and prints it onto the terminal. For example, if you run curl example.com, you should be able to see the HTML page printed, as shown below:

This is the most basic operation cURL can perform. In the next few sections, we will look into the various command line options accepted by cURL.

Downloading Files with cURL

As we saw, cURL directly downloads the URL content and prints it to the terminal. However, if you want to save the output as a file, you can specify a filename with the -o option, like so:

curl -o vlc.dmg http://ftp.belnet.be/mirror/videolan/vlc/3.0.4/macosx/vlc-3.0.4.dmg

In addition to saving the contents, cURL switches to displaying a nice progress bar with download statistics, such as the speed and the time taken:

Instead of providing a file name manually, you can let cURL figure out the filename with the -O option. So, if you want to save the above URL to the file vlc-3.0.4.dmg, you can simply use:

curl -O http://ftp.belnet.be/mirror/videolan/vlc/3.0.4/macosx/vlc-3.0.4.dmg

Bear in mind that when you use the -o or the -O options and a file of the same name exists, cURL will overwrite it.

If you have a partially downloaded file, you can resume the file download with the -C - option, as shown below:

curl -O -C - http://ftp.belnet.be/mirror/videolan/vlc/3.0.4/macosx/vlc-3.0.4.dmg

Like most other command line tools, you can combine different options together. For example, in the above command, you could combine -O -C - and write it as -OC - .

Anatomy of a HTTP request/response

Before we dig deeper into the features supported by cURL, we will discuss a little bit about HTTP requests and responses. If you are familiar with these concepts, you directly skip to the other sections.

To request a resource such as a webpage, or to submit some data to a server, a HTTP client (such as a browser or cURL) makes a HTTP request to the server The server responds back with a HTTP response, which contains the “contents” of that page.

HTTP requests contain the request method, URL, some headers, and some optional data as part of the “request body”. The request method controls how a certain request should be processed. The most common types of request methods are “GET” and “POST”. Typically, we use “GET” requests to retrieve a resource from the server, and “POST” to submit data to the server for processing. “POST” requests typically contain some data in the request body, which the server can use.

HTTP responses are similar and contain the status code, some headers, and a body. The body contains the actual data that clients can display or save to a file. The status code is a 3-digit code which tells the client if the request succeeded or failed, and how it should proceed further. Common status codes are 2xx (success), 3xx (redirect to another page), and 4xx/5xx (for errors).

HTTP is an “application layer protocol”, and it runs over another protocol called TCP. It takes care of retransmitting any lost data, and ensures that the client and server transmit data at an optimal rate. When you use HTTPS, another protocol called SSL/TLS runs between TCP and HTTP to secure the data.

Most often, we use domain names such as google.com to access websites. Mapping the domain name to an IP address occurs through another protocol called DNS.

You should now have enough background to understand the rest of this article.

Following redirects with cURL

By default, when cURL receives a redirect after making a request, it doesn’t automatically make a request to the new URL. As an example of this, consider the URL http://www.facebook.com. When you make a request using to this URL, the server sends a HTTP 3XX redirect to https://www.facebook.com/. However, the response body is otherwise empty. So, if you try this out, you will get an empty output:

If you want cURL to follow these redirects, you should use the -L option. If you repeat make a request for http://www.facebook.com/ with the -L flag, like so:

curl -L http://www.facebook.com/

Now, you will be able to see the HTML content of the page, similar to the screenshot below. In the next section, we will see how we can verify that there is a HTTP 3XX redirect.

Please bear in mind that cURL can only follow redirects if the server replied with a “HTTP redirect”, which means that the server used a 3XX status code, and it used the “Location” header to indicate the new URL. cURL cannot process Javascript or HTML-based redirection methods, or the “Refresh header”.

If there is a chain of redirects, the -L option will only follow the redirects up to 500 times. You can control the number of maximum redirects that it will follow with the --max-redirs flag.

curl -L --max-redirs 700 example.com

If you set this flag to -1, it will follow the redirects endlessly.

curl -L --max-redirs -1 example.com

Viewing response headers with cURL

When debugging issues with a website, you may want to view the HTTP response headers sent by the server. To enable this feature, you can use the -i option.

Let us continue with our previous example, and confirm that there is indeed a HTTP 3XX redirect when you make a HTTP request to http://www.facebook.com/, by running:

curl -L -i http://www.facebook.com/

Notice that we have also used -L so that cURL can follow redirects. It is also possible to combine these two options and write them as -iL or -Li instead of -L -i.

Once you run the command, you will be able to see the HTTP 3XX redirect, as well as the page HTTP 200 OK response after following the redirect:

If you use the -o/ -O option in combination with -i, the response headers and body will be saved into a single file.

Viewing request headers and connection details

In the previous section, we have seen how you can view HTTP response headers using cURL. However, sometimes you may want to view more details about a request, such as the request headers sent and the connection process. cURL offers the -v flag (called “verbose mode”) for this purpose, and it can be used as follows:

curl -v /

The output contains request data (marked with >), response headers (marked with <) and other details about the request, such as the IP used and the SSL handshake process (marked with *). The response body is also available below this information. (However, this is not visible in the screenshot below).

Most often, we aren’t interested in the response body. You can simply hide it by “saving” the output to the null device, which is /dev/null on Linux and MacOS and NUL on Windows:

curl -vo /dev/null / # Linux/MacOS
curl -vo NUL / # Windows

Silencing errors

Previously, we have seen that cURL displays a progress bar when you save the output to a file. Unfortunately, the progress bar might not be useful in all circumstances. As an example, if you hide the output with -vo /dev/null, a progress bar appears which is not at all useful.

You can hide all these extra outputs by using the -s header. If we continue with our previous example but hide the progress bar, then the commands would be:

curl -svo /dev/null / # Linux/MacOS
curl -svo NUL / # Windows

The -s option is a bit aggressive, though, since it even hides error messages. For your use case, if you want to hide the progress bar, but still view any errors, you can combine the -S option.

So, if you are trying to save cURL output to a file but simply want to hide the progress bar, you can use:

curl -sSvo file.html /

Setting HTTP request headers with cURL

When testing APIs, you may need to set custom headers on the HTTP request. cURL has the -H option which you can use for this purpose. If you want to send the custom header X-My-Custom-Header with the value of 123 to https://httpbin.org/get, you should run:

curl -H 'X-My-Custom-Header: 123' https://httpbin.org/get

(httpbin.org is a very useful website that allows you to view details of the HTTP request that you sent to it.)

The data returned by the URL shows that this header was indeed set:

You can also override any default headers sent by cURL such as the “User-Agent” or “Host” headers. The HTTP client (in our case, cURL) sends the “User-Agent” header to tell the server about the type and version of the client used. Also, the client uses the “Host” header to tell the server about the site it should serve. This header is needed because a web server can host multiple websites at a single IP address.

Also, if you want to set multiple headers, you can simply repeat the -H option as required.

curl -H 'User-Agent: Mozilla/5.0' -H 'Host: www.google.com' ...

However, cURL does have certain shortcuts for frequently used flags. You can set the “User-Agent” header with the -A option:

curl -A Mozilla/5.0 http://httpbin.org/get

The “Referer” header is used to tell the server the location from which they were referred to by the previous site. It is typically sent by browsers when requesting Javascript or images linked to a page, or when following redirects. If you want to set a “Referer” header, you can use the -e flag:

curl -e http://www.google.com/ http://httpbin.org/get

Otherwise, if you are following a set of redirects, you can simply use -e ';auto' and cURL will take care of setting the redirects by itself.

Making POST requests with cURL

By default, cURL sends GET requests, but you can also use it to send POST requests with the -d or --data option. All the fields must be given as key=value pairs separated by the ampersand ( &) character. As an example, you can make a POST request to httpbin.org with some parameters:

curl --data "firstname=boolean&lastname=world" https://httpbin.org/post

From the output, you can easily tell that we posted two parameters (this appears under the “form” key):

Any special characters such as @, %, = or spaces in the value should be URL-encoded manually. So, if you wanted to submit a parameter “email” with the value “test@example.com”, you would use:

curl --data "email=test%40example.com" https://httpbin.org/post

Alternatively, you can just use --data-urlencode to handle this for you. If you wanted to submit two parameters, email and name, this is how you should use the option:

curl --data-urlencode "email=test@example.com" --data-urlencode "name=Boolean World" https://httpbin.org/post

If the --data parameter is too big to type on the terminal, you can save it to a file and then submit it using @, like so:

curl --data @params.txt example.com

So far, we have seen how you can make POST requests using cURL. If you want to upload files using a POST request, you can use the -F (“form”) option. Here, we will submit the file test.c, under the parameter name file:

curl -F file=@test.c https://httpbin.org/post

This shows the content of the file, showing that it was submitted successfully:

Submitting JSON data with cURL

In the previous section, we have seen how can submit POST requests using cURL. You can also submit JSON data using the --data option. However, most servers expect to receive a POST request with key-value pairs, similar to the ones we have discussed previously. So, you need to add an additional header called ‘Content-Type: application/json’ so that the server understands it’s dealing with JSON data and handles it appropriately. Also, you don’t need to URL-encode data when submitting JSON.

So if you had the following JSON data and want to make a POST request to https://httpbin.org/post:

{
 "email": "test@example.com",
 "name": ["Boolean", "World"]
}

Then, you can submit the data with:

curl --data '{"email":"test@example.com", "name": ["Boolean", "World"]}' -H 'Content-Type: application/json' https://httpbin.org/post

In this case, you can see the data appear under the json value in the httpbin.org output:

You can also save the JSON file, and submit it in the same way as we did previously:

curl --data @data.json https://httpbin.org/post

Changing the request method

Previously, we have seen how you can send POST requests with cURL. Sometimes, you may need to send a POST request with no data at all. In that case, you can simply change the request method to POST with the -X option, like so:

curl -X POST https://httpbin.org/post

You can also change the request method to anything else, such as PUT, DELETE or PATCH. One notable exception is the HEAD method, which cannot be set with the -X option. The HEAD method is used to check if a document is present on the server, but without downloading the document. To use the HEAD method, use the -I option:

curl -I /

When you make a HEAD request, cURL displays all the request headers by default. Servers do not send any content when they receive a HEAD request, so there is nothing after the headers:

Replicating browser requests with cURL

If you want to replicate a request made through your browser through cURL, you can use the Chrome, Firefox and Safari developer tools to get a cURL command to do so.

The steps involved are the same for all platforms and browsers:

Open developer tools in Firefox/Chrome (typically F12 on Windows/Linux and Cmd+Shift+I on a Mac)
Go to the network tab
Select the request from the list, right click it and select “Copy as cURL”

The copied command contains all the headers, request methods, cookies etc. needed to replicate the exact same request. You can paste the command in your terminal to run it.

Making cURL fail on HTTP errors

Interestingly, cURL doesn’t differentiate between a successful HTTP request (2xx) and a failed HTTP request (4xx/5xx). So, it always returns an exit status of 0 as long as there was no problem connecting to the site. This makes it difficult to write shell scripts because there is no way to check if the file could be downloaded successfully.

You can check this by making a request manually:

curl /404 -sSo file.txt

You can see that curl doesn’t print any errors, and the exit status is also zero:

If you want to consider these HTTP errors as well, you can use the -f option, like so:

curl /404 -fsSo file.txt

Now, you can see that cURL prints an error and also sets the status code to 22 to inform that an error occured:

Making authenticated requests with cURL

Some webpages and APIs require authentication with an username and password. There are two ways to do this. You can mention the username and password with the -u option:

curl -u boolean:world https://example.com/

Alternatively, you can simply add it to the URL itself, with the <username>:<password>@<host> syntax, as shown:

curl https://boolean:world@example.com/

In both of these methods, curl makes a “Basic” authentication with the server.

Testing protocol support with cURL

Due to the wide range of protocols supported by cURL, you can even use it to test protocol support. If you want to check if a site supports a certain version of SSL, you can use the --sslv<version> or --tlsv<version> flags. For example, if you want to check if a site supports TLS v1.2, you can use:

curl -v --tlsv1.2 /

The request takes place normally, which means that the site supports TLSv1.2. Now, let us check if the site supports SSL v3:

curl -v --sslv3 /

This command throws a handshake_failed error, because the server doesn’t support this version of SSL.

Please note that, depending on your system and the library version/configuration, some of these version options may not work. The above output was taken from Ubuntu 16.04’s cURL. However, if you try this with cURL in MacOS 10.14, it gives an error:

You can also test for HTTP protocol versions in the same way, by using the flags --http1.0, --http1.1 or --http2.

Setting the Host header and cURL’s `--resolve` option

Previously, we have discussed about how a web server chooses to serve different websites to visitors depending upon the “Host” header. This can be very useful to check if your website has virtual hosting configured correctly, by changing the “Host” header. As an example, say you have a local server at 192.168.0.1 with two websites configured, namely example1.com and example2.com. Now, you can test if everything is configured correctly by setting the Host header and checking if the correct contents are served:

curl -H 'Host: example1.com' http://192.168.0.1/
curl -H 'Host: example1.com' http://192.168.0.1/

Unfortunately, this doesn’t work so well for websites using HTTPS. A single website may be configured to serve multiple websites, with each website using its own SSL/TLS certificate. Since SSL/TLS takes place at a lower level than HTTP, this means clients such as cURL have to tell the server which website we’re trying to access at the SSL/TLS level, so that the server can pick the right certificate. By default, cURL always tells this to the server.

However, if you want to send a request to a specific IP like the above example, the server may pick a wrong certificate and that will cause the SSL/TLS verification to fail. The Host header only works at the HTTP level and not the SSL/TLS level.

To avoid the problem described above, you can use the --resolve flag. The resolve flag will send the request to the port and IP of your choice but will send the website name at both SSL/TLS and HTTP levels correctly.

Let us consider the previous example. If you were using HTTPS and wanted to send it to the local server 192.168.0.1, you can use:

curl https://example1.com/ --resolve example1.com:192.168.0.1:443

It also works well for HTTP. Suppose, if your HTTP server was serving on port 8080, you can use either the --resolve flag or set the Host header and the port manually, like so:

curl http://192.168.0.1:8080/ -H 'Host: example1.com:8080'
curl http://example.com/ --resolve example1.com:192.168.0.1:8080

The two commands mentioned above are equivalent.

Resolve domains to IPv4 and IPv6 addresses

Sometimes, you may want to check if a site is reachable over both IPv4 or IPv6. You can force cURL to connect to either the IPv4 or over IPv6 version of your site by using the -4 or -6 flags.

Please bear in mind that a website can be reached over IPv4 and IPv6 only if:

There are appropriate DNS records for the website that links it to IPv4 and IPv6 addresses.
You have IPv4 and IPv6 connectivity on your system.

For example, if you want to check if you can reach the website icanhazip.com over IPv6, you can use:

curl -6 https://icanhazip.com/

If the site is reachable over HTTPS, you should get your own IPv6 address in the output. This website returns the public IP address of any client that connects to it. So, depending on the protocol used, it displays an IPv4 or IPv6 address.

You can also use the -v option along with -4 and -6 to get more details.

Disabling cURL’s certificate checks

By default, cURL checks certificates when it connects over HTTPS. However, it is often useful to disable the certificate checking, when you are trying to make requests to sites using self-signed certificates, or if you need to test a site that has a misconfigured certificate.

To disable certificate checks, use the -k certificate. We will test this by making a request to expired.badssl.com, which is a website using an expired SSL certificate.

curl -k https://expired.badssl.com/

With the -k option, the certificate checks are ignored. So, cURL downloads the page and displays the request body successfully. On the other hand, if you didn’t use the -k option, you will get an error, similar to the one below:

Troubleshooting website issues with “cURL timing breakdown”

You may run into situations where a website is very slow for you, and you would like to dig deeper into the issue. You can make cURL display details of the request, such as the time taken for DNS resolution, establishing a connection etc. with the -w option. This is often called as a cURL “timing breakdown”.

As an example, if you want to see these details for connecting to /, run:

curl / -sSo /dev/null -w 'namelookup:\t%{time_namelookup}\nconnect:\t%{time_connect}\nappconnect:\t%{time_appconnect}\npretransfer:\t%{time_pretransfer}\nredirect:\t%{time_redirect}\nstarttransfer:\t%{time_starttransfer}\ntotal:\t\t%{time_total}\n'

(If you are running this from a Windows system, change the /dev/null to NUL).

You will get some output similar to this:

Each of these values is in seconds, and here is what each value represents:

namelookup — The time required for DNS resolution.
connect — The time required to establish the TCP connection.
appconnect — This is the time taken to establish connections for any layers between TCP and the application layer, such as SSL/TLS. In our case, the application layer is HTTP. Also, if there is no such intermediate layer (such as when there is a direct HTTP request), this time will always be 0.
pretransfer — This is the time taken from the start to when the transfer of the file is just about to begin.
redirect — This is the total time taken to process any redirects.
starttransfer — Time it took from the start to when the first byte is about to be transferred.
total — The total time taken for cURL to complete the entire process.

As an example, say, you are facing delays connecting to a website and you notice the “namelookup” value was too high. As this indicates a problem with your ISP’s DNS server, you may start looking into why the DNS lookup is so slow, and switch to another DNS server if needed.

cURL configuration files

Sometimes, you may want to make all cURL requests use the same options. Passing these options by hand isn’t a feasible solution, so cURL allows you to specify options in a configuration file.

The default configuration file is located in ~/.curlrc in Linux/MacOS and %appdata%\_curlrc in Windows. Inside this file, you can specify any options that you need, such as:

# Always use IPv4
-4
# Always show verbose output
-v
# When following a redirect, automatically set the previous URL as referer.
referer = ";auto"
# Wait 60 seconds before timing out.
connect-timeout = 60

After creating the above file, try making a request with curl example.com. You will find that these options have taken effect.

If you want to use a custom configuration file instead of the default one, then you can use -K option to point curl to your configuration file. As an example, if you have a configuration file called config.txt, then you can use it with:

curl -K config.txt example.com

Conclusion

In this article, we have covered the most common uses of the cURL command. Of course, this article only scratches the surface and cURL can do a whole lot of other things. You can type man curl in your terminal or just visit this page to see the man page which lists all the options.

OpenVZ vs KVM vs Xen - Virtualization Technologies Explained

Sun, 09 Sep 2018 15:17:08 +0000

There are many different virtualization technologies available in the market, such as OpenVZ, KVM and Xen. You may have come across these terms when trying to while trying to buy a Virtual Private Server (VPS). In this article, we will discuss and compare between all these three technologies from the perspective of buying a VPS, so that you can choose the one that is best suited to your requirements.

Virtualization and Containerization Overview

Virtualization is a technology that allows you to create multiple “virtual machines” (VMs) that run on the same hardware. Each VM acts just like a physical computer would, and you can install and run an operating system on the VMs. These VMs are managed by a “hypervisor”, which allocates a part of the base (or host) system’s resources (such as the memory, CPU and hard disk) for use by the VM.

Each VM is isolated from each other — software running on a given VM does not have access to the resources of another VM. Many VPS providers use hypervisors installed on physical servers, and provision VMs to individual users as VPSes.

A type-2 hypervisor architecture

Containerization is a very different approach from virtualization. Instead of having a hypervisor, there is an OS installed on the host system, and you can create “containers” on it. Inside these containers, you can run your own applications, and the OS takes care of allocating resources to each container. The underlying kernel and drivers are shared among all these containers.

The general architecture of a containerization system

Thus, containerization depends on the OS, and it is only possible to run programs in the container that are compatible with the host system’s OS. For example, if a containerization solution works with Linux as the host system, you can only run Linux applications inside the container. This is unlike virtualization, where it is possible to run any operating system (and thus any application) in a virtual machine. On the other hand, containerization is much more efficient than virtualization, because there is no overhead of an OS running inside a VM.

For the purposes of this article, we are going to focus on “system containerization”. This kind of containerization allows you to run an OS inside the container. However, the kernel and drivers are shared between the various operating systems inside each container.

Xen and KVM are virtualization technologies, whereas OpenVZ is a Linux-based containerization technology.

OpenVZ

OpenVZ (or Open Virtuozzo) is a Linux-system containerization platform which enables one to run Linux-based OSes in a single host system with a shared kernel. The containers behave like an independent Linux system, with root level access, and isolation at the level of files, users/groups, processes and networking.

Server providers provision OpenVZ containers with some amount of RAM, CPU cores, hard disk space, and sell them as Linux VPSes. Some part of the CPU/memory resources are “dedicated” to a container, and some of it is “burstable” — if a container needs some additional CPU/memory apart from what has been allocated to it, it can temporarily borrow them from the unused resources of other containers.

Since OpenVZ uses a shared kernel, it is not possible to change the kernel’s settings, upgrade the kernel or use additional kernel modules. Many providers use OpenVZ 6, which uses an old Linux 2.6 kernel. Thus, you miss out on the performance benefits and newer features of newer kernels, and you’re stuck with a selection of old Linux distributions. In particular, you will be unable to run Docker, or use tools like ipset or nftables.

OpenVZ 7 is the latest version, which features a modern kernel, and can run Docker. However, there are very few providers who offer it, due to deployment issues and a lack of supporting tools.

Finally, from a server provider’s perspective, OpenVZ is easy to configure and get started with, compared to KVM and Xen. Also, since it is a containerization system, OpenVZ has much lower overhead, and thus providers can provision more VPSes out of a single physical server.

Xen

Xen is an open-source virtualization platform that originally began as a research project in the University of Cambridge. At present, the Linux Foundation helps with the development of the project.

With the help of various tools, server provider provision Xen VMs with a fixed amount of RAM, CPU cores, hard disk space and IP addresses, and offer them as VPSes.

Generally, hypervisors are categorized as Type 1 and Type 2 hypervisors. Type 1 hypervisors run directly on the host, whereas Type 2 hypervisors run on top of an existing operating system. Xen is a type 1 hypervisor.

Since Xen is a virtualization technology, a VM created in this way can run any OS — including Linux, Windows and BSD. Because each VM is running a full operating system, you can upgrade the kernel, use additional kernel modules or change low-level kernel settings.

Running a virtualization setup carries more overhead since there is the overhead of emulating certain hardware features, as well as that of running the OS. To reduce the former overhead, Xen uses a technique called “paravirtualization”. Here, the hypervisor exposes alternate ways to perform the same hardware operations, in a way that is more performant. If the guest OS knows how to make use of these alternative interfaces, it makes a “hypercall” to talk to the hypervisor. This mode of operation is called Xen Paravirtualization (Xen-PV).

When the guest OS does support paravirtualization, a different approach called Xen Hardware Virtual Machine (Xen-HVM) is used. In this mode, Xen uses QEMU to provide complete hardware emulation. To use Xen-HVM, hardware assisted virtualization should be provided by the host system.

KVM

KVM (Kernel Virtual Machine) is a Linux kernel module that provides a framework for third-party tools (such as QEMU) to provide virtualization. Since it is a kernel module, KVM reuses many functions of the Linux kernel for its purposes.

From an end-user perspective, Xen is similar to KVM, in that it allows you to run any OS and tinker with low-level kernel settings. Server providers use third-party tooling to create VMs with a fixed amount of RAM, CPU cores, hard disk space and IP addresses, and offer them as VPSes. Sometimes, VPS providers using KVM even have an option where you can upload your own ISO file to be installed on the VPS.

KVM only runs on hardware that supports hardware-assisted virtualization. Similar to Xen, KVM also provides paravirtualization for I/O devices through the “virtio” API.

Which one should you choose?

The platform that you choose ultimately depends upon your needs. If you need a Linux server for a low cost and don’t mind the old kernel version and the inability to use tools like Docker, you can use OpenVZ. If you need a different OS such as Windows, or you want to use a more recent Linux kernel, you should choose KVM or Xen.

Due to the the “burst” feature in OpenVZ, many providers oversell their systems to accommodate as many VPSes as possible on a single host. So, if too many VPSes have a demand for CPU/memory at the same time, you will get noticeably degraded performance from the VPS.

There are many providers who advertise their KVM/Xen offerings as having “dedicated resources”, but unfortunately this is also not always true. Both KVM and Xen offer a “memory ballooning” feature, where memory that’s unused by a VPS can be claimed by another VPS. A ballooning driver is installed in each VPS which aids in the process. When the hypervisor takes memory away from your VPS, from within your VPS it appears as if the ballooning driver is hogging your memory. However, it is never possible for a VPS to get more memory than it was initially allocated with.

Thus, overselling is possible on all the three platforms. However, providers typically oversell their KVM/Xen offerings a lot less than OpenVZ, due to technical limitations on hypervisor-based systems.

So, to really understand the “performance” of a VPS, you should always use benchmarking tools like bench.sh, speedtest-cli or Geekbench. Additionally, before buying a VPS based on any technology — OpenVZ, KVM or Xen, it is useful to review their pricing and what other people have to say about the company. A provider with rock-bottom pricing or with a bad reputation will have poorly performing VPSes, regardless of the technology used.

How to Download Files and Web Pages with Wget

Thu, 21 Jun 2018 16:35:22 +0000

The wget command allows you to download files over the HTTP, HTTPS and FTP protocols. It is a powerful tool that allows you to download files in the background, crawl websites, and resume interrupted downloads. Wget also features a number of options which allow you to download files over extremely bad network conditions.

In this article, we will learn how to use wget to achieve common downloading tasks from the command line.

Installing wget

Linux

Most Linux distributions have wget installed by default. To check whether it is installed on your system or not, type wget on your terminal and press enter. If it is not installed, it will display “command not found” error. Use the respective commands below to install it on your system.

For Ubuntu/Debian based systems use:

sudo apt update
sudo apt install wget

For CentOS/RHEL systems, use:

sudo yum install wget

MacOS

wget is available as a Homebrew package, so head over to the Homebrew page and follow the instructions to install it. Once you have done so, you can install wget with:

brew install wget

Windows

wget comes as part of msys2, a project that aims to provide a set of Unix-like command line tools. Go to the msys2 homepage and follow the instructions on the page to install it. Then, open a msys2 window and type in:

pacman -S wget

Now, you can access the wget command through the MSYS2 shell.

Downloading a file with its URL

If you have the link for a particular file, you can download it with wget by simply providing the URL as its parameter, like so:

wget http://releases.ubuntu.com/18.04/ubuntu-18.04-desktop-amd64.iso

If you run the above command, it will display progress information, such as the current download speed and how much of the file was downloaded:

wget infers a file name from the last part of the URL, and it downloads into your current directory. So, in our example, the file will be saved to ubuntu-18.04-desktop-amd64.iso.

If there are multiple files, you can specify them one after the other:

wget <url_1> <url_2>

Note that wget works only if the file is directly accessible with the URL. Occasionally, you will come across files that you can’t download directly with a link — such as a file which can be accessed after logging in. There are ways in which you can get around these restrictions, which we’ve discussed later in this article.

Downloading files over FTP

Wget works in the same way for FTP — you provide the FTP URL as an argument, like so:

wget ftp://ftp.example.com/ubuntu/ubuntu-18.04-desktop-amd64.iso

Sometimes, a FTP file may be protected with an username and a password. You can pass them onto wget like so:

wget --ftp-user=booleanworld --ftp-password=passw0rd <ftp_url>

Customizing the output file name and directory

As we’ve seen previously, wget infers file names and it downloads to your current directory. If you want to save the file to a different directory or under a different name, you can use the -O switch. For example, if you want to save the download to a file Ubuntu.iso, run:

wget <url> -O Ubuntu.iso

You can also change the directory in this way. For example, to save the download as a file Ubuntu.iso under /media/sdb1/Software, run:

wget <url> -O /media/sdb1/Software/Ubuntu.iso

Sometimes, you may want to specify a directory, but let wget figure out the file name. You can do so by using the -P switch. If you want to save a file under the /media/sdb1/Software directory, use:

wget <url> -P /media/sdb1/Software

Resuming interrupted downloads

A download may be interrupted due to bad network conditions, or because you’ve interrupted them manually by pressing Ctrl+C while wget was still downloading a file. You can resume an interrupted download with wget’s -c switch:

wget -c <url>

If you’ve downloaded to a custom directory or file, you should combine the -c and the -P/ -O switches, like so:

wget -c <url> -O <path_to_downloaded_file>

If you’re using -O and -c, be sure to provide the correct URL. Otherwise, you’ll end up with a partial combination of two different files, rendering it unusable. In addition, some servers don’t allow you to resume file downloads. (You can tell if a server supports resuming by looking at the progress bar. If, during the initial download, the progress bar moves back and forth instead of left to right, the server doesn’t allow resuming.) In that case, the file will be downloaded afresh from the beginning.

Throttling download speeds

If you’re downloading a big file, you may want to control the download speed so that you have additional bandwidth left for browsing the web. You can do this with the --limit-rate switch. For example, to limit the download speed to 512 KB/s, use:

wget <url> --limit-rate=512k

Similarly, you can use the m suffix for a limit in MB/s.

Downloading files in the background

By default, wget downloads files in the foreground, which might not be suitable in every situation. As an example, you may want to download a file on your server via SSH. However, you don’t want to keep a SSH connection open and wait for the file to download. To download files in the background, you can use the -b option like so:

wget -b <url>

When you run this command, wget will tell you where it’s going to save a download log. This file is typically named wget-log, wget-log.1 or similar:

Later, if you want to monitor the status of the download, you can do so with:

tail -f wget-log

This will show you the last few lines of the download status:

You can press Ctrl+C any time to stop viewing the progress. However, the download progress will continue in the background as usual.

Downloading files over bad network connections

If you are on a patchy internet connection, downloads can often fail, or happen at very slow rates. By default, wget retries a download for up to 20 times in case problems arise. However, on particularly bad internet connections, this might not be enough. If you notice slow download rates with frequent errors, you can run:

wget -t inf --waitretry=3 --timeout=10 --retry-connrefused <url>

First, we’ve set the number of tries (the -t switch) to inf. Should wget face problems downloading the file, it will try infinitely many times as needed to complete the download. Then, we’ve used --waitretry to set the delay between each retry to 3 seconds.

Next, we’ve set the network timeout to 10 seconds with the --timeout switch. In case wget doesn’t hear back from the server for more than 10 seconds, it assumes that the connection is down. Doing this helps wget reissue network requests to fetch the file.

Finally, we’ve used the --retry-connrefused switch. Typically, a “connection refused” error occurs when the server is not running. However, it can also occur on unreliable network connections, and this switch tells wget to retry downloading in case it gets a connection refused error.

Downloading files recursively

So far, we’ve seen how to download particular files with wget. Sometimes, it is more useful to download related parts of a website. Wget has a “recursive downloading” feature for this purpose. In this mode, wget downloads the initial file, saves it, and scans it for links. Then, it downloads each of these links, saves these files, and extracts links out of them. By default, this process continues for up to five times. However, you can increase or decrease this limit (called “recursion depth”), as per your needs.

For example, to download the pages of the English Wikipedia recursively, use:

wget -r https://en.wikipedia.org/

These files will be saved in a directory with the site’s name. In our case, the contents would be saved to a directory named en.wikipedia.org.

To control the “recursion depth”, use the -l flag. If you want it to set it to a custom value, such as 7 use:

wget -r -l 7 https://en.wikipedia.org/

You can also use inf in place of a number. This makes wget retrieve all content of a website, with an infinite recursion depth.

By default, wget downloads all files that it finds in recursive mode. If you’re interested only in certain types of files, you can control this with the -A (accept) options. For example, if you want to download PNG and JPEG images from Wikipedia, use:

wget -r https://en.wikipedia.org/ -A png,jpg,jpeg

This will download all files from Wikipedia (upto 5 levels), but remove any files that don’t end in the extensions png, jpg or jpeg. Similarly, you can also reject certain files with the -R switch. For example, to save all files from Wikipedia except for PDF documents, use:

wget -r https://en.wikipedia.org/ -R pdf

Converting links in downloaded files

When recursively downloading files, wget downloads the files and saves them as-is. The downloaded webpages will still have links pointing to the website, which means you cannot use this copy for offline use. Fortunately, wget has a link conversion feature — it converts the links in a web page to local links. To use this feature, use the -k switch like so:

wget -k -r https://en.wikipedia.org/

Setting headers and user-agent strings

HTTP clients (such as browsers and tools like wget) send a “user-agent string” as part of HTTP headers, that identify the client. Many webmasters block wget by configuring their server to block requests containing “wget” in this string. To get around this, you can override wget’s user agent string with a custom one; such as the one used by your browser. To find out your browser’s user agent, go to a site such as echo.opera.com. Copy the user agent from the page, and run wget with the -U switch:

wget <url> -U "<user-agent-string>"

For example, if you were using a Mac, the command would look something like this:

wget <url> -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"

If you are a developer testing out your website’s functionality, you may need to tell wget to pass other HTTP headers. You can do so with the --header switch. For example, to send out the X-Foo header with the value of 123, run the following command:

wget <url> --header "X-Foo: 123"

If you want to set multiple custom headers, you can repeat the --header switch as many times as necessary.

Downloading protected files

So far, the examples we’ve seen cover downloading publicly available files. However, there are times when you need to download files from a login protected page. In these situations, you can use a browser extension like CurlWget (Chrome) or cliget (Firefox). When you try to download a file, these extensions will generate a complete wget command that you can use to download the file. In this article, we will only discuss CurlWget, although cliget works in a similar way.

Once you’ve installed CurlWget on Chrome, head over to the extension settings and change the Tool option from “curl” to “wget”. Then, head over to Download settings (Three dot menu > Settings > Advanced) and select the option “Ask where to save each file before downloading”.

Next, try to download a file from within Chrome. Dismiss the dialog box asking for the download location, and click on the CurlWget icon on the toolbar. This will give you a wget command, with user-agent, cookie and other headers set, as shown below:

Now, you can copy this command and paste it in a terminal window to download the file. This is extremely useful for situations where you need to download a login protected file to a headless server.

Wget “spider” mode

Wget has a “web spider” feature, that fetches pages but does not save them. This is useful in case you have to check for broken links on a website. To use this option properly, you need to also enable “recursive downloading”, which allows wget to scan through the document and look for links to traverse.

If you want to scan https://example.com/, run:

wget --spider -r https://example.com/

After wget has retrieved all the links, it will show any broken links, as shown:

You can also save this log to a file with the -o option:

wget --spider -r https://example.com -o wget.log

Later, you can use grep to search the log for particular status codes or files. For example, to find any 301 redirects on your site, you can use:

grep -B 301 wget.log

How to Kill a Process in Linux

Tue, 15 May 2018 16:43:12 +0000

In an operating system, there are many programs, which may be either run by an user or by the OS itself (such as system services). Such programs which are running on the system are called “processes”. Usually, a process terminates on its own when they’re done with their task, or when you ask them to quit by pressing a keyboard shortcut or clicking on the “Close” button.

However, sometimes a process can hang up or consume a lot of CPU or RAM. In this situation, you would want to manually “kill” the process. In this article, we will look at various tools you can use to kill processes on a Linux system.

Locating the process to kill

In order to kill a process, you should first locate the details of the process. You can do this through three commands — top, ps, pidof and pgrep. Depending upon the situation, you can use one of these commands for this purpose.

As we will see later in this article, you can kill a process by its name or its process ID (PID). The PID is a number that uniquely identifies a process. Killing by the process ID is useful when you want to kill only a specific process. On the other hand, killing by the process name is useful when you want to kill all running instances of a particular program.

Locating the process with the top command

We will first look at the top command. Fire up the top command by typing:

top

You will get an interactive interface, as shown below. You can browse through this list to find the name or the PID of the process you want to kill.

To browse through this list, you can use the up/down keys. Additionally, the top command also has ways to filter processes by CPU usage, user and process names, which you can read about in this guide.

The leftmost column contains the PID of the process, and the right side contains the program name. As an example, in the above screenshot we have the vnstatd process running with a process ID of 263.

Locating the process with ps and grep commands

Another way to get a list of process is by running:

ps aux

In the above command, we have used the flags aux which have the following meanings:

a: Show processes for all users
u: Display the user who is using the process
x: Show all processes. (Without this, ps won’t show processes running in a GUI environment.)

The output of the command is similar to that of top. The PID is available in second column from the left, and the process name is available on the rightmost column.

The advantage of using ps is that you can easily filter this list with the grep command. For example, to find a process associated with the term “vnstat”, you can use:

ps aux | grep -i vnstat

Here, we got two results — the vnstatd process, as well as the grep process. Since we were searching for all instances of the term “vnstat”, and we were also running grep with “vnstat” as its argument, we got grep as well in the results.

Thus, even when there are no “vnstat” related processes running, we would get one entry showing the grep process:

So, even though we got a result, there are no processes that are of interest to us.

Finding the PID with pidof and pgrep

The top and ps/ grep combination allows us to search for processes. On the other hand, if you know the exact name of a process, you can use pidof to find its PID.

Using pidof is pretty straightforward. To get the PIDs of a process with the exact name of “nginx”, use:

pidof nginx

If there are processes with the exact name of “nginx”, you will get a list of PIDs, as shown below. If there are none, you will get nothing as the output.

If you don’t know the full name, you can use pgrep instead of pidof. As an example, to search for all processes that contain “ngin” somewhere in their name, run:

pgrep ngin

This will match processes with the exact name of “nginx”, as well as any other process that matches the same criteria. For our system, notice that we get all the PIDs that belonged to “nginx” in the above screenshot.

The pidof and pkill commands give you far less information. As we shall see in the next section, there are some circumstances in which you can’t kill a process. The output of top and ps contain additional information that help you determine if you can really kill a process.

What processes can you kill?

Now that we have located the process, it is time to kill it. However, before we learn how to do so, there are a few things you need to know.

If you are a normal user, you can kill your own processes, but not those that belong to other users. Both top and ps show the user under which a process is running. In the case of top, the second column contains the username. With ps aux, the first column contains the username.

However, a root user can kill all processes. You can either add sudo before any command to run it as root, or obtain a root shell by typing su, and then execute the command.

In Linux, when a process is killed, a “terminating signal” is delivered to the process. Although there are many different types of signals, we mostly deal with the “SIGTERM” and “SIGKILL” signals. They have a numeric value of 15 and 9 respectively. By default, all the process killing commands use “SIGTERM”, which allows the program to run some code before it exits, thus allowing it to terminate “gracefully”. If you want to terminate the process forcibly, you can use “SIGKILL” instead.

The Linux kernel maintains some information related to the state of a process. When a process terminates, the kernel must keep the information around, so that the parent process can find out if the child process was able to complete its tasks and whether it terminated on its own, or it was killed. Until the parent has done so, these “zombie” processes will appear in the list of processes. You can’t kill such a process because it’s just an entry in the list of all processes, and it doesn’t have an actual process associated with it.

When a process performs input/output operations (such as reading from or writing to disks), it is said to be in a state of “uninterruptible sleep”. You can’t kill a process while it is in this state.

You can tell if a process is in the “zombie”(Z) or “uninterruptible sleep”(D) state by looking at the 8th column of the top/ ps output.

Killing a process

There are various commands you can use to kill a process — kill, killall, pkill and top. We will begin from the simplest one: the killall command.

Killing processes with the killall command

The killall command is one of the easiest ways to kill a process. If you know the exact name of a process, and you know that it’s not running as another user and it is not in the Z or D states, then you can use this command directly; there’s no need to manually locate the process as we described above.

By default, For example, to kill a process named “firefox”, run:

killall firefox

To forcibly kill the process with SIGKILL, run:

killall -9 firefox

You can also use -SIGKILL instead of -9.

If you want to kill processes interactively, you can use -i like so:

killall -i firefox

If you want to kill a process running as a different user, you can use sudo:

sudo killall firefox

You can also kill a process that has been running for a certain period of time with the -o and -y flags. So, if you want to kill a process that has been running for more than 30 minutes, use:

killall -o 30m <process-name>

If you want to kill a process that has been running for less than 30 minutes, use:

killall -y 30m <process-name>

Similarly, use the following abbreviations for the respective units of time: ssecondsmminuteshhoursddayswweeksMmonthsyyears

Killing processes with the pkill command

Sometimes, you only know part of a program’s name. Just like pgrep, pkill allows you to kill processes based on partial matches. For example, if you want to kill all processes containing the name apache in the name, run:

pkill apache

If you want to use a SIGKILL instead of a SIGTERM, use:

pkill -9 apache

Again, you can also use -SIGKILL instead of -9.

Killing processes with the kill command

Using the kill command is straightforward. Once you have found out the PID of the process that you want to kill, you can terminate it using the kill command. For example, if you want to kill a process having a PID of 1234, then use the following command:

kill 1234

As we mentioned previously, the default is to use a SIGTERM. To use a SIGKILL, use -9 or -SIGKILL as we have seen before:

kill -9 1234

Killing processes with the top command

It is very easy to kill processes using the top command. First, search for the process that you want to kill and note the PID. Then, press k while top is running (this is case sensitive). It will prompt you to enter the PID of the process that you want to kill.

After you enter the PID, press enter. Now it will ask which signal you want to use to kill the process. If you want to use SIGTERM(15), then simply press enter as it is the default signal. If you want to use SIGKILL(9), then type 9 and press enter.

If you leave the process ID blank and hit enter directly, it will terminate the topmost process in the list. You can scroll using the arrow keys, and change the process you want to kill in this way.

Conclusion

In this post, we saw the various ways to kill processes in Linux. Learning these commands is essential for proper system administration and management. If you want to explore more of those commands, have a look at their respective man pages.

A Guide to the Linux "Find" Command

Mon, 07 May 2018 06:49:31 +0000

The find command allows users to search for files and take actions on them. It is part of the “findutils” package and comes bundled with all distributions. It is highly flexible, allowing you to look for files and directories based on a variety of conditions. Optionally, it also allows you to take different types of actions on the results.

In this article, we will understand how to work with the find command. We will also illustrate its usage through various examples throughout this article.

Introduction

The basic structure of the find command is like so:

find [paths] [expression] [actions]

The find command takes a number of paths, and searches for files and directories in each path “recursively”. Thus, when the find command encounters a directory inside the given path, it looks for other files and directories inside it. Again, if there are other directories inside, the find command will look inside them as well. This process continues till it has searched for all items inside the path you specified.

By default, the find command finds everything inside a directory. If you want to filter some of them out based on certain criteria, you can specify an expression to do so.

The default action is to print all the results. However, you can also specify a custom action that the find command can take on the results.

These concepts will become clearer as we go through various examples mentioned later in this article.

Finding all files and directories

Imagine that you want to list all the directories and files for a given path. For example, if you want to list the contents of the /usr/share directory, run:

find /usr/share

This will give you a list of files and directories, as shown in the screenshot below. Depending upon the contents of the directory, this list can be very large!

If you want to list the contents of multiple directories, you can do this like so:

find /usr/share /bin /usr/lib

If you want to list the contents of the current working directory, use a period( .) as the path:

find .

When there are no paths, the find command assumes that it should work with the current directory. So, you can leave the . and simply use:

find

Finding items by their name

As we mentioned previously, you can filter the results of the find command using expressions. In this section, we will learn how to filter items by their name.

If you want to find a file or directory named NEWS.txt inside the /usr directory, use the -name switch like so:

find /usr -name NEWS.txt

The -name switch is case sensitive. If you don’t know the exact case of the item you’re looking for, you can use the -iname switch which is case insensitive:

find /usr -iname news.txt

The -name and -iname switches also accept “wildcards”, which are special characters that act as placeholders. There are two wildcards — the ? character represents a single unknown character, and the * character represents any number of unknown characters (including zero).

If you want to use a wildcard character, you should keep the name/ iname parameter inside single or double quotes. Since the * and ? characters are also special characters for the shell, putting them in quotes ensures that the command works correctly.

As an example, if you have to search for all files and directories inside /usr that have a .txt at the end of their name, you can use:

find /usr -name '*.txt'

If you want to search for files and directories with four letters in their name, run:

find /usr -name '????'

Sometimes, you may want to match the full path to a file/directory, instead of just matching the name. In such a situation you can use the -path switch. As an example, say you want to find all files and directories with a .txt at the end of their name, but only if they are in a directory named src. Thus, in the full path, there will be a /src/ somewhere in the middle, and a .txt at the end. So, the command will be:

find /usr -path '*/src/*.txt'

There is also the -ipath switch, which is the case insensitive version of -path.

Searching for files or directories

All the examples we’ve seen so far return files and directories. However, if you need to search for either files or directories only, you can use the -type switch. The most common parameters of -type are:

f: files
d: directories
l: symbolic links

For example, to find all the files in the /usr directory, run:

find /usr -type f

You can also combine the various switches of the find command. For example, to find all .txt files in the /usr directory, you can use:

find /usr -type f -name '*.txt'

Similarly, to find directories inside /usr that begin with “python”, use:

find /usr -type d -name 'python*'

Finding empty files

The find command supports the -empty flag to search for files and directories that are empty. An empty file is one that has no content in it, whereas an empty directory is one that has no files or subdirectories in it.

For example, if you want to find list empty directories in your home directory, you can run:

find ~ -type d -empty

Negating matches

The find command also allows you to “negate” a match. This is useful when we have a criteria to exclude from our search.

As an example, say you want to find files in the /usr directory which don’t have the .txt extension. You can negate the -name switch by adding a exclamation mark ( !) in front of it, as shown:

find /usr -type f ! -name '*.txt'

You can also invert any other switch. So, if you want to find the non-empty directories in your home directory, run:

find /usr -type f ! -empty

Searching based on ownership

The find command supports searching for files and directories based on their ownership information.

To find the files or directories that belong to a particular user, you can use the -user switch. For example, if you want to find all files on the system that belong to the user booleanworld, run:

find / -type f -user booleanworld

You can also use a user’s ID in the above command, instead of the username. You can look up an user’s ID with the following command:

id -u <username>

Likewise, the -group switch allows you to look for files and directories that belong to a particular group. It can also accept a group name, or a group ID. You can view a user’s group ID with the following command:

id -g <username>

Searching for files based on date and time

Sometimes, you may need to search for files depending on when it was accessed or modified. On a Linux system, there are three types of “time” associated with a file:

Modification time: The last time when the file’s content was modified.
Access time: The last time when the file was read.
Change time: The last time when the file (either its content or metadata, such as permissions) were modified.

To find files based on the modification, access or change time, the find command has the -mtime, -atime and -ctime switches. These switches allows you to filter files based on the number of days. Here, a “day” refers to a period of 24 hours.

Let us understand how to use these flags. As an example, if you use -mtime, then:

-mtime 2: The file was modified 2 days ago (i.e. in the last 48 to 72 hours).
-mtime -2: The file was modified less than 2 days ago (i.e. within the last 48 hours).
-mtime +2: The file was modified more than 2 days ago (i.e. 3 or more days, which means 72 hours or later).

So, to find files in /usr which were modified two days ago, run:

find /usr -type f -mtime 2

The -atime and -ctime switches work exactly in the same way as -mtime.

If you find that a day is a bit too long, you can also use -mmin, -amin or -cmin to filter in minutes. So, -amin +5 means files that were accessed more than 5 minutes ago, -amin -5 means the file was modified less than 5 minutes ago, and so on.

You can use many of these flags at once. If you want to find files modified 2 days ago and accessed 5 minutes ago, run:

find /usr -type f -mtime 2 -amin 5

You can even repeat the same flag! As an example, you want to find files modified between 50 and 100 days ago, use:

find /usr -type f -mtime +50 -mtime -100

Searching based on size

The find command has a -size switch to allow users to find files based on their size. Since only files can have a file size, when you use this switch no directories will be listed.

To specify the file size, we put a number followed by a letter to represent the unit. You can use the following letters:

c: bytes
k: kilobytes
M: megabytes
G: gigabytes

In addition, you can use a + or - to impose a “greater than” or “less than” condition.

Let us understand this with a few examples. If you want to list all files on the system with a size of 10 KB, use:

find / -size 10k

In order to find files that are more than 1 GB in size, use:

find / -size +1G

Similarly, to find files that are less than 10MB in size, use:

find / -size -10M

Searching based on permissions

The find command supports the -perm switch to allow searches based on permissions. It allows you to search based on both numeric and symbolic modes. If you’re unfamiliar with permissions, you can find an overview here.

Using symbolic permissions

Let us begin with a simple example. Suppose, you want to search for all files and directories in the /usr directory with permissions of rwxr-xr-x. In order to do so, run:

find /usr -perm u=rwx,g=rx,o=rx

The letters u, g and o stand for “user”, “group” and “owner” respectively.

You can also use the letter a to check for permissions that apply to all users. For example, to find all files and directories on the system with the permission r-xr-xr-x, run:

find /usr -perm a=rx

Often, we are interested to match a subset of the permissions. For example, you may want to search for files on the system that can be executed by all users. In this case, we only care that the execute bit is set for all three permission classes. However, the rest of the bits may be set or unset.

Since all permission classes should have the execute bit set, we write the permission as a=x. Then, we add a / in front to indicate that we intend to match a subset of permissions. Thus, the command is:

find / -type f -perm /a=x

Using numeric permissions

Let us consider the first example in the previous section. This is pretty straightforward — rwxr-xr-x is 644 in numeric mode, so you can simply run:

find /usr -perm 644

Checking for subset of permissions is a little more complex. As we mentioned previously, searching for files that everyone can execute involves checking if the execute bit is set. We don’t care about the other bits. For every bit that we should check, we put a 1, and for the rest we put a 0. With this process, we obtain a binary number and we convert it to octal as shown:

Next, we add a - in front of this number to indicate that we intend to match a subset of permissions. Thus, the command is:

find / -type f -perm -111

Searching based on access rights flags

Now, say you’re interested to search based on the numeric rights flags. For this purpose, you can use either numeric or symbolic modes.

As an example, say you’re interested to find all files on your system with the setuid bit set. We can extend what we learnt about numeric modes to this situation. In this case, we only care about the setuid flag, which has a value of 4. However, we don’t care about any of the permission bits, which means we get 000. Thus, we can search for these files using:

find / -type f -perm -4000

Of course, you can also filter for permission bits as well. If you’re interested in files that all users can execute, and also have the setuid bit set, replace 4000 with 4111 in the above command. (We’ve previously seen how we get the value 111.)

Similarly, you can use 2000 for the setgid bit and 1000 for the sticky bit. If you want to test for a combination of these bits, add up the values. For example, to test for the setuid and sticky bit, you would use 5000 (= 4000 + 1000).

If you want to do the same with symbolic modes, you can use:

find / -type f -perm /u=s

For the setgid bit, use /g=s, and for the sticky bit use /o=t. To test for a combination of these bits, separate them with commas, such as /u=s,o=t.

Limiting the depth of traversal

As we mentioned previously, the find command looks for items recursively. However, the results obtained may be very large, and you might want to set a limit on how deep the find command can search. This can be done with the -maxdepth switch.

If you run the following command, you will notice that all the displayed items are no more than three levels deep:

find / -maxdepth 3

In other situations, you may want to set a limit on the minimum depth. The find command also supports the -mindepth switch for this purpose. So, if you run:

find / -mindepth 3

Logical operations in the find command

In the previous sections, we have seen the flags supported by the find command. We have also seen how we can combine various flags and negate them. In some situations, we need more powerful constructs.

The find command supports “and” and “or” conditions with the -a and -o switches. It also supports grouping parts of an expression with parentheses (). However, since parentheses are also special characters for the shell, you should put them in single or double quotes. In addition, when you specify multiple flags one after the other, the find command automatically assumes an “and”.

Let us consider an example we saw before — finding directories inside /usr that begin with “python”. There, we used the command:

find /usr -type d -name 'python*'

You can also write the “and” explicitly like so:

find /usr -type d -a -name 'python*'

Now, let us consider a different example. Suppose, you want to search files/directories on the system which were modified in the last 5 minutes or earlier than 50 days. For this command, we will use the “or” operator like so:

find / -mmin -5 -o -mtime +50

Next, suppose we impose an additional restriction to the above problem — you want to search for files only. First, we will keep the conditions for filtering modified files in a group. Then, we will “and” it with the condition to find files. Thus, the final command is:

find / '(' -mmin -5 -o -mtime +50 ')' -a -type f

Since the -a is implicit, you can also leave it out of the above command.

Taking actions

So far, all the find commands print the results on the terminal. Usually, we want to perform some actions with these results, such as copying or deleting these items. Fortunately, the find command also supports taking “actions” on these results.

The default action is to print all the results; you can verify this by adding the -print switch at the end of any find command. There are many other actions as well. We will look at some of the most useful ones below.

Deleting files

To delete any files or directories, use the -delete action. It works with both files and directories. For example, if you want to delete all the empty directories from the home directory, run:

find ~ -type d -empty -delete

Executing commands

The find command also supports executing a custom command with the -exec switch.

Suppose, you want to back up all your MP3 audio from your home directory to your external hard drive. Assume that the external hard drive is available at /media/MyDrive. To copy a single file, you would use:

cp <path to file> /media/MyDrive

To copy all the files, you can use the following command.

find ~ -type f -name '*.mp3' -exec cp {} ';'

Here, the pair of braces ( {}) act as the placeholder for the path to the file. In order to tell the file command where the command ends, we use a semicolon( ;). However, since it is also a special character for the shell, we wrap it in single quotes.

Whenever the find command finds a file matching the condition, it replaces the braces with the actual path, and executes the command. So, the cp command is executed for every MP3 file.

Let us look at another important use — finding a string across many files. To find the string “hello” in a file, we use grep like so:

grep hello <file name>

If you are familiar with grep, you might be tempted to write a command like this:

find ~ -type f -exec grep hello {} ';'

However, if you use this command, you will immediately understand the problem. We want a list of files with the string “hello”; however we get a list of matched lines. Fortunately, grep has a -l switch, which makes it print the path of the file if there is a match. So, the command will be:

find ~ -type f -exec grep -l hello {} ';'

Executing commands: a different variant

Suppose, you want to create a compressed file. To create a Gzip-compressed file with the name music.tar.gz, you would run:

tar -czf music.tar.gz <list of files>

Now, suppose you want to compress all MP3 files in your home directory, to a single compressed file. Perhaps you would come up with the following:

find ~ -type f -name '*.mp3' -exec tar -czf music.tar.gz {} ';'

However, if you open the music.tar.gz file, you will notice that there’s only one MP3 file. Remember that find executes the command every time it finds a new file. So, the previous file got overwritten with a new archive every time; and we don’t want that to happen.

All of our problems could be solved if there was a way to tell find to pass a list of files to tar, after it has found all the files. This is where the second variant of the -exec switch comes in. Instead of ending the command with a ;, we use a + like so:

find ~ -type f -name '*.mp3' -exec tar -czf music.tar.gz {} +

Executing commands on directories

Sometimes, it is useful to execute a command on the directory inside which a file/directory present. The find command has a -execdir switch which does this. It is similar to -exec in all other ways, and even has the + and ; variants.

Viewing file information

If you want to view information about the files/directories (such as permissions and size), you can use the -ls switch. For example, if you want to view details about files on the system that are bigger than 1GB, run:

find / -type f -size +1G -ls

Removing error messages

While trying some of the find commands, you may have got some errors such as “Permission denied”. You can hide these error messages by redirecting them to /dev/null, as shown:

find [paths] [expression] [actions] 2>/dev/null

Conclusion

In this article, we’ve seen how to use the find command, along with examples that cover most of the use cases. If you want to read more, you can read the man page by typing man find in your terminal.

An Introduction to Linux File Permissions

Sun, 15 Apr 2018 09:01:17 +0000

Linux is a multi-user operating system, and it ensures the security of files with the concepts of “ownership” and “permissions”. You can use these concepts to control who can access and modify files. In this article, we will learn these concepts, and how you can change them using various commands.

Users and groups

Before we discuss the concept of “ownership”, let us first look at “users” and “groups”. Linux uses the concept of users to separate various people who use the computer. Every user has some properties associated with them, such as a user ID and a home directory. In order to make managing users easier, you can add users into a “group”. A group can have zero or more users. A particular user is associated with a “default group”, and can also be a member of other groups on the system.

If you want to view the users on your system, you can view the /etc/passwd file by running the following command:

cat /etc/passwd

Here, every line contains details of a user. Specifically, you can see the username at the beginning of each line, before the first :.

Similarly, you can view the groups on your system by viewing the /etc/group file, by running:

cat /etc/group

Again, each line contains the details of a user. The first part of each line contains the group name.

When running these commands, you will notice that there are a number of other users and groups that you didn’t create. These are system users and groups, which are used to run background processes securely.

Ownership and permissions

In Linux, we use permissions to control what a user can do with a file or directory. Linux uses three types of permissions:

Read: For a file, the read permission allows a user to view the contents of a file. For directories, the read permission allows the user to view the names of files and other directories stored in it.
Write: For a file, the write permission allows a user to modify and delete a file. For directories, the write permission allows a user to modify its contents (create, delete and rename files in it). However, this permission has no effect on directories unless the execute permission is also enabled.
Execute: When set on a file, the write permission allows it to be executed. However, the permission has no effect on a file unless the read permission is also enabled. On the other hand, for directories, the write permission allows the user to enter the directory (with cd) and view metadata (like file permissions) of the files and directories within it.

You might wonder: why won’t the write permission work without an execute permission on the directory? This is because of the fact that you can only retrieve the permissions of a directory when its execute bit is set. If it is not set, the OS won’t know whether it actually has the write permission.

Files and directories can be owned only by a single user and a single group. Whenever a user creates a file or a directory, the file is “owned” by the user and the user’s default group.

For any file or directory, there are three types of “permission classes”. You can assign different permissions to these classes, and thus control who can access and modify files. The permission classes are as follows:

User: Permissions in this class affect the owner of the file.
Group: Permissions in this class affect the group which owns the file. However, if the owner user is in this group, the “user” permissions apply, instead of the group permissions.
Other: Permissions in this class affect all other users on the system.

Now that we have the theory in place, let us understand the concept with practical examples.

Viewing permissions

The easiest way to view the permissions of files in a given directory is to run:

ls -l <path to directory>

If you want to view the permissions in your current directory, leave out the directory name at the end:

ls -l

As an example, we have shown the contents of the /etc directory by using ls -l /etc. The output is in the form of columns, as shown below:

In this article, we will only focus on the “file mode”, “owner” and “group” columns. As you might have already guessed, the “owner” and “group” columns tell us which user and which group owns a file. The “file mode” column shows the file type and permissions in a compact way, as shown below:

The first character represents the type of file. A normal file is represented with a -, directories with a d, and symbolic links with l. For example, in the screenshot of ls, you can tell that acpi is a directory, whereas adduser.conf is a file.

Then, we have the permissions of the corresponding to the “user”, “group” and “other” classes. The triplet rwx represents whether these classes have the read, write and execute permissions. If a - appears in this triplet, it means that the corresponding permission is disabled.

As an example, consider the acpi directory, which has the file mode drwxr-xr-x. This means that the “root” user has read, write and execute permissions. Users in the “root” group, as well as any “other” users have read and execute permissions, but not write permissions.

Now, consider the at.deny file, which has the file mode -rw-r-----. Here, the “root” user has read and write permission; and users of the “daemon” group have read permission. However, all “other” users have no permissions at all.

Representing permissions: numeric and symbolic mode

In the previous section, we have seen how permissions are presented using the letters rwx. This is also known as the “symbolic mode”. However, there is an additional way in which we can represent permissions using numbers, called the “numeric mode”.

As an example, consider the acpi directory we’ve talked about previously. The symbolic mode of this directory is rwxr-xr-x. Now, in order to get the equivalent numeric mode, we take the individual permissions of “user”, “group” and “other”. If a permission is enabled we put a 1, and if it is disabled we put a 0. By doing this, we obtain a binary number, and we convert it to octal. So, for the acpi directory, you can get the symbolic mode as shown below:

Thus, we obtain the numeric mode as 755. Similarly, given a numeric mode like 644, you can convert it back — convert the numbers to binary, and put a letter like r, w or x if the permission is enabled, or a - if it is disabled:

Thus, we obtain the symbolic mode rw-r--r--, which means that the owner has read and write permissions; whereas the group users and other users have only read permission.

If you aren’t familiar with converting binary numbers, you can use the file permission chart below: Number**rwx triplet value****Description**0---No permissions1--xExecute2-w-Write3-wxWrite and execute4r--Read5r-xRead and execute6rw-Read and write7rwxRead, write and execute Fortunately, most tools show us and work with symbolic mode. We rarely need to use numeric modes, except for some special circumstances.

Changing permissions: the chmod command

When you create a file or directory, you might want to change some permissions. You can do this with the chmod command.

First, we will create a file for testing purposes by running:

touch file.txt

You can look at the file’s permissions with ls -l. The file was created with the permissions rw-r--r--.

Now, suppose you want to add the execute permission for all users — owner, group and others. Since we want to add execute permissions for all users, we use a+x like so:

chmod a+x file.txt

a represents that we are changing permissions for all users, the + represents that we are adding a permission, and x refers to the execute permission. As you can see below, these permissions were added:

By default, chmod assumes that any changes will apply for all users, so you can leave out the a and simply write:

chmod +x file.txt

chmod also accepts the letters u, g and o for “user”, “group” and “other”. It also supports the - symbol to remove permissions. For example, suppose you want to remove the write and execute permissions for “others”, and the write permission for “group”. To do so, run the following command:

chmod o-rx,g-w file.txt

Notice how we used a comma ( ,) to separate the two actions.

Sometimes, instead of adding or removing permissions, we might want to set a specific permission. chmod also supports this with the = symbol. As an example, if you want to set the user’s permission to rw- and the group’s permission to r--:

chmod u=rw,g=r file.txt

You can also use numeric modes to set permissions. However, numeric modes set permissions for “user”, “group” and “other” at the same time — there is no way to change permissions for each class individually.

Suppose, you want to set the permission of rwxrw-r-- for a file, using numeric mode. The equivalent numeric mode is 764, so you can run the following command:

chmod 764 file.txt

Finally, if you want to set permissions for all files in a directory, you can use the -R switch. For example, if you want to add the execute permission for all users, to all the files in the programs directory, run:

chmod -R +x programs

Keep in mind that you can only change permissions of files you own. To change the permissions of files that you don’t own, you must use sudo as shown:

sudo chmod <file permissions> <file/directory name>

Changing ownership: the chown command

In order to change the ownership of a file or directory, you can use the chown command. It can change both the owner and the group of the owner who owns it. (There is also the chgrp command which changes the group who owns a file, but chown can do it as well.)

Suppose, you have a file named file.txt and you would like to change the ownership to the root user. To do so, run the following command:

sudo chown root test.txt

Unlike the chmod command, you need to be the root user to change ownership information. In order to run the chown command as the root user, we have used sudo at the beginning. (If you are already using the root account, you can leave out the sudo.)

You can verify that the ownership changed using ls -l :

Now, suppose you want to change the owner group to daemon. In order to do so, mention the group name prefixed with a : like so:

sudo chown :daemon test.txt

If you want to change the owner user as well as the owner group, specify the user and group separated with a :. So, if you want to change the owner user to root and the owner group to daemon, run the following command:

sudo chown root:daemon test.txt

Sometimes, you may want to change the owner to a particular user and their default group. Although you could manually find out the user’s default group and then run the command, chown allows you to do this automatically by leaving out the : at the end. For example, if you want to change the ownership to root and their default group, run:

sudo chown root: test.txt

Just like chmod, chown also supports the -R flag. So, for example, if you want to change the owner user of all files to root for the directory test, run:

sudo chown -R root: test

Access rights flags

So far, we’ve learnt about permissions and ownerships, and the commands used to work with them. In addition, there are some “access rights flags”, which affect files and directories in special ways. In this section, we are going to learn about what these flags do and how we can set these flags.

The sticky bit

Sometimes, you may want to make a “common workspace” in which users can share and collaborate on their files. Unfortunately, a careless user (or perhaps a malicious one) can delete or rename files in this directory. We can’t take away the write permission, since that would not allow users to work with each other’s files.

This is where the sticky bit is useful. When you set a sticky bit on a directory, files in that directory can be only removed or renamed by the root user, the directory owner, or the file owner. The sticky bit has the final say on this — other users won’t be able to delete the file, even though the permissions say otherwise.

To set the sticky bit on a directory shared_directory, you can use the +t with chmod like so:

chmod +t shared_directory

Due to the sticky bit, the file mode gets a t at the end, as shown below:

You can also use the numeric mode 1xxx to set the sticky bit. The xxx here represents a general numeric mode like 762, 644 and so on, which we discussed previously in this article.

Suppose that shared_directory had a permission of 775. If we want to set the sticky bit using numeric mode, the command would look like:

chmod 1775 shared_directory

To remove the sticky bit, you can use -t or leave off the 1 at the beginning:

chmod -t shared_directory
chmod 775 shared_directory

A practical example of this is the /tmp directory, where programs (running under different users) can store temporary files. To prevent accidental deletion of other user’s files, Linux distributions set the sticky bit on /tmp.

The sticky bit does not have any effect on files.

setuid and setgid

If you’re somewhat familiar with Linux, you may have used su or sudo to run some commands as the root user. If you’ve wondered how they work, the answer lies in the setuid and setgid bits.

When the setuid bit is set on a file, it runs as the user that owns the file. Similarly, when the setgid bit is set on a file, it runs as the group which owns the file. In particular, the su and sudo programs use the setuid bit — you can look at its permissions by running:

ls -l /usr/bin/sudo

You will find that su/ sudo has the setuid bit set (notice the s in the file mode) and is owned by root. Thus, when you run su/ sudo it runs as root, and launches any programs that you want as root.

The setuid bit has no effect on directories. However, when you set the setgid bit on a directory, a few things happen:

When a user creates new files or subdirectories inside this directory, they inherit the owner group. This differs from the default behaviour, where newly created files are owned by the user’s default group.
Any new subdirectories also have the setgid bit set.

However, any files that were present before the setgid bit was set remain unaffected.

In order to set the setuid/setgid bit, you should first change the ownership of the file to an appropriate user (if needed). Now, to set the setuid bit using symbolic mode, run the command below. Since the file now belongs to a different user, we will use sudo:

sudo chmod +s <filename>

Similarly, if you want to set the setgid bit, use g+s instead, like so:

sudo chmod g+s <filename>

If you want to remove the setuid bit, use -s; for setgid, use g-s.

The numeric mode for setuid is 4xxx and setgid is 2xxx, where xxx is a numeric permission like 755, 642 and so on, as explained above. So, for example, if you want to set the setuid bit on a file and set the permission 755, run:

sudo chmod 4755 <filename>

To remove the setuid/setgid bits with numeric mode, simply leave the 4/ 2 at the beginning:

sudo chmod 775 shared_directory

A note on access rights flags and chmod

If you’re working with files with an access rights flag, there’s one caveat you must be careful of.

As an example, say you’ve used chmod to set the mode of a file to 4750 (setuid bit and permissions of rwxr-x---). Some time later, you want to allow others to have read access to the file. If you’re comfortable with numeric modes, you might be tempted to write:

chmod 752 <file/directory name>

However, this will clear the setuid bit! This is because, chmod assumes a leading 0 when you type a numeric mode that has less than 4 digits. This clears away the setuid bit. The correct command would be:

chmod 4752 <file/directory name>

This applies for the sticky bit and the setgid bits as well.

Also, if you want to set more than one access rights flag with one chmod command, simply add up the flags and run the command. Suppose, you want to create a directory with the setgid and sticky bits set, with permissions of 755. As we mentioned previously, setgid has a mode of 2xxx and sticky bit has a mode of 1xxx. By adding these two, we get 3xxx. Finally, since we want to use 755 as the permission, we get the numeric mode 3755. Thus, you should run the following command:

chmod 3755 <directory name>

Controlling default permissions: umask

Create a new file and a new directory using the touch and mkdir as shown below:

touch file.txt
mkdir dir

On most distributions, you will see that file.txt gets a permission of 664 ( rw-rw-r--), and dir gets a permission of 775 ( rwxrwxr-x ):

But how do these files get their default permissions? An octal number, called “file mode creation mask” controls these default permissions. (Henceforth, we will simply refer to this as “mask”.)

Shells like bash and sh have a built-in command called umask, which allows you to control this mask. Importantly, this only controls the behavior of programs launched from the shell. Any program which isn’t launched from the shell, such as a desktop environment or a web application daemon wouldn’t follow these settings.

Here, we will restrict ourselves to the umask command in shell. For other programs, refer to their help manual to find out how you can change their mask.

Files have a base permission of 666 (or rw-rw-rw-) and directories have a base permission of 777 (or rwxrwxrwx). Next, to get the actual permissions, we take the base permission and subtract it from the mask. By default, shells use a mask of 002. If you perform the calculation, you’ll end up with the default permissions we saw earlier:

To change the mask, simply run umask with the desired mask value:

umask <mask value>

As an example, suppose we don’t want “group” and “other” users to have any permissions. You can do so with the 077 umask:

umask 077

To verify that it works, try creating a file and view its permissions. Notice how the “other” and “group” users have no permissions:

You can also control the access rights flags with umask. For example, if you want add the setuid bit to new files, and use the mask 077, run the following command:

umask 1077

You can set other access rights flags in the same way.

If you want to always use this umask command whenever you use the bash shell, open the ~/.bashrc (on Ubuntu/Debian) or ~/.bash_profile (on other distributions) with your favorite text editor, and add the umask command at the end of the file.

Conclusion

Linux is a multi-user operating system. Thus, we need the concepts of file ownership and permissions to ensure the security of files, as well as the OS.

When working with permissions of system files, you should be very careful as wrong permissions can prevent programs from working correctly.

How to Install and Use Glances to Monitor Linux Systems

Thu, 29 Mar 2018 15:15:59 +0000

Monitoring a system is essential for keeping things running smoothly. Glances is a Python-based tool which provides information on various aspects of the system, such as running processes, I/O, memory and so on. It can display the information on a terminal and also has a web interface which can be accessed from any device. Moreover, you can use it to monitor a remote system through its API or its various interfaces. In this article, we will see how to install and use Glances to monitor local as well as remote systems.

Installation

Installing Glances is the easiest part, thanks to their install script which makes things really simple. On most systems, you can run:

wget -O- https://bit.ly/glances | bash

If you don’t have wget installed, you can use curl instead:

curl -L https://bit.ly/glances | bash

Since Glances is a Python-based tool, you can also use Python’s own package manager.

pip install glances

If you are running as a non-root user, the script will automatically ask for your root or sudo password, depending on how your system is set up.

Using Glances

To view the details of your own system, just run the following command:

glances

It should display information about your system, similar to the image shown below. Keep in mind that Glances may show you more information, depending on the size of the terminal and your system configuration.

At the top left of the interface, you can see the private and public address. The top right has the uptime, which is the time for which the system has been running. This is followed by the CPU and memory statistics. The left side has the network speeds, disk speeds and usage, and any information from sensors, like battery and temperature. The right side has a list of processes, and shows details like the CPU, memory usage and process IDs.

If you are interested to know more about the statistics displayed, you can read about them in the top command article.

Glances uses colors to tell you about the status of various items. Items in green represent they’re “safe”, and those in blue tell you to be “careful” — these may cause problems in the future. Anything in violet is a “warning”, and red means the status of the item is “critical”, and you should take immediate action.

Moreover, Glances also lets you know about anything that needs your immediate attention. In the screenshot above, you can see that Glances tells us so by displaying “No warning or critical alert detected”. When there are warnings or critical items, this message changes accordingly. For example, if you have high swap usage, this changes to “High swap (paging) usage”.

To quit Glances, press the ‘q’ or ‘ESC’ key on the keyboard.

Interactive Commands

There are a number of single-letter commands that you can use to enable various functions in Glances. To see an overview of the most useful commands, type h to show the help screen. The following image shows the help screen with the list of useful commands.

To exit the help screen press h again.

Glances has the ability to sort processes automatically on their CPU, memory or I/O usage. You can toggle this option on or off by pressing lowercase a. This option is turned on by default and will sort the processes automatically based on the resource usage. To sort processes by CPU, press lowercase c. Similarly, you can use m to sort by memory, i to sort by I/O, u to sort by user, p to sort by process name, and t to sort by time. Feel free to try out the other commands shown on the help screen.

Searching/Filtering Processes

To find a particular process, just press the Enter key. A search box will appear as shown below:

Here, you can type in a process name, or a regular expression to search for. You can also search based on other parameters, such as the username or command line arguments. For example, to search for processes belonging to the root user, you can type in username:root.

After you have typed in the search terms, press enter and it should show you a list of processes . To see more details of the filtered process, press e. It will show details of the process, as shown below:

To exit the filtered process list press capital E. It should return to the default Glances page.

Client/Server Mode

If you want to monitor a system remotely, you should install Glances on the local as well as the remote system. Once you have done so, run this command on the remote system:

glances -s

This spins up a server on the remote system. Next, you can monitor the remote system from your local system by running:

glances -c <server-ip>

Replace <server-ip> with the IP address or the hostname of the server.

In addition, Glances provides a “web server mode” that doesn’t require you to install it on your local system. First, launch Glances in this mode on the remote system like so:

glances -w

Now, open a browser and type http://<server-ip>:61208 where <server-ip> is the public IP address/hostname of the server. You should be able to see the stats displayed in a similar format as shown in the terminal. You can even access the page with a mobile browser since the page displayed is responsive.

To change the refresh rate of the page to a faster or slower value than the default rate, just append the date at the end of the URL. For example, if you want the values to refresh after every 5 seconds, then the URL will be http://<server-ip>:61208/5.

Configuring Glances

Glances allows you to configure its behavior through a glances.conf configuration file. For Linux systems, you can save this file either in ~/.config/glances or the /etc/glances directory. If either of these directories are not present in your system, you must create them manually.

The coloring scheme of glances is explained above. Now, if you want to trigger the warnings at some custom values rather than the default values, each section must be stated explicitly in the glances.conf file. For example, if you want to specify some custom alert values for CPU usage, then you can specify the values under the [cpu] section as follows.

[cpu]
user_careful=50
user_warning=70
user_critical=90
iowait_careful=50
iowait_warning=70
iowait_critical=90
system_careful=50
system_warning=70
system_critical=90
steal_careful=50
steal_warning=70
steal_critical=90

You can change the values according to your preferences. Similarly, you can customize other sections such as memory, I/O, network in the configuration file. To see the various sections that you can use, Glances provides a sample template configuration file that you can see here.

Exporting data to Prometheus

Glances has the ability to export data to Prometheus server with the help of an exporter. Prometheus is a great monitoring tool used to pull metrics from various sources and store them in a time-series database. To install and set up Prometheus, follow our article How to Install and use Prometheus for Monitoring. Once you have set it up, create a section named [prometheus] in the glances.ini file. Add the following details in that section.

[prometheus]
host=localhost
port=9091
prefix=glances

Instead of using “localhost”, you can use your IP address or hostname of the machine where Glances is installed. Similarly, you can use any port that you want provided that it is a non-standard one. After defining the values in the configuration file, use the following command to run Glances:

glances --export-prometheus

If you want to check if everything works, use your favorite browser and head over to http://localhost:9091 (or the IP address/hostname that you specified in the configuration file). Here, you will see that Glances makes its data available in a Prometheus-compatible format.

Now, in order to set up Prometheus monitoring, you need to add it to the prometheus.yml file.

scrape_configs:
 - job_name: 'glances_exporter'
 scrape_interval: 5s
 static_configs:
 - targets: ['localhost:9091']

You can set your own custom values in the exporter. After adding the exporter, save the file and start the Prometheus server. You will be able to see the Glances exporter in the service discovery page.

Conclusion

Glances is a great monitoring tool which has more features and a better interface than other tools like htop. However, it consumes way more resources than htop or top and can be a burden for low-powered systems. There are many more options and integrations mentioned in the official documentation that you can use to further customise Glances to your needs.

A Guide to Monitoring Servers with Nagios

Thu, 15 Mar 2018 15:15:31 +0000

Monitoring is an essential part of managing infrastructure. It helps you to ensure that your business keeps running smoothly, by alerting you to problems as they occur. Nagios is a very popular open-source application monitoring application. It features a “plugin”-based approach to monitoring, which makes it very flexible and well-suited to perform any kind of monitoring.

Nagios is available in two variants — “Nagios Core” is the free and open-source product, and “Nagios XI” provides additional functionality, but requires a software license. In this article, we will see how to install Nagios Core on a Linux server and use it to monitor the statistics of another server. Since we will solely focus on Nagios Core, we will simply call it “Nagios” in the rest of the article.

What is Nagios?

Nagios offers monitoring and alerting services for applications and infrastructure. It watches hosts and services that you specify, and alerts you when problems arise and again when things get better.

A Nagios monitoring setup consists of a number of components, as shown below.

The Nagios daemon relies upon plugins for its monitoring and alerting functionality. These plugins, along with your configuration, determine which hosts and services will be checked. Usually, these are “active checks”, which means that daemon invokes the plugin at regular intervals, and gives it information about what needs to be checked. Then, the plugin performs the check, and reports the status of the hosts and services back to the daemon. A check could involve a “ping” to see if it is up, measure a host’s disk space, and so on.

On the other hand, there are “passive checks”, which are initiated by external applications. Instead of the Nagios daemon doing a check, the external application submits data to the daemon.

In addition to reporting the status, a plugin can also add some performance information. Although Nagios itself cannot understand this data, you can ask it to store this data to a file or process it by executing commands. By using a plugin like N2RRD, you can save this data in a “round robin database”, for later querying and graphing.

Nagios provides a web interface to help you see this status information. You can also configure alerts via email. Moreover, there are plugins that can deliver alerts through SMS or other media.

Setting up Nagios

In this section, we will install Nagios on a Linux installation, and then configure it to monitor host resources. We will also set up the Nagios Remote Plugin Executor (NRPE) on another host, so that we can monitor it from the Nagios server. The instructions below have been tested on Ubuntu 16.04 and CentOS 7.

To follow the rest of this guide, we assume you will execute all the following commands as root. Depending on your distribution and setup, you can run sudo -s (and type in your password) or su (and type in the root user’s password) to get access to a root shell.

Prerequisites

In order to begin with the installation procedure, we will first install a few dependencies. These include the Apache web server and PHP for the web interface, as well as tools like gcc and make for compiling programs. Since Nagios uses CGI scripts, you cannot use a server like Nginx.

On Ubuntu, run the following command:

apt update
apt install build-essential libgd2-dev openssl libssl-dev apache2 php libapache2-mod-php snmp libnet-snmp-perl gettext libmcrypt-dev libssl-dev

On CentOS, run the following command instead:

yum install -y gettext wget net-snmp-utils openssl-devel glibc-common unzip perl epel-release gcc php gd automake autoconf httpd make glibc gd-devel net-snmp
yum install perl-Net-SNMP

Then, we will create a user for Nagios to use. Run the following command in order to do so:

useradd nagios

Finally, we will add this user to Apache’s user account. Depending on the distribution you are using, type in:

usermod -a -G nagios www-data # For Ubuntu
usermod -a -G nagios apache # For CentOS

Now that we have the basic setup in place, we can install Nagios.

Installing Nagios Core

Head over to the releases page in order to grab the latest release of Nagios Core, and download it like so:

wget https://github.com/NagiosEnterprises/nagioscore/releases/download/nagios-4.3.4/nagios-4.3.4.tar.gz

Then, we will extract the archive and move into the directory:

tar -xf nagios-4.3.4.tar.gz
cd nagios-4.3.4

The next step is to “configure” the project Makefiles to your system. Depending on your distribution, the commands are slightly different:

./configure --with-httpd-conf=/etc/apache2/sites-enabled # For Ubuntu
./configure # For CentOS

Then, we will compile and install the Nagios components, by running:

make all
make install
make install-init
make install-commandmode
make install-config
make install-webconf

Finally, we will configure Nagios to run automatically on boot. However, we won’t start it just yet, as we have not configured it.

systemctl enable nagios

Configuring Apache

Now, we will configure Apache. Again, depending upon the distribution you use, the steps here vary slightly.

If you use Ubuntu, you need to enable the “rewrite” and “cgi” modules, and then restart it.

a2enmod rewrite cgi

However, for CentOS, you simply need to start the Apache web server, and configure it to run automatically at boot:

systemctl enable httpd

Finally, we will protect the Nagios web interface by adding a username and password. In this article, we will use the username “nagiosadmin”, although you can choose something else as well. You will be prompted for a password.

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

The -c flag tells htpasswd to create a new file. Thus, if you want to add another user, leave out the -c flag, like so:

htpasswd /usr/local/nagios/etc/htpasswd.users booleanworld

Finally, we will restart the Apache web server by using:

systemctl restart apache2 # For Ubuntu
systemctl restart httpd # For CentOS

Installing Nagios plugins

The final step towards a basic Nagios installation is to install its plugins. First, we will navigate out of the Nagios Core directory with:

cd ..

Then, head over to the releases page and grab the latest archive like so:

wget https://github.com/nagios-plugins/nagios-plugins/releases/download/release-2.2.1/nagios-plugins-2.2.1.tar.gz

Next, extract the archive and move into the directory:

tar -xf nagios-plugins-2.2.1.tar.gz
cd nagios-plugins-2.2.1

Now, just like we had done previously, we will configure the “Makefiles” and install the plugins:

./configure
make install

Now that we have all the Nagios components installed, we can start the daemon. Type in the following to do so:

systemctl start nagios

Now, if you visit http://<Nagios server IP>/nagios in a browser, you will be asked for a username and password. Once you enter the details, the following page opens:

If you can see this page, congratulations! You’ve installed Nagios correctly.

Exploring the web interface

The web interface allows you to take a quick look at the status of monitored hosts and services. A “service” in this context refers to some observable aspect of a system, like its disk space or memory consumption. By default, Nagios only monitors the host it is installed on. In order to get a basic overview, you can click on “Host Groups” on the left.

In the above example, you can see that “localhost” (which refers to the system where Nagios is installed) has a host status “UP”. Seven services are marked as “OK” and one is marked “critical”. On your system, you might find a different scenario than the one above.

If you want to see a detailed report about the hosts, click on “View Host Status Detail For All Host Groups”.

Similarly, you can click on “View Service Status Detail For All Host Groups” to view details about the services.

You can click on a particular entry to view more details about it. For example, here is the “Swap Usage” information page for the current node. You can also issue a number of service commands from the right, such as disabling the check.

Monitoring a remote host using NRPE

So far, we have set up Nagios to only monitor the host the host it is running on. Now, we will use NRPE to monitor the disk usage and load average of a remote host. There are several steps to this — installing the check_nrpe plugin on the Nagios server, and the NRPE daemon on the remote host. Finally, we need to edit some configuration files.

Installing the `check_nrpe` plugin

These steps should be run on the Nagios server. If you are inside the Nagios plugins directory, move out of it by running:

cd ..

Then, go to the NRPE releases page and download the latest version like so:

wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz

Just like we did previously, extract the archive and move into the directory:

tar -xf nrpe-3.2.1.tar.gz
cd nrpe-3.2.1

Next, configure the “Makefiles” as shown below.

./configure

Then, run these two commands to build and install the plugin:

make check_nrpe
make install-plugin

Next, open the /usr/local/nagios/etc/objects/commands.cfg file. Here, we will define a check_nrpe command that will be used later in this article. Go to the end of the file and add the following snippet:

define command{
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Installing the NRPE daemon on the remote host

Now, log in to the remote host. Once you have logged in, open a root shell with su or sudo -s, and then create a “nagios” user for NRPE to run:

useradd nagios

Then, if you are using Ubuntu, run the following commands:

apt update
apt install -y autoconf automake gcc libc6 libmcrypt-dev make libssl-dev

Otherwise, if you are running CentOS, use these commands:

yum install -y gcc glibc glibc-common openssl openssl-devel perl wget

Since NRPE needs the Nagios plugins to perform its checks, install it in the same way as we did previously. Then, download the NRPE source code, and run the steps in the “Installing the check_nrpe plugin” section, till “configuring the Makefiles”. Once you have completed, run the following commands:

make all
make install
make install-config
make install-init

Now, open the file /usr/local/nagios/etc/nrpe.cfg and find the line containing the allowed_hosts and server_address and change them like so:

allowed_hosts=127.0.0.1,::1,<Nagios server IP>
server_address=0.0.0.0

Then, we will set the NRPE daemon to run automatically on boot. We will also start the daemon to test it.

systemctl enable nrpe
systemctl start nrpe

Now, from the Nagios server, run the following command:

/usr/local/nagios/libexec/check_nrpe -H <Remote host IP>

If you see a version number, this means that NRPE is up and running, and Nagios can reach it. If you see an error instead, verify that your firewall allows traffic on TCP port 5666.

Configuring NRPE

While NRPE is up and running, we haven’t set up the monitoring yet. While there is a built in command to check load averages, we have to make our own Nagios command to monitor disk usage. In order to do so, we should first find out the disks that are attached to system. Run the following command to see a list of disks and their usage levels:

df -h

You will get an output that looks like this:

Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           396M   41M  356M  11% /run
/dev/sda1        49G  2.1G   47G   5% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           396M     0  396M   0% /run/user/0

Here, look for the disk that is mounted on the location “/”. In the example above, it is /dev/sda1. Let us add a command to monitor the disk space. Open /usr/local/nagios/etc/nrpe.cfg and add a line like this at the end:

command[check_sda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/sda1

Here, we have named our command check_sda1. You can give it any name, as long as you are consistent with your scheme. Then, we define the command. Here, we said that the check_disk plugin should give us a “warning” when the disk space is at 20% or lower. If the disk space falls to 10% or lower, it should mark the status as “critical”.

After you make these changes, restart the NRPE service with:

systemctl restart nrpe

Configuring the Nagios server

Now, we need to tell the Nagios to reach out to the remote host to collect statistics. First, let us enable the directory from which Nagios reads server configuration files. Open the /usr/local/nagios/etc/nagios.cfg file and find this line:

#cfg_dir=/usr/local/nagios/etc/servers

Remove the # in front to uncomment the line. Save and exit your text editor, and then create a .cfg file under /usr/local/nagios/etc/servers. We will name ours as /usr/local/nagios/etc/servers/testserver.cfg. Regardless of the name you choose, open it in a text editor and type in the following contents:

define host {
 use linux-server
 host_name <Remote server hostname>
 alias <An alias for your server, can contain spaces>
 address <Remote server IP>
 max_check_attempts 5
 check_period 24x7
 notification_interval 30
 notification_period 24x7
}

In this above snippet, we have used max_check_attempts to set number of times that Nagios will retry the host check command if it returns any state other than an OK state. Then, we have used check_period to control when the host should be checked. Here, we have set it to 24x7, which has its usual meaning. The notification_interval directive controls the time after which any notifications will be resent. Here, we have set it to 30 minutes. (So far, we haven’t configured notifications yet, but we will do so in the next section.) Finally, we use notification_period to tell Nagios when it should send out alerts.

Next, add in the following text to monitor the load average and disk usage:

define service {
 use generic-service
 host_name <Remote server hostname>
 service_description CPU load
 check_command check_nrpe!check_load
}

define service {
 use generic-service
 host_name <Remote server hostname>
 service_description /dev/sda1 free space
 check_command check_nrpe!check_sda1
}

Save the file and restart Nagios with:

systemctl restart nagios

Now, you can find details of the host in the Host/Status details.

Configuring alerts

At this point, we have a working Nagios configuration, and we can see any problems in the web interface. However, it is more useful if we get notifications instead. By default, Nagios assumes that your system has a /bin/mail command using which it can deliver emails. However, in this article, we will deliver our emails through SMTP, with a service like GMail or Yahoo.

First, if you want to use a service like GMail or Yahoo, ensure that access for “less secure apps” is enabled. This is required so that non-interactive applications can send out mail.

Then, open the /usr/local/nagios/etc/objects/contacts.cfg file. Inside this file, there will be a contact block with an email directive. Find the line and add the email address where you want alerts to be delivered.

define contact{
 ......
 email <The email address where alerts will be delivered>
}

Save this file and exit the editor. The next step is to download a command-line SMTP client — we will use this in our article.

If you use Ubuntu, you have to install a few dependencies first.

apt install php php-cli php-gd php-curl php-zip php-intl php-mbstring php-xml

However, if you are on CentOS, you should run these commands to upgrade your PHP version, and then the necessary dependencies:

yum install yum-utils http://rpms.remirepo.net/enterprise/remi-release-7.rpm
yum-config-manager --enable remi-php70
yum install php php-cli php-gd php-curl php-zip php-intl php-mbstring php-xml

Next, we will download and install “Composer”, which is a PHP package manager:

wget -O - https://getcomposer.org/installer | php

Now, download the SMTP client, and extract it:

wget https://github.com/boolean-world/smtp-cli/archive/master.zip
unzip -d /opt master.zip

Then, we will install a few things that the SMTP client needs. Run the following commands:

cd /opt/smtp-cli-master
composer install

In this directory, create a config.json file, with the sender’s details. This includes the sending SMTP server and the sender’s email address. For example, if you want to send emails from GMail, the configuration file would look like this:

{
 "host": "smtp.gmail.com",
 "username": "the-senders-email@gmail.com",
 "password": "the-password-of-the-account",
 "secure": "tls",
 "port": 587
}

Now, open the /usr/local/nagios/etc/objects/commands.cfg file. Here, find the block with the command_name value set to notify-host-by-email. This block contains commands that are used to send out mails when there is a problem with a host. Below the command_name line, there is a command line that looks like this:

command /usr/bin/printf "%b" ... | /bin/mail -s ...

Replace the part containing /bin/mail -s with /opt/smtp-cli-master/smtp-cli.php. Repeat this process for the block containing the notify-service-by-email.

Now we can test the setup by triggering a critical condition. For example, if the monitored host has 30 GB disk space available, you can fill it up to 28 GB by running the following command:

dd if=/dev/zero of=zero.bin bs=1M count=$((28*1024))

You should wait for a few minutes till Nagios performs the check and sends out an alert. Later, when you get a mail, you can simply remove the zero.bin file.

Conclusion

In this article, we installed Nagios and configured it to monitor another host. We have also seen a few examples of how to write your own commands and how to configure alerting. If you are interested to learn more, be sure check out the Nagios documentation. If you want to have a nicer web interface, have a look at Ninja.

RAID levels 0, 1, 4, 5, 6, 10 explained

Mon, 05 Mar 2018 14:28:26 +0000

As businesses grow, demand arises for more reliable infrastructure that can handle critical systems. An important component in stable and scalable infrastructure is proper memory management. In this article, we will look at RAID which is an abbreviation for R edundant A rray of I ndependent (or Inexpensive) D isks.

What is RAID?

RAID is a storage virtualization technology which is used to organise multiple drives into various arrangments to meet certain goals like redundancy, speed and capacity. RAID can be categorized into Software RAID and Hardware RAID. In software RAID, the memory architecture is managed by the operating system. In case of hardware RAID, there is a dedicated controller and processor present inside the disks that manage the memory. There are various raid levels as discussed below.

RAID 0

RAID 0 is based on data striping. A stream of data is divided into multiple segments or blocks and each of those blocks is stored on different disks. So, when the system wants to read that data, it can do so simultaneously from all the disks and join them together to reconstruct the entire data stream. The benefit of this is that the speed increases drastically for read and write operations. It is great for situations where performance is a priority over other aspects. Also, the total capacity of the entire volume is the sum of the capacities of the individual disks. The downside, as you may have already guessed it is that there is almost no redundancy. If one of the disks fails, the entire data becomes corrupt and worthless since it cannot be recreated anymore.

Advantages:

Performance boost for read and write operations
Space is not wasted as the entire volume of the individual disks are used up to store unique data

Disadvantages

There is no redundancy/duplication of data. If one of the disks fails, the entire data is lost.

RAID 1

RAID 1 uses the concept of data mirroring. Data is mirrored or cloned to an identical set of disks so that if one of the disks fails, the other one can be used. It also improves read performance since different blocks of data can be accessed from all the disks simultaneously. This can be explained in the diagram below. A multi-threaded process can access Block 1 from Disk 1 and Block 2 from Disk 2 at once thereby increasing the read speed just like RAID 0. But unlike RAID 0, write performance is reduced since all the drives must be updated whenever new data is written. Another disadvantage is that space is wasted to duplicate the data thereby increasing the cost to storage ratio.

Advantages

Data can be recovered in case of disk failure
Increased performance for read operation

Disadvantages

Slow write performance
Space is wasted by duplicating data which increases the cost per unit memory

RAID 4

RAID 4 stripes the data across multiple disks just like RAID 0. In addition to that, it also stores parity information of all the disks in a separate dedicated disk to achieve redundancy. In the diagram below, Disk 4 serves as the parity disk having parity blocks Ap, Bp, Cp and Dp. So, if one of the disks fails, the data can be reconstructed using the parity information of that disk. Space is more efficiently used here when compared to RAID 1 since parity information uses way less space than mirroring the disk. The write performance becomes slow because all the parity information is written on a single disk which is a bottleneck. This problem is solved in RAID 5 as we will see next.

Advantages

Efficient data redundancy in terms of cost per unit memory
Performance boost for read operations due to data stripping

Disadvantages

Write operation is slow
If the dedicated parity disk fails, data redundancy is lost

RAID 5

RAID 5 is very similar to RAID 4, but here the parity information is distributed over all the disks instead of storing them in a dedicated disk. This has two benefits — First, there is no more a bottleneck as the parity stress evens out by using all the disks to store parity information and second, there is no possibility of losing data redundancy since one disk does not store all the parity information.

Advantages

All the advantages of RAID 4 plus increased write speed and better data redundancy

Disadvantages

Can only handle up to a single disk failure

RAID 6

RAID 6 uses double parity blocks to achieve better data redundancy than RAID 5. This increases the fault tolerance for upto two drive failures in the array. Each disk has two parity blocks which are stored on different disks across the array. RAID 6 is a very practical infrastructure for maintaining high availability systems.

Advantages

Better data redundancy. Can handle upto 2 failed drives

Disadvantages

Large parity overhead

RAID 10 (RAID 1+0)

RAID 10 combines both RAID 1 and RAID 0 by layering them in opposite order. Sometimes, it is also called as “nested” or “hybrid” RAID. This is a “best of both worlds approach”, because it has the fast performance of RAID 0 and the redundancy of RAID 1. In this setup, multiple RAID 1 blocks are connected with each other to make it like RAID 0. It is used in cases where huge disk performance (greater than RAID 5 or 6) along with redundancy is required.

Advantages

Very fast performance
Redundancy and fault tolerance

Disadvantages

Cost per unit memory is high since data is mirrored

RAID Implementation and Support

Many operating systems have built-in support for RAID. In Linux, there is a tool called mdadm which can be used to manage and monitor RAID devices. The entire setup and operation of mdadm have been explained in detail at the kernel wiki page. In addition, file systems such like ZFS, GPFS, Btrfs and XFS provide built-in support for RAID. On the hardware front, systems like Intel Matrix RAID has a dedicated drive controller chip which contains firmware and drivers to implement the RAID architecture.

Conclusion

Understanding the RAID levels is very crucial for developing storage infrastructure that meets the needs of the organisation. RAID has the capability to protect against disk failures and provide fast performance. However, it does not provide any means to protect against data corruption or implement security capabilities.

How to Install and Use Prometheus for Monitoring

Wed, 14 Feb 2018 12:06:39 +0000

Monitoring is an important part of managing applications deployed on servers. They help you ensure applications run smoothly, as well as troubleshoot any problems that may arise. In this article, we will look at Prometheus, which is a powerful open-source monitoring tool. Originally developed at SoundCloud, it has seen significant adoption in recent times. It pulls in metrics from various sources and stores them in a time-series database. It offers a powerful query language and the ability to visualize data using third-party tools.

What is Prometheus?

The Prometheus ecosystem consists of a number of components working together, as shown below.

The Prometheus server pulls in metrics at regular intervals through HTTP endpoints that provide information about hosts and various applications. These endpoints may be provided by the monitored application itself, or through an “exporter”. The endpoint URL is usually /metrics, and provides data as text or protobuf.

Short-lived jobs may push their metrics to “Pushgateway” for later retrieval by the server. In addition, you can use Prometheus to monitor other instances of it, since it makes its own metrics available in the same way.

After the server has collected the metrics, it saves it in a time-series database. Then, you can use PromQL to query this data. Prometheus provides a web interface to run queries. Other applications can run queries through the HTTP API to retrieve and work with the data. For example, third-party tools like Grafana use this API to help you visualize the data.

If you want to receive alerts via Slack, Email, PagerDuty or another platform, you may set up alerts through “AlertManager”. It is quite powerful, and supports features like grouping similar kinds of alerts or temporarily muting some alerts.

Note that many of these components such as Pushgateway or Alertmanager are optional — you may or may not use them, as per your need.

Setting up Prometheus

In this section, we will download and install Prometheus on a Linux system. Then, we will set it up to monitor two types of metrics — metrics of our current system (such as load average and free memory) using node_exporter, and metrics of the Prometheus server itself. This is a highly atypical setup — usually you would monitor other Prometheus instances or other machines. However, this simple setup will help us understand the basics of Prometheus monitoring. It will also serve as a good starting point for you to set up more complex monitoring setups.

Installing node_exporter

First, we will download node_exporter.Head over to the downloads page and download the latest version, like so:

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz

Then, we will extract the downloaded archive. In our case, this results in a directory named node_exporter-0.15.2.linux-amd64.

tar -xf node_exporter-0.15.2.linux-amd64.tar.gz

Next, move the node_exporter binary to /usr/local/bin:

sudo mv node_exporter-0.15.2.linux-amd64/node_exporter /usr/local/bin

Finally, remove the residual files with:

rm -r node_exporter-0.15.2.linux-amd64*

Creating users and service files for node_exporter

For security reasons, it is always recommended to run any services/daemons in separate accounts of their own. Thus, we are going to create an user account for node_exporter. We have used the -r flag to indicate it is a system account, and set the default shell to /bin/false using -s to prevent logins.

sudo useradd -rs /bin/false node_exporter

Then, we will create a systemd unit file so that node_exporter can be started at boot like a regular system service. Create a file in /etc/systemd/system/node_exporter.service with the following contents:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Since we have created a new unit file, we must reload the systemd daemon:

sudo systemctl daemon-reload

Finally, we can set the service to always run at boot and start it:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Installing Prometheus

The next step is to download and install Prometheus. Again, head over to the downloads page and download Prometheus like so:

wget https://github.com/prometheus/prometheus/releases/download/v2.1.0/prometheus-2.1.0.linux-amd64.tar.gz

Just like how we installed node_exporter, we will extract the Prometheus archive and move the binaries to /usr/local/bin:

tar -xf prometheus-2.1.0.linux-amd64.tar.gz
sudo mv prometheus-2.1.0.linux-amd64/prometheus prometheus-2.1.0.linux-amd64/promtool /usr/local/bin

Now, we need to create directories for configuration files and other prometheus data. We will use /etc/prometheus and /var/lib/prometheus respectively.

sudo mkdir /etc/prometheus /var/lib/prometheus

Then, we can move the configuration files to the directory we made previously:

sudo mv prometheus-2.1.0.linux-amd64/console prometheus-2.1.0.linux-amd64/console_libraries /etc/prometheus

Finally, we can delete the leftover files as we do not need them any more:

rm -r prometheus-2.1.0-linux-amd64*

Configuring Prometheus

After having installed Prometheus, we have to configure Prometheus to let it know about the HTTP endpoints it should monitor. Prometheus uses the YAML format for its configuration, which forbids tabs and requires you to use spaces for indentation.

If you want to specify default scraping settings, you can do so in the global section. On the other hand, if you want to specify which jobs/exporters should be monitored and any settings specific to them, you should do it through the scrape_configs section.

We will use /etc/prometheus/prometheus.yml as our configuration file. Create this file with a text editor and enter the following contents:

global:
 scrape_interval: 10s

scrape_configs:
 - job_name: 'prometheus_metrics'
 scrape_interval: 5s
 static_configs:
 - targets: ['localhost:9090']
 - job_name: 'node_exporter_metrics'
 scrape_interval: 5s
 static_configs:
 - targets: ['localhost:9100']

In this file, we have defined a default scraping interval (with scrape_interval) of 10 seconds. We have also define two sources of metrics, named prometheus_metrics and node_exporter_metrics. For both of them, we have set the scraping interval to 5 seconds, overriding the default. Then, we have specified the locations where these metrics will be available. Since our targets are present on the same system, we use localhost as the hostname. Prometheus uses port 9090 and node_exporter uses port 9100 to provide their metrics.

Finally, we will also change the ownership of files that Prometheus will use:

sudo chown -R prometheus: /etc/prometheus /var/lib/prometheus

Creating users and service files for Prometheus

We will create an user for Prometheus, just like we had done for node_exporter:

sudo useradd -rs /bin/false prometheus

Then, we will create a systemd unit file in /etc/systemd/system/prometheus.service with the following contents:

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
 --config.file /etc/prometheus/prometheus.yml \
 --storage.tsdb.path /var/lib/prometheus/ \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

This file is similar to the one we previously created for node_exporter. The ExecStart directive in the file specifies a set of switches that define the paths that Prometheus should use for the configuration, storing the time-series database, and for rendering the web interface.

Next, we will reload systemd:

sudo systemctl daemon-reload

Finally, we will enable and start the service:

sudo systemctl enable prometheus
sudo systemctl start prometheus

Exploring Prometheus

Now that we have finished installing and Prometheus, it is time to explore some of its functionality. Prometheus provides a web UI for running basic queries located at http://<your_server_IP>:9090/. This is how it looks like in a web browser:

The “Console” tab is used to view the results of a query, while the “Graph” tab is used to create graphs based on a query. Later in this article, we will discuss how to write, run and graph queries.

If you want to see a list of metrics sources, go to the Status → Service Discovery page. Here, you will find a list of all services that are being monitored, including the path at which the metrics are available. In this case, the default path /metrics is used.

If you’re curious to see how the metrics page looks like, head over to one of them. The screenshot below shows the metrics page for node_exporter, which you can find at http://<your_server_IP>:9100/metrics.

You can return to the query interface by visiting the “Graph” link on the top of the interface.

Prometheus metrics and queries

As we previously discussed, the Prometheus server collects metrics and stores them in a time series database. Individual metrics are identified with names such as node_filesystem_avail. A metric may have a number of “labels” attached to it, to distinguish it from other similar sources of metrics. As an example, suppose node_filesystem_avail refers to the free memory available on the disks attached to a node. It may have a label such as mountpoint, which helps you inspect individual storage devices by mentioning the mountpoint. Thus, to get the available space on the device mounted as /mnt/sda1, you have to use node_filesystem_avail{mountpoint="/mnt/sda1"}.

In PromQL, an expression or subexpression should always evaluate to one of the following data types:

Instant vector — It represents a time-varying value at a specific point of time.
Range vector — it represents a time-varying value, over a period of time.
Scalar — A simple numeric floating point value.
String — A string value. String literals can be enclosed between single quotes, double quotes or backticks ( `). However, escape sequences like \n are only processed when double quotes are used.

Note that depending on the use case, only some of these data types are legal as a return value. For example, while constructing graphs (through the “Graph” tab in the web interface), only expressions that result in instant vectors can be used.

Now that we have the basic theory in place, let us see how we can query Prometheus to get information about monitored hosts and applications.

Prometheus query examples

In this section, we will look at a few examples of queries. Let us start with a simple problem — we have to find out the free memory available on a system. In our setup, node_exporter provides information about the system it is monitoring. This metric is available as node_memory_MemFree. Type this in the “Expression” field and hit Enter.

Note that the results are in bytes. In order to get a value in MBs, you can convert the value with: node_memory_MemFree/(1024*1024):

In addition, notice that the result has a set of labels like instance and job, which you can use to filter out certain instances or jobs. Since we are monitoring a single node, filtering is not very useful in in this case. For a better example, we will find out the free disk space on our system with node_filesystem_avail:

Notice how each storage device gets a label based on its mountpoint. Now, if you wanted to view the free space for the device at mountpoint /, you can type in node_filesystem_avail{mountpoint="/"}:

Apart from the = operator, the != (not equal to), =~ (regex match) and !~ (negated regex match) are also supported. Also, you can provide two or more labels in a query like: node_filesystem_avail{job=~".*metrics",mountpoint!="/"}.

Now, suppose you want to obtain a graph of the free memory available on the system. Enter node_memory_MemFree, hit Enter and click on the Graph tab. A graph will appear like the one shown below. You can also change the time range using the controls above the graph.

So far, all our queries yield the present value of a metric. You can obtain values in the past by using the offset keyword. For example, you could view the free memory value two days ago with node_memory_MemFree offset 2d:

You can use the letters s, m, h, d, w and y to refer to seconds, months, hours, days, weeks and years respectively.

Until now, all our examples returned instant vectors. Now, we shall look at how to work with range vectors. To obtain a range vector, simply add a time period in square brackets after an instant vector. For instance, if you want to get the free memory values over the last one minute, run node_memory_MemFree[1m]:

As you can see above, range vectors correspond to a set of values, along with the time at which the values were obtained. Range vectors cannot be graphed, which you can check by going to the “Graph” tab.

PromQL also provides many functions, which are helpful in analyzing metrics. As an illustration of PromQL’s capabilities, we will use the rate() function. Suppose, you are interested in the rate of context switches occuring on your system, for the last two hours. We obtain a range vector as discussed previously, and then obtain the rate as: rate(node_context_switches[2h]).

Conclusion

In this article, we have learnt how to install Prometheus and use it to monitor metrics of our current node. We have also seen a few examples of how to query and graph these metrics. If you want to learn more about Prometheus, the documentation is a good place to start.

How to Monitor Network Traffic on Linux Using vnStat

Thu, 28 Dec 2017 13:36:22 +0000

Developers and sysadmins may need to monitor various network parameters, such as bandwidth consumption and the traffic flowing in/out. There are many tools available for this purpose, one of them being vnStat. It is a simple CLI utility that shows network statistics over various interfaces and periods for a particular computer. In this article, we will see how to install and work with vnStat, and (optionally) view this data graphically using vnstat-dashboard.

Installation

For Ubuntu and Debian systems, vnStat is available in the default repository. Use the following commands in order to install it:

sudo apt update
sudo apt install vnstat

The default repositories for CentOS and Fedora don’t have vnStat, but you can get them from the EPEL repositories. Thus, you should enable the EPEL repositories, and then install vnStat. Use the following commands to do so:

sudo yum install epel-release
sudo yum install vnstat

For Fedora systems, type in the following:

sudo dnf update
sudo dnf install vnstat

Setting up vnStat

After installing vnStat, you have to set the network interface which you want to monitor. However, first you need to know the list of network interfaces available on our system before you can ask vnStat to monitor it. To do so, use any of the two commands:

ip link show

netstat -i

This will show a list of all network interfaces available in the system.

For this article, we will just monitor the interface representing Ethernet connection to our system, which is eth0 in our example. On your system, you may find the interface named differently, such as enp0s3 or venet0. If you use WiFi, there will be a interface such as wlan0 or wifi0. Any interfaces with names like lo or lo0 represent the loopback interface, which is used for local traffic, such as communication between daemons.

So, to select the eth0 interface for monitoring, use the command:

sudo vnstat -u -i eth0

It should tell you that a database has been created. Then, you should set proper permissions for the database with:

sudo chown -R vnstat: /var/lib/vnstat

Finally, enable and restart the daemon with:

sudo systemctl enable vnstat
sudo systemctl restart vnstat

You can verify that the daemon is running with top, or with systemctl status:

sudo systemctl status vnstat

Viewing bandwidth consumption

By default, vnStat writes fresh statistics to the database every 30 seconds. After waiting for 30 seconds, run vnstat to view the monthly and daily statistics in a tabular form. If your system doesn’t receive too much traffic, you can try downloading these files to generate some traffic before viewing the statistics.

The “rx” and “tx” columns show the received and transmitted traffic respectively, and the “total” column gives the sum of these two values. The “avg. rate” shows the average rate of bandwidth consumption per second, over the course of the day or month.

Sometimes, you may want to see the updated bandwidth usage immediately without waiting. In that case, you can force vnStat to update the database with:

sudo vnstat -u

Then, run vnstat as usual; you will get to see the updated statistics after you do so.

If you want to view statistics for a different timeframe, there are different switches you can use:

-d: Daily statistics for the last 30 days.
-m: Monthly statistics for the past 12 months.
-w: Statistics for the last 7 days, and the current and previous week.
-h: Hourly statistics for the last 24 hours.
-t: Top 10 days with the highest traffic.

You can use these switches like so:

vnstat -h

The hourly statistics also feature a graph, along with the “rx”/“tx” values, as shown below. The other switches show a tabular output, like the one in the previous example.

In addition, vnStat also features a live traffic monitoring mode. To use this mode, use the -l switch and give the name of the interface with -i like so:

vnstat -l -i eth0

Once you’re done, press Ctrl+C to exit.

Exporting and importing data

So far, we have seen how to retrieve bandwidth usage data from vnStat. However, the tables are not very useful if you want to perform your own calculations. You can export the data in JSON and XML by using the --json and --xml switches respectively. For example, if you want to create a file named output.json with all bandwidth usage data, run:

vnstat --json > output.json

If you want to import the database on to another system, you should first export the database with the --exportdb switch. The following command writes the database to a plain-text file named vnstat_db.

vnstat --exportdb > vnstat_db

Now, you can transfer this file on the other system, and use the --importdb switch to import. For example, if you want to import the vnstat_db file for the eth0 interface, run:

sudo vnstat --importdb vnstat_db -i eth0 --force

View vnStat data over the web

If you want to monitor network statistics over the web, there are various tools that work alongside vnStat. In this article, we will look at vnstat-dashboard, a web application that helps you visualize vnStat data. In order to use it, you need to install a web server, PHP and a few other tools. We will use Apache simply as an example; but you are free to use another web server.

On an Ubuntu/Debian system, run these commands to install the necessary tools:

sudo apt install apache2 php wget unzip

If you are on a CentOS/RHEL system, run these commands instead:

sudo yum install httpd php wget unzip

(For a Fedora system, simply replace yum with dnf.)

Then, download the source code from Github, and unzip in the proper directory:

wget https://github.com/alexandermarston/vnstat-dashboard/archive/master.zip
unzip master.zip
sudo mv vnstat-dashboard-master/ /var/www/html/vnstat/

Next, we need to change a few configuration parameters. To do so, open the configuration file in a text editor like vi or nano:

sudo nano /var/www/html/vnstat/config.php

Find the following lines in this file and replace the default values with the ones shown in red:

$vnstat_bin_dir = '/usr/bin/vnstat';
$use_predefined_interfaces = true;
if ($use_predefined_interfaces == true) {
 $interface_list = array("eth0");
 // ...
}

After you’ve made these changes, open http://<your_server_IP>/vnstat in a browser. You should see a page like this open up:

You can easily toggle between the hourly, daily and monthly tabs and also switch between various network interfaces.

Conclusion

vnStat is a very useful tool to monitor and analyze network information of a system. There are a variety of options available to view and log the data of network interfaces which can be used for several use cases.

How to Set up Basic HTTP Authentication on nginx

Tue, 17 Oct 2017 15:17:43 +0000

Files placed on a web server are publicly accessible by default. If you want to protect files from unauthorized access, you can make use of HTTP basic authentication. It is a simple authentication method, where users need to provide a username and password to access files on your server.

In this article, we will learn at how to use the basic authentication feature built into nginx. Additionally, you can combine this with other restrictions, such as allowing access from certain IP addresses.

Prerequisites

You should obviously have a copy of nginx on your system before you can configure it. In addition, you should install Apache’s htpasswd utility to generate hashed passwords. On Debian/Ubuntu systems, you can install it with:

sudo apt install apache2-utils

On CentOS, you can install it like so:

sudo yum install httpd-tools

If you’re using Fedora, simply substitute dnf for yum:

sudo yum install httpd-tools

Creating a htpasswd file

A htpasswd file contains a list of usernames and passwords. nginx uses it to check credentials of users who want to access protected areas of the website.

First, create an empty file named /etc/nginx/htpasswd like so:

sudo touch /etc/nginx/htpasswd

Now, you can add usernames and passwords to this file with the htpasswd command. For example, if you want to add an user bob to the file, run:

sudo htpasswd /etc/nginx/htpasswd bob

You will be prompted to enter the password twice. When you type the password in, there will be no feedback.

Later on, if you want to change an existing password, simply run the command again. Similarly, you can add other users to the file by running the above command with a different username.

The htpasswd file is a simple text file, and you can inspect it with a text editor. The usernames are stored as-is, but the passwords are hashed and salted. Here is a file which has two users alice and bob:

alice:$apr1$Q9qBLkLy$YQNxI8/GYuHLmoOUkbjN70
bob:$apr1$5rz0Qz20$Xuo4c3yrZRjkAk89il7Us/

If you want to remove an user from the file, use the -D switch like so:

sudo htpasswd -D /etc/nginx/htpasswd john

In the next section, we will configure nginx to use this file to restrict access.

Configuring nginx

Now, open up the site-specific configuration file with an editor like nano. If you are using Debian or Ubuntu, the site specific configuration is present in /etc/nginx/sites-available/default:

sudo nano /etc/nginx/sites-available/default

However, if you use another Linux distribution, it will be in /etc/nginx/nginx.conf:

sudo nano /etc/nginx/nginx.conf

In either case, go ahead and locate the server block. Once you have done so, add a location block for the path you would like to protect. Inside this block, you should set the realm name and the htpasswd file path. In the example below, we have restricted access for everything under files, and the “realm name” is set to Restricted. The realm name is displayed by browsers, and it is used for differentiating between different protected parts of the website.

server {
 # Many lines of configuration...

 location /files {
 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/htpasswd;
 }
}

Save the file and quit the editor. Then, to bring these changes into effect, you must reload the nginx configuration with:

sudo systemctl reload nginx

If you are on an older, non-systemd distribution, you should run this instead:

sudo service nginx reload

Now, if you use a web browser to access anything under /files, your browser will prompt for an username and password:

If you want to protect another path, simply add another location block with the same content:

location /photos {
 # ...
}

Keep in mind that you can also use other directives inside these location blocks.

Site-wide restriction

In the previous example, we restricted access to certain directories on the server. What if you want to place a site-wide restriction? You can do so by moving the basic authentication directives to the server block:

server {
 # Many lines of configuration...

 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/htpasswd;
}

You can also leave out certain paths by setting auth_basic to off in a location block. For example, the following block allows users to access

location /files {
 auth_basic off;
}

Combining basic authentication and other restrictions

So far, we have seen simple examples of basic authentication. nginx supports a wide variety of restrictions, and it is possible to combine them in various ways. In this section, we will consider an interesting case: combining IP restrictions with basic authentication.

Say, you have a section on your website meant for use by company staff. Any employees accessing the web server from within the network can access a page without using a password. However, if they try to access it over the Internet, they should use a password.

This means we need to have two restrictions: an IP restriction and a basic authentication directive. We have already seen how to specify basic authentication directives. IPs can be allowed or blocked by using the allow and deny directives. In addition, you can tell nginx to satisfy one of these conditions using satisfy any. Thus, the location block might look something like this:

location /files {
 satisfy any;

 allow 10.2.0.0/16;
 allow 10.3.155.26;
 deny all;

 auth_basic "Restricted";
 auth_basic_user_file /etc/nginx/htpasswd;
}

In the above example, any user with an IP of 10.2.0.0-10.2.255.255 and 10.3.155.26 are allowed without a password. For other users, we fall back to basic authentication.

Now, consider a different situation. Perhaps you want employees to access these files only from within the company network, with a password. Employees shouldn’t access these files when outside the company network. This means both conditions need to be satisfied. In order to make this change, change any to all:

location /files {
 satisfy all;

 # Other directives...
}

Conclusion

In this article, we have learnt about HTTP basic authentication and how to use this feature in nginx. It provides a very easy way to restrict people from accessing files on your web server, without having to install anything.

Building Cross-Distribution Linux Applications with Flatpak

Mon, 02 Oct 2017 12:14:26 +0000

Fragmentation has been a problem for quite some time in the Linux ecosystem. A variety of distributions exist today, with their own package managers and their own notions of what a “base system” consists of. This is tedious for application developers — after building distribution specific packages and maintaining repositories, they have to fix any distribution-specific bugs that may arise. Also, unlike mobile platforms such as iOS and Android, Linux applications run unsandboxed with access to all functionality on the base system.

To tackle these problems, a wide variety of solutions have been proposed. In this article, we will take a look at Flatpak — a way to distribute applications and run them in a restricted sandbox.

What is Flatpak?

Flatpak is a technology for building, distributing, installing and running applications, primarily aimed at the Linux desktop. It allows applications to bundle dependencies and ship with runtimes, so that applications don’t end up depending upon the quirks of a particular distribution. In addition, it increases the security of Linux applications by isolating them into sandboxes.

The architecture of a Flatpak application is as shown in the diagram below:

Every Flatpak application is backed by a “runtime”, which contains the basic dependencies used by applications. These runtimes are shared among applications. Multiple runtimes, and even different versions of the same runtime can coexist on a system at the same time. In this way, applications have a fairly consistent base system to rely on.

In order to build applications for this runtime, special runtimes called SDKs (Software Development Kits) are provided. They contain tools such as compilers and development headers. If your application depends on libraries not bundled in the runtime, you must compile them with the SDK, along with your application.

In addition, there are “extensions”, which are optional add-ons for runtimes and applications. They are most often used to separate out translation and debugging information from the original runtime or application.

One of Flatpak’s aims is to increase the security of applications by isolating them from one another. By default, Flatpak provides very limited access to the host system’s resources. Unless explicitly requested, applications are not permitted to use the network, access files on the base system, or talk to services such as X, DBus or Pulseaudio. In addition, “portals” allow selective access to resources through high-level APIs.

Installation, Flatpak commands and naming conventions

Flatpak is available in the repositories of most distributions. On Arch Linux and Fedora, you have to install the flatpak package to install the flatpak command, which helps you to install and build applications.

sudo dnf install flatpak # Fedora
sudo pacman -S flatpak # Arch

Ubuntu users don’t have it in the default repositories, so installing it is a bit more involved. First, you have to add this PPA to your system:

sudo add-apt-repository ppa:alexlarsson/flatpak

Then, you can install the package with:

sudo apt update
sudo apt install flatpak

While the flatpak command can certainly help you to build applications, it isn’t the most convenient way. There is another tool, named flatpak-builder, which allows you to build Flatpak applications from a JSON file. (We will discuss this tool later in this article.) You can install this tool with:

sudo dnf install flatpak-builder # Fedora
sudo pacman -S flatpak-builder # Arch
sudo apt install flatpak-builder # Ubuntu

Now, in order to install applications or runtimes, you have to add repositories to Flatpak’s list of remotes. These remotes can be hosted in a local directory, or online. Runtimes/applications are identified using an inverse DNS address, like com.booleanworld.foo. In some circumstances, you may also need to specify the architecture and the “branch” of the runtime. This is done with help of a slash-separated triplet, like com.booleanworld.foo/x86_64/1.3.

The flatpak command usually adds software and repositories for all users. However, this is unsuitable for trying things out, so we will use the --user flag to restrict such changes to the current user only.

Now that we have the basics in place, we will take a look at some theory on building Flatpak applications.

Flatpak application structure

When you bundle an application with Flatpak, the following directory structure is automatically created:

metadata: This file contains information about the SDK and runtime the application will run on, as well as a list base system resources that the application needs to access.
files: This directory contains files that make up the application, including any application data. The bin subdirectory within here contains the application executables.
export: This directory contains any files that the base system needs to access. Examples include AppStream data, the .desktop file and the application icon. Any files you put here should be prefixed with the identifier. For example, a desktop file may have the name com.booleanworld.foo.desktop.

While you could build this structure by hand, this is not necessary. Flatpak has tools to help you automatically create this structure, and it requires very little configuration if you have a typical autotools-based project. With non-autotools based projects, you usually need to set the “prefix” directory to /app to get things up and running.

Prerequisites

The following sections of the article has a few examples of building Flatpak applications. As we have mentioned previously, applications need a runtime to run and a SDK for the build process. Thus, the first step is to add a repository from which they are available. We will add the repository available on sdk.gnome.org for our use.

flatpak remote-add --from gnome https://sdk.gnome.org/gnome.flatpakrepo --user

In order to build the examples below, we need two runtimes: org.freedesktop.Platform and org.gnome.Platform, and two SDKs: org.freedesktop.Sdk and org.gnome.Sdk. Here, we have used versions 1.6 and 3.24 of the Freedesktop and Gnome runtime/SDK.

flatpak install --user gnome org.freedesktop.Platform/x86_64/1.6 \
 org.freedesktop.Sdk/x86_64/1.6 \
 org.gnome.Plaform/x86_64/3.24 \
 org.gnome.Sdk/x86_64/3.24

Next, check out the flatpak-resources repository, which contains all the resources needed to follow through this article.

git clone https://github.com/boolean-world/flatpak-resources
cd flatpak-resources

We will take a look at two examples of building applications in the following sections.

Building “kitten”, a lightweight “cat”

In this section, we will learn how to build a basic Flatpak application. The application is a tiny version of the cat command, named kitten.

First, you have to create the directory structure we have discussed earlier. The build-init command can do this for you. You need to give it a build directory, the name of your application, and the SDK and runtime to use (in that order). You can also add an optional “branch name”, but we will leave it out in this example.

flatpak build-init kitten-build com.booleanworld.kitten \
 org.freedesktop.Sdk \
 org.freedesktop.Platform

As we mentioned previously, the prefix directory of Flatpak applications is /app, and you need to create the bin directory in it. When you want to run a command within the build directory, use the build command:

flatpak build kitten-build mkdir -p /app/bin

The next step is to compile kitten.c with gcc:

flatpak build kitten-build gcc kitten.c -o /app/bin/kitten

Then, you can complete the build with build-finish. The command will be kitten, and it needs to access the host’s filesystem (so that you can view any file on the host system). So, you can complete the build with those parameters set:

flatpak build-finish kitten-build --filesystem=host --command=kitten

Now, if you want to test the application, you have to install it. The first step is to export the application into a local repository. In our example, the repository folder is my-repo:

flatpak build-export my-repo kitten-build

Next, we will add the repository with the name test-repo. Since we are not using GPG signatures, we will let Flatpak know with the --no-gpg-verify switch.

flatpak remote-add --user --no-gpg-verify test-repo my-repo

Finally, you can install it with:

flatpak install --user test-repo com.booleanworld.kitten

Now, you can test if it works by running it like so:

flatpak run com.booleanworld.kitten ~/.bashrc

If you’ve correctly followed all the steps so far, this should display the contents of your .bashrc file:

Congratulations on building your first Flatpak app!

The steps have been fairly simple so far, but the build-finish command warrants further discussion. This is what we are going to do in the next section.

A deeper look into the “build-finish” command

One of Flatpak’s primary goals is to sandbox applications and provide only minimal access to the host system’s resources. In the previous example, we allowed the “kitten” app to access any file on the system with build-finish. The switches you can use to allow access to various resources are listed below:

--filesystem={path}

This switch allows access to specific paths on the system. You can provide a specific path (such as /usr/share). home allows access to the user’s home directory, and host allows access to all files on the system. In addition, you can also make a path read-only with :ro. For example, --filesystem=/usr/share:ro --filesystem=home allows read-only access to /usr/share and read-write access to the user’s home directory.

--share={type}

This switch allows access to particular resources. The most commonly used types are ipc and network, which allow IPC and network access. In most GUI applications, --share=ipc is used for X shared memory to work, which improves performance.

--socket={type}

This switch allows you to use particular sockets. The most commonly accessed socket types are x11, wayland and pulseaudio. The first two are used by applications to render GUI elements, and the latter is used to play audio through the Pulseaudio daemon.

--device={device}

This allows applications to selectively access some devices. Most often, dri is used here so that applications can use OpenGL rendering.

--talk-name={name} and --system-talk-name={name}

These switches allow applications to talk to named services through session bus and system bus of DBus.

--env={varname}={value}

The env switch allows you to export environment variables. {varname} is the variable name, which is set to the given {value}.

If an application requires access to dconf, you must use the following switches:

--filesystem=xdg-run/dconf
--filesystem=~/.config/dconf:ro
--talk-name=ca.desrt.dconf
--env=DCONF_USER_CONFIG_DIR=.config/dconf

We will see a practical example involving these switches in a later section.

Building applications automatically: flatpak-builder

Although the steps involved in building Flatpak applications are fairly simple, it is still tedious to build them step-by-step. flatpak-builder is a tool that allows you to build applications declaratively with a JSON manifests. As long as the source code is laid out according to the “build API”, the builder can compile and install applications from the source archives. Most autotools based software already conform to the API, so no further configuration is usually needed.

A typical manifest file usually looks like this:

{
 "app-id": "com.booleanworld.kitten",
 "runtime": "org.freedesktop.Platform",
 "runtime-version": "1.6",
 "sdk": "org.freedesktop.Sdk",
 "command": "kitten",
 "finish-args": [
 "--filesystem=host"
 ],
 "modules": [
 {
 "name": "kitten",
 "sources": [
 {
 "type": "archive",
 "url": "https://opensource.example.com/kitten-0.0.1.tar.gz",
 "sha256": "38ecfd39b76a2505376f5a57ebcc7b19d35efaf9f8c724a24382c45aa1378018"
 }
 ]
 }
 ]
}

As you can see, this file contains the same information we had specified earlier with various commands. In addition, it contains a list of modules to build. In the example above, there is only a single module. If you have a complex application that depends on multiple modules, you would have to list all of them. For example, gnome-calculator depends on mpc and mpfr, and the manifest looks like this.

Different source types are supported. Another popular source type is git, and you have to provide the repository URL and optionally, the branch which must be checked out.

The manifest is very powerful, and it supports setting CFLAGS/CXXFLAGS, renaming files for export, and removing unnecessary files. In the next section, we will analyze a practical application, and learn how to package it for Flatpak.

A real example: packaging galculator

In this section, we will take galculator, a simple calculator application and build it with flatpak-builder. We need to perform a bit of analysis to find out which files need to be renamed or deleted. Thus, as a first step, we will make a test installation. Then, we can build the manifest as necessary.

Since galculator is a GTK application, we will use the Gnome runtime and SDK. You should install them now if you haven’t already done so. In addition, you should download the source archive from here. (We have used version 2.1.4, which is the latest version at the time of this writing.)

Finding files that should be exported/deleted

Once you have downloaded and extracted the source archive, you should try to build it using the SDK. The Gnome SDK also contains tools like bash, so you can start that up by using:

flatpak run --command=bash --filesystem=host --devel org.gnome.Sdk/x86_64/3.24

Now, move into the galculator source directory and use the regular incantation to build it. We will set our prefix directory to ~/galculator-test.

./configure --prefix=$HOME/galculator-test
make install

Now, you can explore the files in the galculator-test directory. As you can probably tell, the following files need to be exported:

AppStream information in share/appdata/galculator.appdata.xml
Icons in share/icons
The desktop entry in share/applications/galculator.desktop

Unfortunately, they are not prefixed with the application name. Thus, you have to rename them with the following properties in the manifest:

{
 "rename-appdata-file": "galculator.appdata.xml",
 "rename-icon": "galculator",
 "rename-desktop-file": "galculator.desktop"
}

Once these files are renamed, flatpak-builder will automatically export them. Once the Flatpak package is built, this will help in creating menu entries on the base system.

In addition, the share/man and share/pixmaps directories are not necessary, so you can remove them with:

{
 "cleanup": [
 "/share/man",
 "/share/pixmaps"
 ]
}

Configuring the sandbox

Since this is a GTK application, you must allow it to access X11 and Wayland, and share IPC mechanisms. You also must allow it to use OpenGL rendering. In addition, galculator needs access to dconf, and you have to add in the entries we discussed earlier. The switches, which would have been passed to build-finish are listed in the finish-args property in the manifest:

{
 "finish-args": [
 "--share=ipc",
 "--socket=x11",
 "--socket=wayland",
 "--device=dri",
 "--filesystem=xdg-run/dconf",
 "--filesystem=~/.config/dconf:ro",
 "--talk-name=ca.desrt.dconf",
 "--env=DCONF_USER_CONFIG_DIR=.config/dconf"
 ]
}

Completing the manifest

At this point, you have configured most of the important parts of the manifest. Configuring the rest should be fairly easy, and the final manifest we used is here. We have added a set of build-options, which sets CFLAGS and CXXFLAGS to -O3.

Building and testing the application

Finally, we can build the application. Simply run flatpak-builder with a build directory and the manifest. In this example, galculator is the build directory and org.mnim.galculator.json is the manifest file.

flatpak-builder galculator org.mnim.galculator.json

Next, export it into the repository like we did previously:

flatpak build-export my-repo galculator

Now, you can test the application by installing it:

flatpak install --user test-repo org.mnim.galculator

The application is immediately installed, and you even get desktop entries like a normal application! You can launch it from the menu entry, or run it with flatpak run as we did earlier.

Distributing applications

Now that you have built an application, how do you go about distributing it? The easiest way to do so is to transfer the repository directory on a server, and serve it over the web. Then, you can ask users to add your repository to their list, and install applications that way.

Many people prefer to install software signed with GPG, since it proves that the application indeed comes from the original maintainer. You can create GPG keys easily with the gpg2 command:

gpg2 --quick-key-gen dev@example.com

Make sure to note down the key ID as you will need it later. In our example, we will assume that the key ID is A318C188C20D410A.

Then, you can create a GPG-signed repository when running build-export like so:

flatpak build-export my-repo galculator --gpg-sign=A318C188C20D410A

If you have an existing unsigned repository, then you can sign it like so:

flatpak build-sign my-repo --gpg-sign=A318C188C20D410A
flatpak build-update-repo my-repo --gpg-sign=A318C188C20D410A

When you want to publish the repository, be sure to provide users with a copy of the public key. To export your public key, run:

gpg2 --export A318C188C20D410A > public.gpg

You should publish this key on your website. When your users want to add your repository, they should download the public key. If the user downloaded a key named public.gpg, they can now add the repository by running:

flatpak remote-add --gpg-import=public.gpg test-repo http://software.example.com/repo/

However, this method is extremely inconvenient for users. Fortunately, you can simplify the process by creating flatpakref and flatpakrepo files, which are discussed in detail in the documentation.

Conclusion

In this article, we have learnt about Flatpak, and how you can distribute software using this new technology. It is more security oriented and features better integration and update capabilities compared to traditional methods such as AppImage.

If you are interested in reading more about this topic, the developer documentation and the flatpak-manifest man page are good places to start with. If you want to read through more complex examples of manifests/flatpakref files, take a look at the Gnome-apps repository and the Flathub examples.

Getting Started with Docker - Part 2

Tue, 19 Sep 2017 12:15:22 +0000

Docker has changed the way developers run, organize and deploy their code and infrastructure. In the docker series articles, we will be understanding how Docker works, installation and running code, deployment, container management and more. In part 1, we explained how Docker works and how it is different from traditional systems involving virtual machines. Do check out the first part, An Introduction to the Docker Ecosystem before starting out with this article.

Installation

Docker comes in two versions, Community Edition(CE) and Enterprise Edition(EE). The Enterprise Edition is focused towards corporate production environments where stability is very crucial. They also give support for the infrastructure by charging a fee. The Community Edition is the free version which is perfect for developers and small teams to experiment with the architecture and deploy apps for non-mission critical environments.

To install Docker CE in your system, please follow the official installation instructions for your respective operating system.

NOTE: If you want to install docker in Windows, do note that it will only work in Windows 10. Also, make sure you have enabled Hyper-V and also Virtualization in your BIOS settings.

Getting started with the commands

After you have installed Docker, the view the list of available commands, just run:

docker

It will output the following commands along with their descriptions.

Management Commands:
 config Manage Docker configs
 container Manage containers
 image Manage images
 network Manage networks
 node Manage Swarm nodes
 plugin Manage plugins
 secret Manage Docker secrets
 service Manage services
 stack Manage Docker stacks
 swarm Manage Swarm
 system Manage Docker
 volume Manage volumes

Commands:
 attach Attach local standard input, output, and error streams to a running container
 build Build an image from a Dockerfile
 commit Create a new image from a container's changes
 cp Copy files/folders between a container and the local filesystem
 create Create a new container
 diff Inspect changes to files or directories on a container's filesystem
 events Get real time events from the server
 exec Run a command in a running container
 export Export a container's filesystem as a tar archive
 history Show the history of an image
 images List images
 import Import the contents from a tarball to create a filesystem image
 info Display system-wide information
 inspect Return low-level information on Docker objects
 kill Kill one or more running containers
 load Load an image from a tar archive or STDIN
 login Log in to a Docker registry
 logout Log out from a Docker registry
 logs Fetch the logs of a container
 pause Pause all processes within one or more containers
 port List port mappings or a specific mapping for the container
 ps List containers
 pull Pull an image or a repository from a registry
 push Push an image or a repository to a registry
 rename Rename a container
 restart Restart one or more containers
 rm Remove one or more containers
 rmi Remove one or more images
 run Run a command in a new container
 save Save one or more images to a tar archive (streamed to STDOUT by default)
 search Search the Docker Hub for images
 start Start one or more stopped containers
 stats Display a live stream of container(s) resource usage statistics
 stop Stop one or more running containers
 tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
 top Display the running processes of a container
 unpause Unpause all processes within one or more containers
 update Update configuration of one or more containers
 version Show the Docker version information
 wait Block until one or more containers stop, then print their exit codes

We will go through some of the most important commands that are used almost always.

Playing with Docker Images

In our previous article, we talked about the concept of images. There are a lot of repositories which offer pre-built images. The official one is called the Docker Hub.

Suppose that I want to use python in my Docker container. To search for the pre-built python image in the Docker Hub, use the following command.

docker search python

It will return a list of public images from the Docker Hub. The one at the top with the official tag and most number of stars is the official python image. Similarly, you can search for any image that you want to use in your container using the docker search command.

After you have searched your image, run it by using:

docker run -d <image_name>

The -d flag is optional and is used to run the command in the background without capturing the entire terminal. It is specifically useful for services like web servers or databases like Redis.

Now, there may be a lot of commands running in the background. To list all of them use:

docker ps

It will list a table with the following columns:

CONTAINER ID - This is a random ID which is used to identify the processes.
IMAGE - This is the name of the image that you pulled from the Docker Hub
COMMAND - This is the command which is being run by the respective process
CREATED - The time at which the process is created
STATUS - It shows the amount of time the process was running
PORTS - This is the ports to which the process gets attached to
NAMES - Docker assigns a random friendly name which can be used to identify the process apart from the CONTAINER ID

By default, the ps command shows the list of running processes. To list all the processes use the -a or the --all flag

docker ps -a

Now, if you want to get to the details of a single process, use the inspect command.

docker inspect <container-id>

You can also use the friendly-name instead of the container id. It will give the same output.

There may be situations where you may just pull an image from the Docker Hub and run it later sometime. Use the following command to do it.

docker pull <image-name>

To see the list of all the images that you have pulled/downloaded use:

docker images

It will list out all of the images along with their sizes, image id and creation date.

To stop a docker image use the command:

docker stop <container-id|friendly-name>

Similarly to permanently delete a container, type:

docker rm <container-id|friendly-name>

Portable environments

To run apps in docker, we do not need to install any languages or development environments natively on our host machine. Now your question may be that how are the applications going to run without installing any environmental dependencies? This is achieved by using portable instances or runtimes without the need of installing anything. The biggest benefit is that it is like sandboxing and does not mess up with your host system. We have already seen how to download pre-built images from the Docker Hub. Every time we set up our environment in any machine, the dependent images of the application needs to be downloaded from the Docker Hub. Before doing that, we need to define the instances/dependencies that are necessary to run our application. We define them in a file called the Dockerfile.

Dockerfile is a text file in which steps are defined to create the environment. After it has been defined and configured, we perform some build steps to form an image which can be run to get a container up and running. All these may seem intimidating, so let’s understand all of it with some examples.

Example app

We will be building a demo PHP app that prints the text “Boolean World” in the browser. Now just imagine the dependencies that would have been required if we went for the traditional architecture. We would need a web server and also the different PHP modules installed in our system. This list goes huge for any modern web app and you certainly don’t want to install those dependencies manually every time you run your app on a different machine. So, we will be using some pre-built images that are available on the Docker Hub.

So, head over to Docker Hub and search for php. Click on the top result which says php official. That is the official php repository which we will be using. Now you should be seeing a big list of supported tags for the Dockerfile.

These are nothing but the different php modules like php-fpm or php-cli. We need the image with Apache on it for the example. So we will be using the 7-apache from the list of tags. If you scroll down in that page, you can see there are a few examples given on how to use the tags in the Dockerfile.

Similarly, you can use the docker search command instead of searching in the Docker Hub page.

Create a new folder called Example and inside it, create a file named index.php. Now paste the following contents inside the file:

<?php
echo "Boolean World"

Save the file. In the same Example folder, create another file named Dockerfile. Paste the following content inside it:

FROM php:7-apache
COPY Example/ /var/www/html
EXPOSE 80

The first line is the tag that we used to fetch the respective image from the Docker Hub. The second line links up the directory to the webroot of Apache. The third line opens up HTTP port 80 so that we can see the output in a browser.

The final directory structure should look like this:

Example
 |
 +-- index.php
 |
 +-- Dockerfile

Now we need to build our customised image for the specifications we mentioned in the Dockerfile. Open up the terminal and navigate to the Example directory. Type in the following command to build the image.

docker build -t demo-app .

We named the image as demo-app. After the name, there is also a period (.) which specifies that the Dockerfile is in the current directory. The command should take a while to execute since it has to pull in all the components from the Docker Hub. After it’s done, type in the following command.

docker run -p 80:80 demo-app

The -p specifies port forwarding where we forwarded port 80 from the host machine to port 80 of the container.

After you run the above command Apache server should start. Open your browser and type in localhost at the address bar. You should see Boolean World displayed on the page.

The interactive mode

Docker also provides an interactive shell for any application/image that supports CLI based inputs. For example, if you want to use the Ubuntu shell, simply write:

docker run -it ubuntu

The -it flag sets up the interactive shell. It will give the following output:

root@e0e0689e3490

This is the shell for Ubuntu. Type in any of the Ubuntu commands and it will work. To exit the interactive shell, simply type:

exit

Commit and push

After you have worked with your image, you may want to save the state of the container for future use. This can be done using the commit command which is very similar to that of git version control.

docker commit -m "Commit message" <container-id> <repository-location>:<tag>

For example:

docker commit -m "This is a demo" 197387f1b436 bool/test-image:blog

Do note that here bool is the username. Usually, the username should be the one which you created when you signed up for Docker Hub. Also, blog is the tag given to easily identify between different commits.

Before you can push your image to the Docker Hub, you need to log in.

docker login -u <username>

After you log in by providing your password, push the image you want by typing:

docker push <username>/<docker-image-name>

After that, it should reflect on the repository’s dashboard.

Conclusion

In this article, we familiarised ourselves with the different commands of docker and made a demo application to display something on a web page. The biggest benefits are the sandboxing and portability features. Running applications in docker consume just a fraction of resources than it would in a full-fledged traditional virtual machine. Do let us know about your experience on playing with the commands.

A Guide to the Linux "Top" Command

Mon, 11 Sep 2017 18:55:44 +0000

The top command allows users to monitor processes and system resource usage on Linux. It is one of the most useful tools in a sysadmin’s toolbox, and it comes pre-installed on every distribution. Unlike other commands such as ps, it is interactive, and you can browse through the list of processes, kill a process, and so on.

In this article, we are going to understand how to use the top command.

Getting started

As you might have already guessed, you simply need to type this in to launch top:

top

This starts up an interactive command line application, similar to one in the screenshot below. The upper half of the output contains statistics on processes and resource usage, while the lower half contains a list of the currently running processes. You can use the arrow keys and Page Up/Down keys to browse through the list. If you want to quit, simply press “q”.

There are a number of variants of top, but in the rest of this article, we will talk about the most common variant — the one that comes with the “procps-ng” package. You can verify this by running:

top -v

If you have this variant, this will show up in the output, like so:

procps-ng version 3.3.10

There is quite a bit going on in top’s interface, so we will break it down bit by bit in the next section.

Understanding top’s interface: the summary area

As we have previously seen, top’s output is divided into two different sections. In this part of the article, we’re going to focus on the elements in half of the output. This region is also called the “summary area”.

System time, uptime and user sessions

At the very top left of the screen (as marked in the screenshot above), top displays the current time. This is followed by the system uptime, which tells us the time for which the system has been running. For instance, in our example, the current time is “15:39:37”, and the system has been running for 90 days, 15 hours and 26 minutes.

Next comes the number of active user sessions. In this example, there are two active user sessions. These sessions may be either made on a TTY (physically on the system, either through the command line or a desktop environment) or a PTY (such as a terminal emulator window or over SSH). In fact, if you log in to a Linux system through a desktop environment, and then start a terminal emulator, you will find there will be two active sessions.

If you want to get more details about the active user sessions, use the who command.

Memory usage

The “memory” section shows information regarding the memory usage of the system. The lines marked “Mem” and “Swap” show information about RAM and swap space respectively. Simply put, a swap space is a part of the hard disk that is used like RAM. When the RAM usage gets nearly full, infrequently used regions of the RAM are written into the swap space, ready to be retrieved later when needed. However, because accessing disks are slow, relying too much on swapping can harm system performance.

As you would naturally expect, the “total”, “free” and “used” values have their usual meanings. The “avail mem” value is the amount of memory that can be allocated to processes without causing more swapping.

The Linux kernel also tries to reduce disk access times in various ways. It maintains a “disk cache” in RAM, where frequently used regions of the disk are stored. In addition, disk writes are stored to a “disk buffer”, and the kernel eventually writes them out to the disk. The total memory consumed by them is the “buff/cache” value. It might sound like a bad thing, but it really isn’t — memory used by the cache will be allocated to processes if needed.

Tasks

The “Tasks” section shows statistics regarding the processes running on your system. The “total” value is simply the total number of processes. For example, in the above screenshot, there are 27 processes running. To understand the rest of the values, we need a little bit of background on how the Linux kernel handles processes.

Processes perform a mix of I/O-bound work (such as reading disks) and CPU-bound work (such as performing arithmetic operations). The CPU is idle when a process performs I/O, so OSes switch to executing other processes during this time. In addition, the OS allows a given process to execute for a very small amount of time, and then it switches over to another process. This is how OSes appear as if they were “multitasking”. Doing all this requires us to keep track of the “state” of a process. In Linux, a process may be in of these states:

Runnable (R): A process in this state is either executing on the CPU, or it is present on the run queue, ready to be executed.
Interruptible sleep (S): Processes in this state are waiting for an event to complete.
Uninterruptible sleep (D): In this case, a process is waiting for an I/O operation to complete.
Stopped (T): These processes have been stopped by a job control signal (such as by pressing Ctrl+Z) or because they are being traced.
Zombie (Z): The kernel maintains various data structures in memory to keep track of processes. A process may create a number of child processes, and they may exit while the parent is still around. However, these data structures must be kept around until the parent obtains the status of the child processes. Such terminated processes whose data structures are still around are called zombies.

Processes in the D and S states are shown in “sleeping”, and those in the T state are shown in “stopped”. The number of zombies are shown as the “zombie” value.

CPU usage

The CPU usage section shows the percentage of CPU time spent on various tasks. The us value is the time the CPU spends executing processes in userspace. Similarly, the sy value is the time spent on running kernelspace processes.

Linux uses a “nice” value to determine the priority of a process. A process with a high “nice” value is “nicer” to other processes, and gets a low priority. Similarly, processes with a lower “nice” gets higher priority. As we shall see later, the default “nice” value can be changed. The time spent on executing processes with a manually set “nice” appear as the ni value.

This is followed by id, which is the time the CPU remains idle. Most operating systems put the CPU on a power saving mode when it is idle. Next comes the wa value, which is the time the CPU spends waiting for I/O to complete.

Interrupts are signals to the processor about an event that requires immediate attention. Hardware interrupts are typically used by peripherals to tell the system about events, such as a keypress on a keyboard. On the other hand, software interrupts are generated due to specific instructions executed on the processor. In either case, the OS handles them, and the time spent on handling hardware and software interrupts are given by hi and si respectively.

In a virtualized environment, a part of the CPU resources are given to each virtual machine (VM). The OS detects when it has work to do, but it cannot perform them because the CPU is busy on some other VM. The amount of time lost in this way is the “steal” time, shown as st.

Load average

The load average section represents the average “load” over one, five and fifteen minutes. “Load” is a measure of the amount of computational work a system performs. On Linux, the load is the number of processes in the R and D states at any given moment. The “load average” value gives you a relative measure of how long you must wait for things to get done.

Let us consider a few examples to understand this concept. On a single core system, a load average of 0.4 means the system is doing only 40% of work it can do. A load average of 1 means that the system is exactly at capacity — the system will be overloaded by adding even a little bit of additional work. A system with a load average of 2.12 means that it is overloaded by 112% more work than it can handle.

On a multi-core system, you should first divide the load average with the number of CPU cores to get a similar measure.

In addition, “load average” isn’t actually the typical “average” most of us know. It is an “exponential moving average”, which means a small part of the previous load averages are factored into the current value. If you’re interested, this article covers all the technical details.

Understanding top’s interface: the task area

The summary area is comparatively simpler, and it contains a list of processes. In this section, we will learn about the different columns shown in top’s default output.

PID

This is the process ID, a unique positive integer that identifies a process.

USER

This is the “effective” username (which maps to a user ID) of the user who started the process. Linux assigns a real user ID and an effective user ID to processes; the latter allows a process to act on behalf of another user. (For example, a non-root user can elevate to root in order to install a package.)

PR and NI

The “NI” field shows the “nice” value of a process. The “PR” field shows the scheduling priority of the process from the perspective of the kernel. The nice value affects the priority of a process.

VIRT, RES, SHR and %MEM

These three fields are related with to memory consumption of the processes. “VIRT” is the total amount of memory consumed by a process. This includes the program’s code, the data stored by the process in memory, as well as any regions of memory that have been swapped to the disk. “RES” is the memory consumed by the process in RAM, and “%MEM” expresses this value as a percentage of the total RAM available. Finally, “SHR” is the amount of memory shared with other processes.

As we have seen before, a process may be in various states. This field shows the process state in the single-letter form.

TIME+

This is the total CPU time used by the process since it started, precise to the hundredths of a second.

COMMAND

The COMMAND column shows the name of the processes.

Top command usage examples

So far, we have discussed about top’s interface. However, it can also manage processes, and you can control various aspects of top’s output. In this section, we’re going to take at a few examples.

In most of the examples below, you have to press a key while top is running. Keep in mind that these keypresses are case sensitive — so if you press “k” while Caps Lock is on, you have actually pressed a “K”, and the command won’t work, or do something else entirely.

Killing processes

If you want to kill a process, simply press ‘k’ when top is running. This will bring up a prompt, which will ask for the process ID of the process and press enter.

Next, enter the signal using which the process should be killed. If you leave this blank, top uses a SIGTERM, which allows processes to terminate gracefully. If you want to kill a process forcefully, you can type in SIGKILL here. You can also type in the signal number here. For example, the number for SIGTERM is 15 and SIGKILL is 9.

If you leave the process ID blank and hit enter directly, it will terminate the topmost process in the list. As we’ve mentioned previously, you can scroll using the arrow keys, and change the process you want to kill in this way.

Sorting the process list

One of the most frequent reasons to use a tool like top is to find out which process is consuming the most resources. You can press the following keys to sort the list:

‘M’ to sort by memory usage
‘P’ to sort by CPU usage
‘N’ to sort by process ID
‘T’ to sort by the running time

By default, top displays all results in descending order. However, you can switch to ascending order by pressing ‘R’.

You can also sort the list with the -o switch. For example, if you want to sort processes by CPU usage, you can do so with:

top -o %CPU

You can sort the list by any of the attributes in the summary area in the same way.

Showing a list of threads instead of processes

We have previously touched upon how Linux switches between processes. Unfortunately, processes do not share memory or other resources, making such switches rather slow. Linux, like many other operating systems, supports a “lightweight” alternative, called a “thread”. They are part of a process and share certain regions of memory and other resources, but they can be run concurrently like processes.

By default, top shows a list of processes in its output. If you want to list the threads instead, press ‘H’ when top is running. Notice that the “Tasks” line says “Threads” instead, and shows the number of threads instead of processes.

You may have noticed how none of the attributes in the process list changed. How is that possible, given that processes differ from threads? Inside the Linux kernel, threads and processes are handled using the same data structures. Thus, every thread has its own ID, state and so on.

If you want to switch back to the process view, press ‘H’ again. In addition, you can use the -H switch to display threads by default.

top -H

Showing full paths

By default, top does not show the full path to the program, or make a distinction between kernelspace processes and userspace processes. If you need this information, press ‘c’ while top is running. Press ‘c’ again to go back to the default.

Kernelspace processes are marked with square brackets around them. As an example, in the above screenshot there are two kernel processes, kthreadd and khelper. On most Linux installations, there will usually be a few more of them.

Alternatively, you can also start top with the -c argument:

top -c

Forest view

Sometimes, you may want to see the child-parent hierarchy of processes. You can see this with the forest view, by pressing ‘v’/‘V’ while top is running.

As you can tell from the screenshot above, the systemd process was the first one to start up on the system. It has started processes such as sshd, which in turn has created other sshd processes, and so on.

Listing processes from a user

To list processes from a certain user, press ‘u’ when top is running. Then, type in the username, or leave it blank to display processes for all users.

Alternatively, you can run the top command with the -u switch. In this example, we have listed all processes from the root user.

top -u root

Filtering through processes

If you have a lot of processes to work with, a simple sort won’t work well enough. In such a situation, you can use top’s filtering to focus on a few processes. To activate this mode, press ‘o’/‘O’. A prompt appears inside top, and you can type a filter expression here.

A filter expression is a statement that specifies a relation between an attribute and a value. Some examples of filters are:

COMMAND=getty: Filter processes which contain “getty” in the COMMAND attribute.
!COMMAND=getty: Filter processes which do not have “getty” in the COMMAND attribute.
%CPU>3.0: Filter processes which have a CPU utilization of more than 3%.

Once you’ve added a filter, you can further prune down things by adding more filters. To clear any filters you have added, press ‘=’.

Changing the default look of the CPU and memory statistics

If you are mostly at home in a GUI environment, you might not like top’s default way of showing CPU and memory statistics. You can press ’t’ and ’m’ to change the style of the CPU and memory statistics. Here is a screenshot of top, where we have pressed ’t’ and ’m’ once.

If you press ’t’ or ’m’ repeatedly, it cycles through four different views. In the first two presses, it cycles through two different types of progress bars. If you press the key for a third time, the progress bar is hidden. If you press the key again, it brings back the default, text based counters.

Saving your settings

If you’ve made any changes to top’s output, you can save them for later use by pressing ‘W’. top writes its configuration to the .toprc file in your home directory.

Conclusion

The top command is extremely helpful for monitoring and managing processes on a Linux system. This article only scratches the surface, and there is quite a bit we haven’t covered. For example, there are a lot more columns you can add to top. For all this stuff, be sure to check out the man page by running man top on your system.

How to Configure nginx as a Load Balancer

Mon, 28 Aug 2017 19:40:56 +0000

For very small websites, a single server running a web application is usually enough. However, this doesn’t work well for larger websites. In such cases, we make use of “load balancing”, a technique that distributes incoming requests to a number of other servers, each running a copy of the web application. This is one of the techniques large websites use to stay up and running even with large volumes of traffic.

Although most people use nginx as a HTTP server, it also has built-in load balancing features. In this article, we’re going to look at how to set it up on your system.

Load balancing basics

Load balancing is a method to distribute workloads across multiple computing resources, such as computers, network links or disks. It aims to improve use of resources, maximize throughput, improve response times, and ensure fault-tolerance. In this article, we will talk specifically about the types of load balancing supported by nginx. However, other applications (such as database servers) can also make use of load balancing.

A typical HTTP load balancing setup looks like the one shown in the diagram below. There are many users who send in requests, and a load balancer distributes them to many servers (called backends). Each of these backends run a copy of the same application. When these servers send out a response, the load balancer send proxies them out to the clients.

Load balancers are usually categorized as hardware-based and software-based. Hardware-based load balancers are dedicated equipment, and they carry application-specific integrated circuits (ASICs) that routes traffic very efficiently. On the other hand, software-based load balancers such as nginx or HAproxy perform the load balancing in software.

In addition, load balancing may be performed at various network layers. As the name suggests, layer 4 load balancers balance traffic by inspecting the requests and responses at the transport layer. In addition, there are layer 7 load balancers, which inspect traffic at the application layer to figure things out. The latter is slightly more powerful, as it has a high level view of things. For instance, a layer 7 balancer that’s aware of HTTP can inspect URLs, and dispatch requests to different sets of backends depending upon the URL.

nginx can perform both layer 4 load balancing for TCP and UDP, as well as layer 7 HTTP load balancing. In the next few sections, we’re going to see how to configure nginx for this purpose.

A note about the configuration file

On most Linux distributions, the nginx configuration file is in /etc/nginx/nginx.conf. However, on Debian/Ubuntu, this file is split into two different ones: /etc/nginx/nginx.conf for the main configuration, and /etc/nginx/sites-enabled/default for the individual websites that you host. If you are using Debian/Ubuntu, you should add the upstream blocks into the former, and the location block in the latter.

Load balancing web applications

Before you continue any further, you need to set up a few backend servers where our web application will be running. These servers will be running a HTTP server on each of these servers. For our example, we will assume that we’ve already set them up, and these backend servers are available at 192.168.0.1, 192.168.0.2 and 192.168.0.3. Of course, these need not be internal IPs — they can be external IPs or hostnames as well.

Now, there are two things you need to configure nginx as a load balancer. First, in the http section, you need to specify a set of backend servers with an upstream block. Then, you need to proxy these requests to the servers. In order to do so, open the configuration file and add the following lines:

http {
 upstream backend1 {
 server 192.168.0.1;
 server 192.168.0.2;
 server 192.168.0.3;
 }

 server {
 listen 80;

 location / {
 proxy_pass http://backend1;
 }
 }
}

This tells nginx to pass over HTTP requests for any URL to the set of backend servers we’ve named backend1. If you want to pass over specific URLs only, you could do something like:

location /shop {
 proxy_pass http://backend1;
}

This will match all URLs that begin with “/shop” and hand them over to the backend servers. You can also extend this concept to pass over different URLs to different backends. In this example, we’ve added another backend for the servers that host a blog. Then we’ve configured nginx to pass over URLs that begin with “/blog” to the new backend.

upstream backend1 {
 # list of servers
}

upstream backend2 {
 # list of servers
}

location /shop {
 proxy_pass http://backend1;
}

location /blog {
 proxy_pass http://backend2;
}

nginx also supports a couple proxying to a couple of other protocols with directives such as FastCGI and uWSGI with the fastcgi_pass and uwsgi_pass directives. Say, for example, there is a PHP-FPM daemon running on every backend server on port 9000, and you want to proxy the requests to them. This is how the configuration would look like:

http {
 upstream backend1 {
 server 192.168.0.1:9000;
 server 192.168.0.2:9000;
 server 192.168.0.3:9000;
 }

 server {
 listen 80;

 location / {
 fastcgi_pass backend1;
 }
 }
}

Load balancing strategies

nginx has a couple of strategies to pick a server to send requests to. By default, it uses a round robin algorithm to decide the server to which a request should be sent. However, other strategies are available too, which you can enable manually. The least_conn strategy chooses a server that is handling the least number of connections. On the other hand, the ip_hash strategy chooses servers based on the result of running a hash function on the IP. This means that requests from the same IP end up going to the same server.

To use a strategy other than the default, simply mention its name in the upstream block:

upstream backend {
 ip_hash;
 server 192.168.0.1;
 server 192.168.0.2;
 server 192.168.0.3;
}

There’s also a a generic hash directive, which can be used on any HTTP value. In this example, we’ve distributed requests based on the URL:

hash $request_uri;

You can further change the default strategy by assigning weights to a server. By default, all servers have a weight of 1, which means that nginx does not “prefer” a server over another, apart from applying the rules of the strategy used. An example configuration with weights looks like the following:

upstream backend {
 server 192.168.0.1 weight=4;
 server 192.168.0.2 weight=2;
 server 192.168.0.3;
}

As you can tell, we’re using the round robin strategy in this example. With the weights we’ve shown in the example above, every 4 out of 7 requests would go to the first server, 2 of them to the second, and 1 to the third. (As we’ve mentioned previously, the default weight is 1, so we’ve left it out in the above example.)

Other advanced configuration

When nginx proxies requests to these servers, it also passively performs a health check. If a server fails to send out a response, it is automatically marked as such, and nginx avoids sending requests to it for some time. The fail_timeout value controls the time till which nginx avoids sending requests to the server. The max_fails value sets the number of requests that should fail, after which nginx marks the server as such.

In the example below, we’ve set max_fails to 3, and the fail_timeout to 20 seconds for the first server:

upstream backend {
 server 192.168.0.1 max_fails=3 fail_timeout=20s;
 server 192.168.0.2;
}

In addition, you can mark a particular server to be treated differently. If you’re performing maintenance on one of the backend servers, you can mark it as down. The requests that would be processed by this server are now sent over to the next server in the group. You can also mark a particular server as a backup server. Requests will be passed to it when all other servers are unavailable. An example of these two directives is shown below:

upstream backend {
 server 192.168.0.1;
 server 192.168.0.2;
 server 192.168.0.3 down;
 server 192.168.0.4 backup;
}

Some common problems

Depending upon the kinds of applications that you have deployed, a load balancer may cause certain problems. Often, you will get 404 or other errors from the web application. This is usually because the “Host” header contains the address of the backend server instead of its original value. To fix this, you can manually set it in the location block:

location / {
 proxy_set_header Host $host;
 proxy_pass http://backend;
}

Many complex web applications store state locally, and they can fail to work if a load balancer distributes requests of a session across different servers. In order to resolve this, the load balancer must keep track of the original backend server that responded to the initial request. This concept is known as “session persistence”. Unfortunately, this feature is only available in the commercial version of nginx. However, as a workaround, you can try the ip_hash directive, which will forward requests from the same IP to the same backend server. IPs usually don’t change very frequently, so this may work well enough for your use case.

Transport layer load balancing

As we’ve mentioned previously, nginx can also perform load balancing at transport layer. The syntax is slightly different than the one we’ve seen previously. Specifically, both the upstream and server sections are contained inside a stream block.

To use this feature, you have to compile nginx with the --with-stream flag. On Debian and Ubuntu, the version of nginx in the repositories is already compiled with this flag, so you can directly use it.

Suppose, you have a number of DNS servers, and you want to use nginx as a load balancer. You simply need to put this in the configuration file:

stream {
 upstream dns_backends {
 server 8.8.8.8:53;
 server 8.8.4.4:53;
 }

 server {
 listen 53 udp;
 proxy_pass dns_backends;
 proxy_responses 1;
 }
}

Here, we’ve defined a number of DNS backends, and then configured nginx to listen to incoming UDP packets on port 53. The proxy_pass directive sends them over to the backend servers. Also, by default, nginx expects that the backend may send out one or more responses. Because there’s going to be a single reply against a single request, we’ve set proxy_responses to 1.

Load balancing with TCP is quite similar.

stream {
 upstream tcp_backend {
 server srv1.example.com:3306;
 server srv2.example.com:3306;
 }
 server {
 listen 3306;
 proxy_pass tcp_backend;
 }
}

The load balancing strategies and other configuration options that we’ve described previously also apply here.

Conclusion

nginx provides powerful load balancing capabilities with a wide variety of options. If you want to learn more, be sure to check out the official documentation.

How to Install Grav CMS on Debian/Ubuntu

Mon, 21 Aug 2017 17:38:57 +0000

The Content Management System (CMS) market today is captured by the three giants — Wordpress, Drupal, and Joomla with Wordpress being the leader. Grav CMS is a new kid on the block featuring an entirely different principle. It is an open-source flat-file CMS with exciting functionalities. We will look at some of Grav’s features and then explain how to deploy it on a server running Debian or Ubuntu.

Grav CMS features

Grav offers some promising features, unlike other traditional CMS. The main selling point is that it is based on a flat-file based architecture which basically means is that it does not need any database system to work. In addition, there are many other features that make Grav stand out:

Native markdown support: WYSIWYG editors are good but the HTML output that comes out of it is definitely not clean. Grav has native support for Markdown which makes it easier to write content in a better format.
SEO Friendly: Grav provides clean URLs along with configuration files that can be used to set the global and per-page metadata.
Debugging/Logging support: There is a visual debugging panel which can be used to log and identify errors easily.
In built routing and redirection: You don’t have to mess around with the web server’s configuration to handle redirection and routing. There is a site configuration file in which you can specify the routes and redirects and it will handle the rest on its own.
Version Control Support: You can use file storage services to sync and backup your data and also use version control systems like GIT or SVN.
Native Retina image support: Grav provides native Retina images without the need to install any external plugins.
Flexible CSS framework: You can use any CSS framework like Bootstrap, Materialize or Foundation.
Awesome technologies: Grav uses the most modern coding standards and technologies like Twig for templating, Doctrine cache, Symfony framework modules, markdown, YAML and many others.
Easy installation: The best part about a flat-file CMS is that there is no need to install anything and the dependencies are also minimal. You just need to place the Grav files at the webroot of your web server and it’s done. No configuration is required to get a site up and running.

Installation

The only requirement for installation is that you should have a web server like Nginx or Apache and PHP installed on your system. Some additional PHP modules are required. So, install them using the command below.

sudo apt update
sudo apt install php-fpm php-gd php-curl php-zip php-mbstring php-xml

Now download Grav core + admin ZIP file from the downloads page and extract the folder to the webroot of your server. Usually, the location is /var/www/html. We can do so by running:

wget <latest URL for Grav core + admin file> -O grav.zip
unzip grav.zip
sudo mv grav-admin/ /var/www/html/grav

Now, let’s fix the permissions to the webroot folder. Enter the commands below in your terminal:

sudo chown -R www-data: /var/www/html/grav/

Now, depending upon your web server you are using, use the instructions below to complete the installation.

Instructions for Apache

If you are using Apache, we need to enable some modules and permissions for Grav to run smoothly. We need the proxy and proxy_fcgi modules so that PHP scripts can work. Additionally, we also need the rewrite module for clean URLs to work.

sudo a2enmod proxy proxy_fcgi rewrite

Now we also need to enable the configuration file which allows Apache to talk with PHP-FPM. Now, this is dependent on the PHP version that you are using. If you have PHP 7.0 installed then the command will be:

sudo a2enconf php7.0-fpm

If you have any other version of PHP, then replace 7.0 with your version number. If that didn’t work either, try using php-fpm instead of php7.0-fpm in the command above.

After enabling the modules, restart Apache. On newer versions (such as Debian >= 8 and Ubuntu > 16.04), you can do so using:

sudo systemctl restart apache2

However, if you are using an old version, you should use:

sudo service apache2 restart

Now, open your browser and visit your site by typing in the IP address/domain name of your server in which you installed Grav. It should be http://<your_server_IP>/grav/. If the site loads up properly without any errors, skip to the next section.

If things didn’t work out…

Now if you get a error, it’s probably due to the fact that Apache is not able to read the .htaccess file provided by Grav and apply the rewrite rules. In order to solve this problem, we need to set AllowOverride to All in the Apache configuration file.

Open the configuration file at /etc/apache/apache2.conf in an editor and navigate to the <Directory /var/www> block and change AllowOverride from None to All. The block should look like this:

<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>

Save the file and restart Apache in the same way we have done earlier.

Instructions for Nginx

If you’re using Nginx, we need to add some rules to configure it. Open /etc/nginx/sites-enabled/default with a text editor.

Here, you’ll find a line in the server block that contains index directive. It tells Nginx to serve a default file when an user navigates to a directory. We will add index.php to this list, like so:

server {
 index index.php index.html index.htm index.nginx-debian.html;
}

Then, we need to add in the following directives to the server block so that Grav works properly. If you want to change the path of your Grav installation later, simply replace all instances of /grav with the appropriate directory.

location /grav {
 try_files $uri $uri/ /grav/index.php?_url=$uri&$query_string;
}

location ~* ^/grav/(\.git|cache|bin|logs|backup|tests)/.*$ {
 return 403;
}

location ~* ^/grav/(\.git|cache|bin|logs|backup|tests)/.*$ {
 return 403;
}

location ~* ^/grav/(system|vendor)/.*\.(txt|xml|md|html|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ {
 return 403;
}

location ~* ^/grav/user/.*\.(txt|md|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ {
 return 403;
}

location ~ ^/grav/(LICENSE\.txt|composer\.lock|composer\.json|nginx\.conf|web\.config|htaccess\.txt|\.htaccess) {
 return 403;
}

Now, we’ll tell how Nginx should handle PHP files. Again, we will add this block within the server block. Depending upon the version of your distribution, you may need to change the value of fastcgi_pass. For example, if you’re using PHP 7.1, you have to change it to /run/php/php7.1-fpm.sock. However, if you’re using PHP5, you should change it to /var/run/php5-fpm.sock.

location ~ \.php$ {
 fastcgi_pass unix:/run/php/php7.0-fpm.sock;

 fastcgi_split_path_info ^(.+\.php)(/.+)$;
 fastcgi_index index.php;
 include fastcgi_params;
 fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
}

Finally, we’ll restart Nginx in the same way as we did for PHP-FPM:

sudo systemctl restart nginx
sudo service nginx restart

Setting up the admin account

After you have performed all the above steps successfully, open your browser and head over to http://<your_server_IP>/grav/. You will see the following page.

Since we installed the Grav + Admin plugin version, we need to create an account to access the admin dashboard. Fill in all the details and click on “Create User” button. You will be presented with the admin dashboard.

You can now create articles, install plugins and themes, and change the settings to customize Grav according to your needs.

Conclusion

Grav is a beautiful, fast and modern CMS that has the capabilities to compete with the likes of Wordpress, Drupal or Joomla. In this article, we saw how to install and configure Grav in your server. You can read more about customization options and workflow at the official documentation. Do give Grav a shot and tell us what you think about it.

How to Install Gogs, a GitHub-Style Git Remote Service

Mon, 14 Aug 2017 08:53:05 +0000

There are many online services such as GitHub and Bitbucket, which allow users to host Git repositories online. However, when you are working on closed source projects, you may want to use a self-hosted solution. Gogs is a lightweight, web based self-hosted solution that allows you to host your Git repositories online. In fact, if you are familiar with GitHub, you will find yourself right at home using Gogs — they have a very similar interface.

In this article, we are going to learn how to install and set up Gogs on a Linux server. The instructions here should work for recent versions of Ubuntu (16.04 and later), Debian (version 8 and later) and CentOS/RHEL (version 7 and later).

Setting up the system

Before you can get started with Gogs, you will need to set up your system. In order to install Gogs, you should install Git, and optionally you should install SSH and a database such as MySQL or PostgreSQL. Gogs also carries its own bundled copy of SSH, and it can fall back on SQLite when a database isn’t available.

For this article, we will use MySQL as our database and we won’t use git over SSH. Thus, we only need to install Git and MySQL. On a Debian/Ubuntu system, you should run:

sudo apt update
sudo apt install git mysql-server

During the installation, you will be asked to set a password for the MySQL root user. Note down this password as you will need it later.

If you are on CentOS/RHEL, you can install these dependencies using commands below. (At present, CentOS ships with MariaDB, an open source, drop-in replacement for MySQL.)

sudo yum install git mariadb-server

Next, we will set up a restricted user account for Gogs to run on. In our example, we will create a user named gogs with the following command. We’ll also use /bin/true as the shell to prevent anyone from logging in as this user.

sudo useradd -rm gogs -s /bin/true -k ''

Now that we have got the basic things set up, we will move on to setting up the database.

Configuring the database

If you are on Debian/Ubuntu, the database server will be already started when you installed MySQL. However, with CentOS/RHEL, you will need to start it up manually. You will also have to “enable” it so that MySQL starts up whenever your system boots up. In order to do so, run the following commands:

sudo systemctl start mariadb
sudo systemctl enable mariadb

After this step, you should tighten up the security on MySQL by running:

mysql_secure_installation

This command will ask for the root user’s password. If you are a Debian/Ubuntu user, you have already provided a password, so type it in to continue. When the command asks if you would like to set a different password, answer No by entering “n”.

However, if you are a CentOS/RHEL user, you haven’t provided a password yet. Simply leave this field blank and hit Enter to continue. Next, you’ll be asked if you want to set a new password. Answer yes to the question by entering “y”, and enter a password here.

Then, the command will ask a few more questions, like whether it should remove the test database and disallow anonymous users. Answer yes to each of them by entering “y”.

Now, you should set up a user account and a database for Gogs. In order to do so, log in to the MySQL console as root with:

mysql -u root -p

Enter the MySQL root user’s password to continue.

First, you should create a user in MySQL. We will name our user gogs and give it a password super-strong-password:

CREATE USER gogs@localhost IDENTIFIED BY 'super-strong-password';

Then, you should create a database and allow the newly created user to work on the database. We will name our database gogs.

CREATE DATABASE gogs;
GRANT ALL PRIVILEGES ON gogs . * TO gogs@localhost;

Again, you should note down the database name, username and password you used — you will need them later.

Installing Gogs

First, you should grab the latest version of Gogs from the GitHub releases page. In this example, we will use the latest x64 version (0.11.43 at the time of this writing).

wget https://github.com/gogits/gogs/releases/download/v0.11.43/linux_amd64.tar.gz
sudo tar -xf linux_amd64.tar.gz -C /home/

Once you have completed this step, you should fix the permissions on the /home/gogs directory like so:

sudo chown -R gogs: /home/gogs

Now that we have downloaded the binaries and put them in their proper places, we need to tell systemd to start Gogs when the system boots up. In order to do so, create a file in /etc/systemd/system/gogs.service with the following contents:

[Unit]
Description=Gogs git remote daemon

[Service]
User=gogs
Group=gogs
ExecStart=/home/gogs/gogs web
WorkingDirectory=/home/gogs

[Install]
WantedBy=multi-user.target

While it might seem like there’s a lot going on here, it’s actually quite simple. We have told systemd to start up Gogs’ daemon under the gogs user. When the daemon starts up, it will serve the web interface over port 3000. You can change this port with the -p parameter like so:

ExecStart=/home/gogs/gogs web -p 4000

If you want to use a low port value (such as 80), you should grant the CAP_NET_BIND_SERVICE privilege by running:

sudo setcap 'cap_net_bind_service=+ep' /home/gogs/gogs

Then, you need to tell systemd to run this service on boot, and then start it up with:

sudo systemctl enable gogs
sudo systemctl start gogs

You can now visit http://<your_server_IP>:3000/ and you will see the Gogs installation page (pictured below).

On this page, enter the MySQL database name, username and password you have created for Gogs. You should also set the “Run user” to gogs and change the “Application URL” and “Domain” according to your domain name/IP. Finally, you should review the “Optional Settings” section and configure the administrator account. If you want to enable emails, fill out the SMTP settings.

If everything worked correctly and you have filled in the administrator account details, you will be taken to the dashboard:

Congratulations! You have installed Gogs successfully. You can now create repositories, push your projects and ask others to register to the service.

Configuring HTTPS

If you are deploying Gogs outside of a trusted internal network, you may want to use HTTPS to protect traffic coming in and leaving out of your Gogs instance. Fortunately, the built-in web server is quite fully featured, and supports HTTPS by default.

First, you should obtain a certificate from a certificate authority, or create a self-signed certificate yourself. (You can follow our Let’s Encrypt guide and follow the “Other web server” instructions to get a free, domain-validated certificate.) Once you have completed this step, open /home/gogs/custom/conf/app.ini in a text editor. Under the [server] section, change the protocol of ROOT_URL from http to https:

ROOT_URL = https://<your_server_hostname>:3000/

In addition, you also need to add these values to the [server] section:

CERT_FILE = /path/to/cert.pem
KEY_FILE = /path/to/key.pem
PROTOCOL = https

Then, restart Gogs with:

sudo systemctl restart gogs

Now, open https://<your_server_hostname>:3000/ in a browser. You will find that HTTPS has been deployed correctly.

If you want to use Gogs over HTTPS without a port number, you need to use port 443. This is a “low port”; and we’ve described how to run Gogs on a low port in the previous section. After you make changes to the service file, you should restart Gogs with:

sudo systemctl daemon-reload
sudo systemctl restart gogs

Running Gogs in combination with another server

If you are running multiple web applications on your server, you may prefer to access Gogs through the main HTTP server (which is usually Apache or nginx). You simply need to proxy all the requests from your main HTTP server to Gogs’ server.

In this article, we will assume you have a server named www.example.com, and you want to deploy Gogs on www.example.com/git. If you want it to be directly available on www.example.com, simply replace all instances of /git in the snippets below with /.

If you are using nginx, you simply need to use the proxy_pass like so:

server {
 # other server configuration

 location /git {
 proxy_pass http://127.0.0.1:3000/;
 }
}

If you are using Apache, you should enable the proxy and proxy_http modules. Then, you can configure Apache in a similar manner:

<VirtualHost *:80>
 # Other configuration
 ProxyPreserveHost On
 ProxyRequests off
 ProxyPass /git http://127.0.0.1:3000/
 ProxyPassReverse /git http://127.0.0.1:3000/
</VirtualHost>

Then, you need to tell Gogs about the URL it’s going to be accessed from. In order to do so, open /home/gogs/custom/conf/app.ini and change the value of ROOT_URL like so:

ROOT_URL = http://www.example.com/git

Finally, you should add a firewall rule to prevent anyone from directly accessing the Gogs installation through port 3000. If you’re using iptables, this is fairly easy to do:

sudo iptables -A INPUT ! -i lo -p tcp --dport 3000 -j DROP

Now you can access Gogs over http://www.example.com/git.

Updating Gogs

When a new version of Gogs comes out, you need to update it. Fortunately, updating Gogs is fairly easy. First, you need to stop the daemon with:

sudo systemctl stop gogs

Then, rename the existing folder, so that we can install the new version:

sudo mv /home/gogs /home/backup

Now, download the latest archive and extract it as we had done previously.

sudo tar -xf linux_amd64.tar.gz -C /home/

Next, copy over the data and logs from the old directory to the new directory:

sudo mv /home/backup/{custom,data,log} /home/gogs

Next, you need to fix the permissions on the new directory:

sudo chown -R gogs: /home/gogs/

Then, you can start up the service again with:

sudo systemctl enable gogs

If anything goes wrong, you can always restore the previous version from the backup directory.

Conclusion

In this article, we’ve seen how to install and configure Gogs. It has very minimal system requirements and features a familiar interface, thus making it a great alternative over other self-hosted git remote servers.

The configuration cheat sheet has some useful information if you want to configure your setup further.

Managing Your Linux Server with Webmin

Thu, 10 Aug 2017 08:52:41 +0000

Managing Linux servers can be very difficult at times. It can be very hard for someone who doesn’t have a lot of experience with the command-line tools. In addition, it’s also difficult for administrators to manage their servers while on the go. Logging in from a terminal emulator on a mobile device isn’t the most comfortable way to work. This is where control panels such as Webmin come in. Webmin is a web-based control panel for Linux systems. It provides an intuitive, easy-to-use interface for managing your server.

In this article, we’ll learn how to install and configure Webmin for your Linux server. We’ll also see how to use Webmin for the most common use cases.

Installing Webmin

First, we’ll begin by installing Webmin. Depending on the distribution you’re running, you will have to use different commands. In this article, we’ve given instructions for the most commonly used systems.

For Ubuntu/Debian (and derivatives)

The Webmin project provides a repository for Debian-based systems. You need to add it to apt’s list before you can use it.

echo "deb http://download.webmin.com/download/repository sarge contrib" | sudo tee /etc/apt/sources.list.d/webmin.list

Next, we need to add the Jamie Cameron’s PGP key so that apt can verify the packages.

wget -qO - http://www.webmin.com/jcameron-key.asc | sudo apt-key add -

Then, you can install Webmin with apt:

sudo apt update
sudo apt install webmin

For RHEL/CentOS

If you’re on RHEL/CentOS, the Webmin project provides a repository, which you should add to yum’s list of repositories like so:

cat << EOM | sudo tee /etc/yum.repos.d/webmin.repo
[Webmin]
name=Webmin Distribution Neutral
baseurl=http://download.webmin.com/download/yum
enabled=1
gpgcheck=1
gpgkey=http://www.webmin.com/jcameron-key.asc
EOM

Then, you can install Webmin with yum, just like we would have done for any other package:

sudo yum install webmin

Post installation

At this point, Webmin has been successfully installed on your system. You can access it by viewing https://<your_server_ip>:10000/ from your browser. If you’re having problems, you should check whether you’ve blocked this port, or if another software is running on the same port.

After you have entered the above URL in your browser, you will get an error stating that the certificate is self-signed. This is normal, and you can trust the certificate and proceed. Later on, you can configure Webmin to use a custom certificate that you’ve obtained from a certificate authority. (This is discussed later in this guide.)

After you’ve trusted the certificate, you will be taken to the login screen. Here, you can log in with the same username and password you use on your Linux system. You can either log in as root, or use another user who is allowed to use sudo.

Next, you will be taken to the dashboard. This page displays general system information, and you can control various aspects of your server from the various sections on the right.

In the next section, we’ll look at some of the functionality Webmin provides.

Some commonly used Webmin modules

Webmin provides a plethora of functionality in the form of modules. There are modules to manage every aspect of a Linux system — whether it’s updating packages, configuring your firewall or managing log rotation. However, we’re going to only scratch the surface in this guide. We’ll look at some commonly used modules which will help you get started with your Linux server. Some modules, such as those for managing a DNS server are disabled until you install the appropriate software.

Managing packages

You can install and uninstall packages very easily from Webmin. From the left column, select System → Software packages. This will take you to a page that looks like the one below. The exact details shown here may vary, depending on the distribution you’re running Webmin on.

Here, you can search for a package installed on your system, in order to view more information about it or to uninstall it. Also, you can install a package using apt or from a local or remote package file.

Updating packages is quite easy too. If software updates are available, you can click on the “package update is available” notification on the dashboard. This will take you to the “Software Package Updates” page. Alternatively, you can also go to this page by clicking on System → Software Package Updates from the menu on the left.

Here, you can select the individual packages you want to update, and select “Update Selected Packages” as shown. You can also configure automatic updates from the “Scheduled checking options” on this page.

Managing users

You can add or delete users from your system very easily with the user configuration option. From the menu on the left, select Webmin → Webmin Users. To add users, click on “Create a new Webmin user” as shown in the image below.

If you want to remove an user, you should first select the user by clicking on the checkbox and then delete them with the “Delete Selected” button.

File manager

If you want to browse or upload files installed on a remote system, you generally have to use SFTP or rsync. However, these tools aren’t available everywhere. Webmin provides a simple web based file manager, through which you can browse, upload and download files. To open the file manager, select Others → File Manager from the main menu.

To create a new file or to upload a file, you can use the “File” menu on the top right. You can also perform advanced tasks like changing file permissions from the “Tools” menu.

Working with the command line

Although Webmin has a lot of modules, you may sometimes need to fall back to the command line for some tasks. For these situations, Webmin also provides a command-line shell. You can open this from Others → Command Shell. This command line isn’t a full fledged web based terminal emulator, so, interactive programs may fail to run. However, it’s good enough for some quick command-line based tasks.

Configuring Webmin

Webmin offers a wide variety of configuration options. To access these options, click on Webmin → Webmin Configuration from the menu on the left.

On this page, you’ll find a huge number of options you can configure, ranging from security to theming. Again, for the purposes of this guide, we’ll restrict ourselves to discussing a few of the most important ones.

Keeping attackers out

Anything that’s exposed to the internet is bound to attract the attention of hackers, sooner or later. An attacker can try to get into your server by trying out many different usernames and passwords. Fortunately, Webmin comes with built-in tools to prevent this from happening.

If you want to allow or block certain IPs Webmin, you can use the “IP Access Manager”. Depending on whether you want to block or allow IPs, select “Only allow from listed addresses” or “Deny from listed addresses”. Next, in the “Allowed IP address” box, enter the IPs you want to block or allow.

Unfortunately, it isn’t always feasible to use IP based rules. Again, Webmin has options to block IPs with too many failed logins. You can configure these options on the “Authenticator” page. The default is to block IP addresses with 5 failed logins for a minute, but you can increase it to something more effective — such as a 12 hour block (43200 seconds) on 25 failed logins. You can also block an user the same way, by enabling the “Block users with more than…” check box.

If you’re looking for even better security, you can also configure two-factor authentication from the configuration page.

Configuring SSL certificates

As we’ve discussed previously, Webmin uses an auto-generated, self-signed certificate by default. If you’d like to configure Webmin to use a different certificate, head over to the “SSL configuration” section. Here, you can upload your own certificate from the “Upload certificate” tab.

You can also get an SSL certificate from Let’s Encrypt, which is an automated certification authority. It should be noted that you need a hostname to use Let’s Encrypt. In addition, you should also install Apache or another web server. On Debian/Ubuntu, you should install the apache2 package, whereas on CentOS/RHEL you should install the httpd package.

Once you’ve installed the web server, fill in the hostname field. In the “Website root directory” section, select “Other directory” and type in /var/www/html. If you’re already using another web server, you should use the web directory that you’ve configured. Then, set “Months between automatic renewal” to 1 or 2 since Let’s Encrypt certificates are only valid for 3 months. Next, click on “Request Certificate” and it should be issued in a few seconds.

Conclusion

Webmin is one of the best tools for managing your Linux servers. It provides a wide array of useful functionality that both novice and power users will appreciate.

Creating Linux Apps That Run Anywhere with AppImage

Fri, 28 Jul 2017 16:47:07 +0000

Most Linux distributions allow users to install software by using a package manager. Unfortunately, this system leads to quite a few rough edges. If you’re a user using a “long term support” version of a distro, you’re often stuck with older packages. For app developers, it’s even more frustrating – updates often don’t make their way to the distros. If you’re developing a new app, you have to get your package into many different repositories, or you could build packages for the major ones and hope nobody uses one of those rarely used distributions.

To fix these problems, many people have proposed alternative ways to distribute software. In this article, we’re going to look at one of these methods — AppImage. Unlike some systems such as Snap and Flatpak, AppImages work with very little support from the base distribution.

What is an AppImage?

AppImage is a format to distribute an entire application as a single executable file. To run the app, an user simply needs to run this file — there’s no intermediate installation step. The application runs off this package, and it doesn’t place files on the base system. In addition, the AppImage file contains all libraries and files it needs to run, and this allows them to work on a large number of distributions.

So, how does that work? Technically, an AppImage is a ELF file that also contains an embedded filesystem — usually a squashfs filesystem. All the files needed to run the application are stored here. When you run the file, the program embedded in this file mounts the filesystem in a directory under /tmp. Then, it starts up the application inside this directory.

For security reasons, mounting a file system still needs root permissions. Fortunately, most desktop distributions have FUSE support, which makes it possible to mount file systems without root permissions. This is used by AppImage files to work seamlessly, and this is the only bit of support that they expect from the base system.

Because AppImage files are so easy to distribute and run, they’ve been adopted by a large number of projects, such as Subsurface, MuseScore and Avidemux. The AppImage project also has a number of unofficial Appimages available for users to try out.

Now that we’ve learnt the basics, let us take a deep dive into the process of creating an AppImage file.

AppDirs and AppImageTool

In order to build an AppImages, the AppImage project provides a tool named appimagetool. It builds applications from an “AppDir”, which contains the following files:

The files of the original application.
A .desktop file that tells appimagetool about the name of the application, and the icon it should use.
A PNG, SVG or an XPM icon with the name given in the .desktop file with the Icon entry.
An AppRun file, which is used to start up the application inside the filesystem. Once the AppImage ELF has mounted the filesystem, it invokes this file. In the AppRun file, you can run some initialization procedures (such as setting environment variables), and then start up the real application.
Optionally, you should also add AppStream metadata in usr/share/metainfo.

In order to build an AppImage, first you should download appimagetool. If you’re on a x86-64 system, you can grab the latest version like so:

wget https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage
chmod +x appimagetool-x86_64.AppImage

The “Hello World” example

In this section, we’ll learn how to build a basic AppImage with the classic “Hello World” example. So, for this example, our “app” is simply a shell script that prints “Hello World”. Although it’s very basic, this example will help us examine the basic structures necessary to create AppImages.

Go ahead and check out the hello-world-appimage directory for the code. In this repository:

The AppRun file starts up the helloworld.sh file.
helloworld.sh is the actual application that prints “Hello World!”
The helloworld.desktop file sets the name of the application to hello-world-appimage and the icon to hello-world-icon. Notably, other entries found in desktop files (such as Exec) are unnecessary for our purposes, and we’ve left them out.
hello-world-icon.png is the application icon.

Next, download the source code for the “Hello World” application.

git clone https://github.com/boolean-world/appimage-resources
cd appimage-resources

Then, we’ll build the AppImage from the contents of this directory. Replace the path below with the actual location where you’ve saved appimagetool.

/path/to/appimagetool-x86_64.AppImage hello-world-appimage/

You’ll find that an executable named ./hello-world-appimage-x86_64.AppImage has been created. When you run this file, you’ll get the expected output of “Hello World!”.

In the next section, we’ll look at practical examples of building AppImages.

Packaging a real application

The example we saw previously is fairly simple. However, packaging a real application isn’t that simple. There are a number of factors you should take into account when building an AppImage for something more practical.

For example, on other OSes such as Windows and Linux, there’s a notion of a “base system” — an installation of these OSes provides high level functionality by default, such as GUI and media handling libraries. On the other hand, Linux systems don’t share this concept. An installation of Linux might just be the kernel, an init system and a shell.

You probably wouldn’t want to support something that’s so minimal, though. For example, for packaging a GUI application, you might take a very old distribution that’s still supported. Then, you can install a very minimal desktop environment/window manager and define that as the base system to target.

In addition, running an application from inside an AppImage is a lot different than when it’s installed on your system. For example, a GUI torrent client might depend on a torrent library, which is typically installed in /usr/lib. Once it’s run from an AppImage, the torrent client will fail to run because it’ll still look for it in /usr/lib. In this case, you should export LD_LIBRARY_PATH in AppRun before starting up your application. Alternatively, you can also use chrpath/ patchelf to change the rpath of the binaries.

Depending on what you’re working with, you might have to do a number of additional things. As an example, when you’re working with Qt applications, you should export QT_QPA_PLATFORM_PLUGIN_PATH so that Qt plugins work properly.

A real example: packaging qBittorrent

In order to better understand the steps we’ve previously mentioned, we’ll package qBittorrent into an AppImage. One of the quickest ways to create it would be to use the files from a pre-built package. However, in this article we’ll use the other approach: we’ll compile qBittorrent and pack up the needed files.

The steps we’ve discussed below might seem fairly tedious and daunting. However, these are just the steps you need to run to figure out what should go into the AppImage. Once you’ve completed this process, you can automate it, and you only need to check back again when something major changes or when new dependencies are introduced.

Setting up the base system

As we’ve said previously, it’s best to choose an old distro with a minimal configuration. In this example, we’ve use Ubuntu 14.04 with a minimal LXDE installation as our base system.

To make our analysis easier, we’ll want to keep track of all the different files and libraries installed during compilation. In addition, we also want to keep our base installation clean. Thus, we need a mechanism to isolate writes to a separate directory. This can be usually done by coupling a chroot or systemd-nspawn to an union file system. A very crude script that uses chroot is available here, and we’ll use this for our example.

Before we can start working in the “isolated-writes” environment, we’ll need to initialize a minimal set of devices and directories. To do so, run:

sudo ./tempenv.sh setup

Then, we can start up this environment with:

sudo ./tempenv.sh start

This will mount the union file system and the minimal set of devices, and then chroot into it. Here, we’ll run the commands to install additional dependencies, such as Qt (needed by qBittorrent).

To ensure too much stuff isn’t inside this environment, we’ll install compilation tools directly on the base system:

sudo apt install gcc g++ automake libtool pkg-config

Installing packages and compiling qBittorrent

After you’ve started up the environment, you can install packages in it. We’ll use the instructions outlined here, and use Stephan Binner’s QT 5.8 PPA.

add-apt-repository ppa:beineri/opt-qt58-trusty
apt update
apt install qt58base qt58tools libboost-dev libboost-system-dev libboost-chrono-dev libboost-random-dev libssl-dev libgeoip-dev libgl1-mesa-dev

Once we’re done with installing various packages, we’ll proceed to compile and install qBitTorrent. Here, we’ve assumed you have the libtorrent (the 1.0 branch) and qBitTorrent (3.3.13) sources under your home directory.

. /opt/qt58/bin/qt58-env.sh
# Install libtorrent
cd ~/libtorrent
./autotool.sh
./configure --disable-debug --enable-encryption --prefix=/usr --with-libgeoip=system
make install
# Install qBittorrent
cd ../qBittorrent
./configure --prefix=/usr
make install

We’re done compiling and installing qBittorrent, so we’ll exit the shell here.

Cutting the fat

As you may have noticed, the writes into our “environment” have gone into the tempenv/files directory. If you look into its contents, you’ll find a couple of directories and files. This directory is the starting point for our AppDir. However, we can’t package this directory right now — there’s a lot of stuff we have to remove and add in before we can do that.

Before we do any of that, we’ll copy this directory to somewhere else, and then we’ll move in to the new directory.

sudo cp -r tempenv/files/ ~/qbittorrent.AppDir
cd ~/qbittorrent.AppDir

Despite the fact that there are a huge number of files, we can eliminate some of them through intelligent guesses. For example, we don’t need the /var, /home or /tmp directories, so we can delete them. Again, we don’t need developer documentation, tooling and libraries, so we could delete those too. Installing Qt also gives us a number of files unnecessary for our purposes, and again, they can be deleted.

sudo rm -r .unionfs* etc/ home/ var/ tmp/ opt/qt58/bin/ opt/qt58/doc/ \
 opt/qt58/include/ opt/qt58/mkspecs/ usr/include/ \
 usr/lib/pkgconfig/ usr/share/sgml/ usr/share/aclocal/ \
 usr/share/bug/ usr/share/doc/ usr/share/lintian/ \
 usr/share/man/ usr/share/pixmaps/opt-qt58-* \
 usr/share/pkgconfig/ usr/share/freetds/ opt/qt58/examples/ \
 usr/lib/geoip/ usr/share/applications/opt-qt58-* \
 usr/bin/geoiplookup* usr/share/applications/mimeinfo.cache
sudo find -type f -name '*.la' -o -name '*.a' -o -name '*.prl' | sudo xargs rm

We’ve managed to trim things down by a huge margin. Unfortunately, there are quite a few files and libraries left, but we can’t decide about them directly. We’ll deal with this in the next section.

Including and removing more files by analyzing binaries

Linux distributions are configured in a great variety of ways. Despite the fact that we’ve taken a minimal install, there may still be some dependencies which won’t get included in the AppDir. Thus, the AppImage might fail on an exotic configuration. Again, as we’ve discussed previously, there are some files in the AppDir which aren’t needed. In order to find out which libraries are actually needed, we’ll ldd tool:

LD_LIBRARY_PATH="opt/qt58/lib:usr/lib:usr/lib/x86_64-linux-gnu" ldd usr/bin/qbittorrent

This gives a number of libraries we can copy into the usr/lib directory. Unfortunately, ldd can’t detect every library that a program needs. Moreover, it doesn’t say anything about the other files that may or may not be needed. For these situations, we’ll try to run the application with strace. Files not referenced when the program is running can be safely removed.

export QT_QPA_PLATFORM_PLUGIN_PATH=opt/qt58/plugins/
LD_LIBRARY_PATH="opt/qt58/lib:usr/lib:usr/lib/x86_64-linux-gnu" strace usr/bin/qbittorrent 2>&1

Since there’s quite a bit of output from strace, you could save this to a file. So, for example, if you were wondering if any of the plugins from opt/qt58/plugins were actually used by the application, you could grep this file appropriately.

You should consider running the application this way on a number of popular distributions. This will allow you to verify if things work well on other distributions as well.

After removing and adding files in this way, we ended up with a set of files which we’ve listed here.

Adding icons, desktop entries, metadata and an AppRun

Before we can create the AppImage, there are some small tasks left.

First of all, we have to add a .desktop file and an icon. qBittorrent already provides them, and we have to just copy them into the root directory:

sudo cp usr/share/applications/qbittorrent.desktop ./
sudo cp usr/share/hicolor/128x128/apps/qbittorrent.png ./

Then, we’ll add the AppStream metadata for the application. Again, qBittorrent already provides this, but it’s in usr/share/appdata instead of usr/share/metainfo. You can move or symlink the directory to the correct location:

sudo mv usr/share/appdata/ usr/share/metainfo

Finally, we need an AppRun. This is fairly easy, and we’ve already seen a minimal version of it in the previous section. This is the AppRun we’ll add in. Most importantly, it uses absolute paths instead of relative ones. It also handles cases where there may already be an existing value of LD_LIBRARY_PATH.

Building the AppImage

Finally, it’s time to build the AppImage! Again, as we previously saw in the “Hello World” example, this is extremely easy to do:

cd ..
/path/to/appimagetool-x86_64.AppImage qbittorrent.AppDir

Finally, we’ve obtained an AppImage of qbittorrent. It works across many distributions in the exact same manner. In our case, we executed it on our base system (Lubuntu 14.04) as well as on Linux Mint and Manjaro XFCE, and it works perfectly across all of them.

Desktop integration and updates

So far, we’ve seen how AppImages provide cross-distribution compatibility. Unfortunately, this is also their biggest weakness over a traditional package manager based approach. Because an AppImage is self contained, it can’t provide desktop integration through desktop entries. Also, a vanilla AppImage doesn’t provide a nice way to update it.

The developers of AppImage have noticed this, and they’ve come up with some solutions. The project has a number of recipes for building AppImages, and they use a “desktop integration wrapper script”. Simply put, the AppRun starts up a script instead of the real application. This script then asks if the user would like to create a desktop entry. If the user agrees, the script puts a modified version of the embedded desktop entry into ~/.local/share/applications. Finally, the script starts up the real application.

Alternatively, users can install appimaged, an optional daemon that watches the user’s downloads directory. When the user downloads a new AppImage here, it automatically registers it with the system.

For updates, there’s AppImageUpdate — it’s an external program that updates AppImages from the metadata embedded in them. It’s a completely decentralized way of distributing updates, and it uses delta updates to minimize bandwidth usage.

Conclusion

In this article, we’ve learnt what an AppImage is, how they aim to solve problems in Linux software distribution, and how you can build AppImages for your own applications. AppImages provide a nice user experience, and they bring Linux closer to MacOS and Windows in terms of ease-of-use.

How to Install Nextcloud on Ubuntu

Wed, 19 Jul 2017 04:15:19 +0000

There are many online file storage services such as Google Drive and Dropbox. However, these are run by third parties, which means that you often don’t have control over the data you’ve uploaded. Fortunately, there are many privacy-aware, open-source alternatives that you can host on your own server. One of them is Nextcloud, a PHP-backed application that allows you to access files over a web interface and WebDAV.

In this article, we’ll learn how to install Nextcloud on a server running Ubuntu.

Installing Nextcloud via Snap

Installing Nextcloud via Snap is extremely easy. Snap is a new packaging technology that allows developers to ship software with multiple dependencies as a single package. Unfortunately, this method uses up port 80 (or 443 when using HTTPS) and you can’t change this. If you need to host multiple web applications at the same time on ports 80 or 443, use the manual installation method outlined below.

In the latest version of Ubuntu, Snap is installed by default. However, if it is not installed, run the following command:

sudo apt install snapd

The Snap package comes with its own, bundled version of Apache which runs on port 80. If you have an existing web server, you should first remove it before continuing.

Next, run the following command:

sudo snap install nextcloud

After installation has been completed, you may visit localhost in your browser. On the web page that opens, fill in the username and password to set up an administrator account.

Now, you’ll be logged in to Nextcloud. You can now upload files to your Nextcloud installation and change its settings from the admin panel.

The Snap package also include a number of CLI utilities. These commands need root permissions to work, so you should add sudo before each command when you run them.

With nextcloud.enable-https, you can configure the built-in Apache server to use HTTPS. You can use a custom certificate, or it can obtain one from Let’s Encrypt. To disable HTTPS, you should run nextcloud.disable-https.
nextcloud.mysqldump and nextcloud.mysql-client allows you to interact with the built-in MySQL server. You can retrieve and change the data stored in the database with these commands.
nextcloud.occ is the Snap version of Nextcloud’s occ tool. You can manage users and settings with this tool.

Manually installing Nextcloud

Setting up the prerequisites

Here, we will install Nextcloud along with Apache and MySQL manually. You can install each of these components separately, or you can use the lamp-server task to install all of them at one go:

sudo apt update
sudo apt install -y lamp-server^ php-zip php-mbstring php-dom php-gd php-curl

After the installation completes, you should enable the rewrite and headers modules. This is necessary for Nextcloud to work correctly.

sudo a2enmod rewrite headers

Then restart Apache for the changes to take place:

sudo systemctl restart apache2

Configuring MySQL

The next step is to configure MySQL. First, you should log into MySQL as the root user.

sudo mysql -u root

When you run the above command, the MySQL monitor should open, like so:

Now, you have to create a new database in which Nextcloud will store information. This can be done with:

CREATE DATABASE nextcloud;

Now, you should also create a new MySQL user which Nextcloud will use to access the database. We’ll name the user nextcloud and assign it a password of superstrongpassword:

CREATE USER nextcloud@localhost IDENTIFIED BY 'superstrongpassword';

Then, you should grant the newly made account with all permissions to the database:

GRANT ALL PRIVILEGES ON nextcloud . * TO nextcloud@localhost;

Keep a note of the username and password that you’ve used here — you’ll need them in the next section.

Install Nextcloud

Next, head over to the Nextcloud downloads page and download the latest ZIP file with wget or curl. Depending on the current version, the URL will vary. Here, we’ve downloaded the ZIP file for version 12, which is the latest release at the time of this writing.

wget https://download.nextcloud.com/server/releases/nextcloud-19.0.0.zip

Then, unzip the downloaded file into your web directory. Typically, this would be in /var/www/html, although you can provide another directory if your web directory is located elsewhere.

sudo unzip -d /var/www/html nextcloud-19.0.0.zip

This will create a directory named nextcloud inside /var/www/html. This will make Nextcloud available under http://<your_server_name>/nextcloud. If you wish to make this directory available elsewhere, you can use the mv command to rename the nextcloud directory.

After extracting the file, you have to set the owner of the extracted files to the Apache user, www-data:

sudo chown -R www-data: /var/www/html/nextcloud

After you have completed the above steps without any issues, open your browser and type http://<your_server_name>/nextcloud in the address field. You should see a setup screen like the one below.

In the username and password fields at the top, you should fill in the details of the Nextcloud administrator account that will be created. You can manage various aspects of Nextcloud from this account.

The data folder setting controls the folder in which user’s files are saved in. You can leave this setting as-is.

Next, fill in the details for the database. In our case, the database user and name will be nextcloud, and the password will be superstrongpassword.

Now, scroll down and click on the “Finish setup” button to complete installation. You’ll be taken to the recommended apps page, where Nextcloud will take a few minutes to install apps like a calendar and contacts app. This will happen automatically without you having to do anything.

Once the installation is complete, you will be taken to the files page, where you will see a message about “NextCloud Hub” like so. Dismiss it by clicking on the “Close” button at the top, or by clicking outside the Nextcloud Hub message.

Finally, you will see the “files” page, where you can upload, delete, and modify files:

If you’ve run all these steps without problems, congratulations! Nextcloud has been successfully installed on your server. We recommend you to go through the Nextcloud Manual, which explains all the functionality available.

Conclusion

Nextcloud is a great alternative to proprietary services such as Google Drive and Dropbox. It also provides a set of apps which you can install to extend its functionality in various ways. There are also apps for iOS and Android which you can use to access your files on the go.

Now that you’ve set up Nextcloud, why not configure HTTPS so that your files are secure while in transit? Check out our guides on configuring HTTPS with Cloudflare and Let’s Encrypt to find out more.

An Introduction to the Docker Ecosystem

Thu, 06 Jul 2017 05:35:32 +0000

Traditionally, developing an application and deploying it in a production environment hasn’t been easy. Development and production environments are often configured differently, which makes it difficult to resolve problems quickly when something was broken. A popular tool named Docker solves this problem by allowing developers to package their applications in a portable way without all of the hassles of a traditional system.

In this article, we’re going to look at the basic concepts of Docker: the way it works, the general workflow when using Docker, and the related terminology.

Hypervisors vs containerization

Development and production environments often differ in configuration and the software installed. In addition, different applications may have conflicting software requirements. These problems make deploying applications rather difficult. Developers did have a solution that somewhat eased the problem: a software that emulates a full computer, otherwise known as a “hypervisor”.

A hypervisor allows you to create a virtual machine(VM) into which you can install an operating system. You can install the necessary versions of software in them, and then you can run your application in it. In this way, all applications are isolated from one another. This eliminates conflicts and adds a level of fault tolerance: a bug in an application can, at most, affect the virtual machine it’s running on. Multiple applications can be run this way, and individual VMs can be packaged and deployed on other systems.

The diagram below shows multiple applications running on a single host in this way:

Unfortunately, emulating hardware takes up significant resources, making this method slow, inefficient and power hungry. In addition, a separate OS is installed inside each VM, which adds to this overhead. Fortunately, a lightweight form of virtualization, called “operating system level virtualization” or “containerization” solves this problem.

OS-level virtualization allows creating “containers” inside which applications can be run. The resources available to the base operating system are shared to applications running inside these containers. However, these applications can’t affect others that are running on the base system or on other containers. Implementations such as Docker also allow you to package these containers and run them on other systems. Thus, these systems provide the same benefits of virtualization while being very light on resources.

The diagram below illustrates how applications run inside containers:

Although Docker has made containerization popular, various implementations have existed before it, although their goals have been usually different. For example, OpenVZ(Linux) allows for provisioning containers as virtual servers, while Sandboxie(Windows) allows you to create throwaway containers in which unsafe files can be opened.

Images

In Docker terminology, an “image” is a package that contains an application. Typically, an image includes executable files and shared libraries — all that’s needed for the application to run. This approach makes applications portable — they can be run on any system without needing to know anything about the application itself.

Typically, an image is created from a “Dockerfile”, which contains the steps needed to create an image. Each instruction in this file creates a “layer” in the file. When you change the Dockerfile and rebuild the image, only the changed layers are rebuilt, thus minimizing disk usage. Images can be also be created by persisting changes made to a container (discussed in the next section).

An image can also depend upon other images. As an example, a web application written in Python may depend upon an image containing a Python interpreter.

For easy distribution, images are stored in a registry. Once an image is “pushed” to it, you simply need to “pull” it on another system. The most popular public registry is Docker Hub, and there are many services which provide private registries. In addition, Docker has built-in functionality to run your own registry. If you don’t want to use a registry, you can “save” images as a tar archive and “import” them on another system.

Containers

An image is simply that — a packaged application. To run the application, you need to create a container. You can think of a container as the runnable instance of an image, with its own persistent file system and network access added to it. The container is completely isolated from the base system and the rest of the containers. However, you can configure it to access files or share ports from the base system if you need to.

After creating a container, you can “start” them. This executes the default command that’s been specified in the Dockerfile. However, you can also start up any other command. For example, you might want to inspect some files on an Ubuntu-based nginx container by running the bash shell in the image.

To make things easier, you can also “run” images directly, and Docker will automatically create a container for you. You can also stop, re-run and delete containers as and when you need.

When Docker attaches a persistent file system to the container, this also creates a layer. This forms the basis for another way of creating images. So, for example, you can take a base Ubuntu image, install your application in it, and then create an image off it. Docker takes the layers of the base image and that of the container to create a new image.

Services and orchestration

An application is usually dependent on a number of different components. For example, a video sharing application may be written in Python, which runs as a daemon. It persists data in a database server and depends on a component that transcodes video. When designing an application to run within Docker, it’s best to split up these components into individual containers. This form of design is often called “service oriented architecture”.

Separating services this way allows a specific component to be scaled up as per the requirements. Services built in this way should also preferably be independent of the host system. In addition, they should also provide consistent APIs for other services to use. This allows individual components to be changed, as long as the API through which they interact is kept the same.

A reasonably sized organization usually has multiple physical servers to run their applications. Docker allows you to scale services across multiple hosts through “orchestration”. It consists of a “scheduler” that selects nodes to run services. It also provides functionality to manage these clusters.

The Docker Engine provides a built-in “swarm mode” for this purpose, which we have described in the next section. In addition, there are other advanced orchestration tools, such as marathon and Kubernetes.

Docker’s swarm mode

Docker’s swarm mode consists of nodes participating in a cluster. A node is an instance of the Docker engine; thus, a single system can run one or more nodes. In most production environments, nodes are spread across many different systems.

These nodes are categorized into workers and managers. To deploy applications in a swarm, you submit a service definition to a manager. The manager node then dispatches various tasks out to workers. By default, services are load balanced among the different workers. You can also tell the manager to maintain a number of replicas across all the nodes available, or to run a single instance of the service on every node.

Despite its distributed nature, the service appears to be a single application to the consumer.

Networking and communication in Docker

A service oriented architecture makes communication necessary among the different services for coordination. Unlike some other containerization mechanisms, Docker sets up networking automatically. Specifically, it creates a network interface named docker0 on the host system, which allows it to create a virtual subnet from which IP addresses are allocated to individual containers. It also sets up the proper iptables rules so that inbound and outbound traffic are properly routed.

Many applications listen for inbound connections, and you can “expose” ports inside a container. You can also allow Docker to “publish” a port, which maps it to the host interface and makes it available to the outside world.

Docker also provides another mechanism known as “links” for configuring communication between containers. You simply need to specify the names of available services. Docker contains an embedded DNS server, and whenever an application in a container refers to the service through its name, it will be automatically resolved to the IP address at which it is available.

Service discovery

Apart from the concept of links we saw earlier, there are a variety of tools that allow for service discovery. Service discovery is a strategy that makes it easy to perform deployments. It does this by allowing containers to figure out details of the other services they need without any manual configuration by an administrator. Each service also registers itself to the discovery mechanism, so that other applications can find out about them.

The diagram below shows an example of an application finding its dependency through a discovery mechanism. App 1 registers itself with a discovery service on startup. App 2 has a dependency on App 1, so it first tries to locate App 1. When the discovery service replies with the details, it proceeds to connect to App 1.

These discovery services are usually simple key-value based stores with an HTTP API to get and set values. These stores are also used to store configuration data for various services.

Some popular service discovery tools used with Docker are Consul, etcd and ZooKeeper.

Conclusion

Docker is a popular containerization technology that makes it easy to deploy applications. In this article, we’ve looked at a number of core concepts associated with Docker, and how services are deployed and run. More practical aspects of Docker such as creating images, managing containers and using registries will be explored in a later article.

Serverless Computing, Explained

Tue, 27 Jun 2017 00:00:00 +0000

The world is changing dynamically at a fast pace. The thing that is changing even faster is technology and the underlying architecture beneath it. In this article, we will dig into the depths of the new buzzword “Serverless Architecture”. We’ll find out what is really is, its advantages and disadvantages and whether you should consider moving your stack to serverless technology.

Drawbacks of traditional architecture

Before introducing any formal definitions to serverless architecture, we will look into the existing traditional architecture and find out its drawbacks.

Conventionally, applications were written and run in servers which are allocated fixed resources. Soon problems arose with sudden spikes of traffic as demands increased and the servers were not able to handle the enormous amount requests. To address these problems, came Platform as a Service (PaaS) in which providers offered scaling but it has its drawbacks compared to serverless architecture which we will discuss later in this article.

In a typical stack, most of the time the CPU remains idle, because the server is busy with networking and I/O requests. These drawbacks were handled by serverless architecture as we will see in the following sections.

What is serverless architecture?

Now you may be wondering whether the term " Serverless" means there are no servers involved. That is certainly not true. Serverless does involve servers, only the implementation and management is different. As discussed in the above section, traditional servers/compute instances have fixed resources, run all the time and need administrators to manage them. Serverless architectures , on the other hand, is run in compute instances that are event driven. What it basically means is that for each event or request to the server, a state is created and after the request is served, the state is automatically destroyed.

Unlike typical servers, serverless provisions are completely managed by the provider using automated systems that eliminate the need for server administrators. In serverless architecture, short-lived functions are invoked upon each request and the provider bills the client for running each individual function or business logic. Thus came the name Functions as a Service(FaaS). In most of the article, we will be referring serverless architecture as simply FaaS.

According to Wikipedia, the definition of serverless computing is:

Serverless Computing is a cloud computing execution model in which the cloud provider dynamically manages the allocation of machine resources, and bills based on the actual amount of resources consumed by an application, rather than billing based on pre-purchased units of capacity.

Some of the popular FaaS services are AWS Lambda, Azure Functions, Google Cloud Functions, IBM Bluemix OpenWhisk, webtask.io and hook.io.

FaaS vs Managed Servers

FaaS provisions allow running code without the need to manage the server system and server applications. So, what’s the difference between 3rd party managed servers and Faas? Both seem the same as you do not need to manage the server systems in either of them. The fundamental difference is that in FaaS, you also do not need to manage server applications which were not possible in typical systems.

Let’s take an example of a social media application to understand this. As users interact with the site, the application collects behavioural data and processes them in a dedicated server to learn more about the users and provide relevant content. Now, that server is only dedicated to machine learn those behavioural patterns. In the case of FaaS, it is replaced with functions that do not require that physical server to run and can be executed among other functions in a shared pool of resources.

There is another advantage to the FaaS architecture. You do not need to code your app to any particular framework or library. Any language can be used. What’s even more interesting is that two FaaS functions written in different languages can interact with each other. Multiple functions can be connected and chained together to implement reusable and composable systems.

Deployment and Scaling

Deploying FaaS applications is very easy. You need to upload the code to the provider in a compressed or executable format and it takes care of everything else. Later, to update the code, you just need to upload the newer version and initiate the update by calling an API.

Let’s come back to our social media application. There may be normal days where is traffic is average. But imagine a national holiday where the number of users may peak suddenly. This is very common in the case of e-commerce websites where the traffic may surge enormously in the event of a sale like Black Friday. In those cases, the server will not be able to serve so many requests and eventually crash. This can only be solved through scaling the server resources.

Perhaps the biggest benefit of FaaS is automatic scaling. As discussed earlier PaaS also provides scaling but the problem is that you need to estimate how much resources that you need and then provision them accordingly. This has problems since you may always over-estimate or under-estimate. With FaaS, you don’t even have to think about scaling. Just upload your code and forget the rest. Automatic horizontal scaling is completely managed by the provider and is totally elastic. If there’s a sudden spike in requests, the FaaS provider will handle it seamlessly without any extra configuration or management. So you can clearly see that there is no over or under estimation and accordingly cost benefits are huge.

In Backend as a Service (BaaS) which integrates into FaaS architecture, entire application components are provided as a service. For example, there are services like Kinvey which provides backend services like data storage, push notifications, analytics and more so that developers do not have to code their components. They just need to integrate those ready-made services in their apps. It saves development time and testing time and also reduces development costs.

The same benefits hold true against container technologies like Docker or Kubernetes. Container platforms do not offer automatic and transparent scaling solutions as of yet and serverless FaaS wins here too.

The problem with states

FaaS architecture is designed in such a way that individual functions are created upon each request and is destroyed after while. The state is not maintained only for a short while and then destroyed completely. So subsequent functions can not access the state and its data. So, FaaS functions are also called stateless functions. Now, this can be a problem as application architecture needs to be redesigned. Stateless functions can work for most of the use cases but not for all. Currently, it is handled by storing the state across multiple requests either with in-memory databases like Redis or simple object storages. But that is always slower than preserving states in cache or RAM in traditional architecture.

Recently Amazon announced a new service called AWS Step Functions. It comprises a state machine that provides a semi-persistent state that can be shared among multiple functions. You can learn more about AWS Step functions at Cloudacademy blog article.

Startup and Execution

Cold starting a function in serverless platform takes a considerable amount of time to load which can range from about a couple of milliseconds all the way up to a few minutes. This is certainly bad but happens only in a few set of conditions where the functions are accessed very infrequently. It can be overcome by a process called warming. A scheduled event is run for a fixed time interval (say every 5 minutes) and it invokes the functions so that it can trigger quickly in case of a real event.

Now let’s talk about execution. FaaS functions have a time limit till which they can be run. If they exceed it, they will be automatically killed. So long-lived processes cannot be created without changing the architecture of the application. The apps must be redesigned in which there are multiple coordinated functions that perform the task equivalent to a traditional long-lived function.

Efficiency

Most of the commercial servers that are used today in production environments are not utilized to their maximum extent. As a result, tonnes of kilowatts of power is wasted while the servers sit idle. This misutilization will come to an end once FaaS becomes mainstream. Serverless providers have large pools of servers in which billions of functions run parallel in a shared set of resources. So multiple clients can use the same hardware without any noticeable difference than that of a dedicated server.

The concept is explained in the following diagrams. Different applications have their dedicated servers in traditional architecture.

In the case of a serverless architecture, each server runs various functions of several apps. In the diagram below you can see one application has various functions and they can be run parallel on multiple servers. So, multiple functions of a single app are running simultaneously. Therefore, resource utilization is much more efficient.

It even pays off by writing more efficient code which consumes fewer resources and runs faster since the provider bills the client based on the execution time of each function and the amount of resources it uses.

Vendor lock-in and control

Unfortunately, FaaS deployments also come with a major disadvantage: vendor-lock in. Different providers have different implementations of their serverless architectures. When you need to shift from one provider to another, you’ll have to port your code to the other provider, which could prove to be expensive. Most of the management is done by the provider which means you have very little control left over your system. There may also be security concerns in some cases where highly confidential information like financial transactions is involved.

Resources

There are a couple of good resources for developing FaaS applications. You should check out the Serverless Framework which is a great open-source CLI tool for building and deploying apps over various providers listed above.

The awesome-serverless GitHub repository is a nice resource which consists of everything like projects, blog posts, talks, plugins, services and much more all related to serverless architecture. It is a curated list of all the resources available which you should definitely have a look at.

We’ll now summarize the advantages and disadvantages of serverless computing.

Advantages

Automatic Scaling: FaaS provides automatic horizontal scaling and handles sudden spikes easily.
Reducing costs: Everything in serverless architecture is completely managed by the provider itself, eliminating the needs for system administration.
Simple deployment: FaaS apps are very simple to deploy and update. All you need to do is to provide the compiled code and the platform takes care of the rest.

Disadvantages

Startup and execution time: As discussed above startup and execution time of programs can be a problem in certain scenarios.
State: Architectural redesign of code is necessary as states are not preserved.
Vendor lock-in and control: Since FaaS deployments are highly coupled to the provider, you have very little control over your application and you’re usually locked in to a single provider.

Conclusion

It’s important to remember that Serverless Computing is very new and not yet mature. There are some use cases where the traditional architecture still fits best. If you think to move over to serverless, you should test your apps for FaaS first and make sure everything works perfectly. FaaS is a great revolution in cloud technology and should stabilize in the coming years.

An In-Depth Guide to iptables, the Linux Firewall

Sun, 18 Jun 2017 10:48:10 +0000

The Linux kernel comes with a packet filtering framework named netfilter. It allows you to allow, drop and modify traffic leaving in and out of a system. A tool, iptables builds upon this functionality to provide a powerful firewall, which you can configure by adding rules. In addition, other programs such as fail2ban also use iptables to block attackers.

In this article, we’re going to take a look at how iptables works. We’re also going to look at a few examples, which will help you write your own rules.

How does iptables work?

iptables is just a command-line interface to the packet filtering functionality in netfilter. However, to keep this article simple, we won’t make a distinction between iptables and netfilter in this article, and simply refer to the entire thing as “iptables”.

The packet filtering mechanism provided by iptables is organized into three different kinds of structures: tables, chains and targets. Simply put, a table is something that allows you to process packets in specific ways. The default table is the filter table, although there are other tables too.

Again, these tables have chains attached to them. These chains allow you to inspect traffic at various points, such as when they just arrive on the network interface or just before they’re handed over to a process. You can add rules to them match specific packets — such as TCP packets going to port 80 — and associate it with a target. A target decides the fate of a packet, such as allowing or rejecting it.

When a packet arrives (or leaves, depending on the chain), iptables matches it against rules in these chains one-by-one. When it finds a match, it jumps onto the target and performs the action associated with it. If it doesn’t find a match with any of the rules, it simply does what the default policy of the chain tells it to. The default policy is also a target. By default, all chains have a default policy of allowing packets.

Now, we’re going to take a deeper look into each of these structures.

Tables

As we’ve mentioned previously, tables allow you to do very specific things with packets. On a modern Linux distributions, there are four tables:

The filter table: This is the default and perhaps the most widely used table. It is used to make decisions about whether a packet should be allowed to reach its destination.
The mangle table: This table allows you to alter packet headers in various ways, such as changing TTL values.
The nat table: This table allows you to route packets to different hosts on NAT (Network Address Translation) networks by changing the source and destination addresses of packets. It is often used to allow access to services that can’t be accessed directly, because they’re on a NAT network.
The raw table: iptables is a stateful firewall, which means that packets are inspected with respect to their “state”. (For example, a packet could be part of a new connection, or it could be part of an existing connection.) The raw table allows you to work with packets before the kernel starts tracking its state. In addition, you can also exempt certain packets from the state-tracking machinery.

In addition, some kernels also have a security table. It is used by SELinux to implement policies based on SELinux security contexts.

Chains

Now, each of these tables are composed of a few default chains. These chains allow you to filter packets at various points. The list of chains iptables provides are:

The PREROUTING chain: Rules in this chain apply to packets as they just arrive on the network interface. This chain is present in the nat, mangle and raw tables.
The INPUT chain: Rules in this chain apply to packets just before they’re given to a local process. This chain is present in the mangle and filter tables.
The OUTPUT chain: The rules here apply to packets just after they’ve been produced by a process. This chain is present in the raw, mangle, nat and filter tables.
The FORWARD chain: The rules here apply to any packets that are routed through the current host. This chain is only present in the mangle and filter tables.
The POSTROUTING chain: The rules in this chain apply to packets as they just leave the network interface. This chain is present in the nat and mangle tables.

The diagram below shows the flow of packets through the chains in various tables:

Targets

As we’ve mentioned before, chains allow you to filter traffic by adding rules to them. So for example, you could add a rule on the filter table’s INPUT chain to match traffic on port 22. But what would you do after matching them? That’s what targets are for — they decide the fate of a packet.

Some targets are terminating, which means that they decide the matched packet’s fate immediately. The packet won’t be matched against any other rules. The most commonly used terminating targets are:

ACCEPT: This causes iptables to accept the packet.
DROP: iptables drops the packet. To anyone trying to connect to your system, it would appear like the system didn’t even exist.
REJECT: iptables “rejects” the packet. It sends a “connection reset” packet in case of TCP, or a “destination host unreachable” packet in case of UDP or ICMP.

On the other hand, there are non-terminating targets, which keep matching other rules even if a match was found. An example of this is the built-in LOG target. When a matching packet is received, it logs about it in the kernel logs. However, iptables keeps matching it with rest of the rules too.

Sometimes, you may have a complex set of rules to execute once you’ve matched a packet. To simplify things, you can create a custom chain. Then, you can jump to this chain from one of the custom chains.

Now that we know about the theory behind iptables, we’ll look at some examples.

A note regarding iptables commands

There are two versions of the Internet Protocol — IPv4 and IPv6. These protocols have some differences and are handled differently in the kernel. Thus, iptables provides different commands for these protocols — iptables for IPv4 and ip6tables for IPv6.

However, the options accepted by these commands doesn’t vary too much. We’ve discussed these differences later in the appropriate sections in this article.

You also need to execute all iptables commands as root. You can launch a root shell by typing in su -c and then typing in your root password and then run the commands in this article. Alternatively, you can add sudo in front of every iptables command.

Blocking IPs

The most common use for a firewall is to block IPs. Say for example, you’ve noticed the IP 59.45.175.62 continuously trying to attack your server, and you’d like to block it. We need to simply block all incoming packets from this IP. So, we need to add this rule to the INPUT chain of the filter table. You can do so with:

iptables -t filter -A INPUT -s 59.45.175.62 -j REJECT

Let us break that down. The -t switch specifies the table in which our rule would go into — in our case, it’s the filter table. The -A switch tells iptables to “append” it to the list of existing rules in the INPUT chain. However, if this is the first time you’re working with iptables, there won’t be any other rules, and this will be the first one.

As you might have guessed, the -s switch simply sets the source IP that should be blocked. Finally, the -j switch tells iptables to “reject” traffic by using the REJECT t arget. If you want iptables to not respond at all, you can use the DROP target instead.

Previously, we’ve mentioned that the filter table is used by default. So you can leave it out, which saves you some typing:

iptables -A INPUT -s 59.45.175.62 -j REJECT

You can also block IP ranges by using the CIDR notation. If you want to block all IPs ranging from 59.145.175.0 to 59.145.175.255, you can do so with:

iptables -A INPUT -s 59.45.175.0/24 -j REJECT

If you want to block output traffic to an IP, you should use the OUTPUT chain and the -d flag to specify the destination IP:

iptables -A OUTPUT -d 31.13.78.35 -j DROP

Listing rules

Now, say that we’ve blocked a couple of IPs by appending rules. If you want to see these rules later, you can use the -L switch. Also, as we will see in the next section, it’s very helpful to see line numbers for these rules, so we’ll also use the --line-numbers switch.

iptables -L --line-numbers

Running this command gives us a list of rules like the ones below. This list is also from the filter table, and you can list other tables with the -t switch.

Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 DROP all -- 59.45.175.0/24 anywhere
2 DROP all -- 221.194.47.0/24 anywhere
3 DROP all -- 91.197.232.104/29 anywhere
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 DROP all -- anywhere 31.13.78.0/24

In our example, we’ve blocked three IP ranges in the INPUT chain and one in the OUTPUT chain. There are no rules in the FORWARD chain.

iptables also tries to be helpful by doing reverse DNS lookups on the IPs. Often, this is unnecessary and slows down the listing process. To disable this, you can use the -n switch:

iptables -L -n --line-numbers

Deleting rules

Now, say you’ve blocked the IP range 221.194.47.0/24 by mistake. Removing it is easy: simply replace -A with -D, which deletes a rule:

iptables -D INPUT -s 221.194.47.0/24 -j REJECT

You can also delete rules through their line numbers. If you want to delete the second rule from the INPUT chain, the command would be:

iptables -D INPUT 2

When you delete a rule that isn’t the last rule, the line numbers change, so you might end up deleting the wrong rules! So, if you’re deleting a bunch of rules, you should first delete the ones with the highest line numbers. If you were deleting the 9th and 12th rules from the INPUT chain, you would run:

iptables -D INPUT 12
iptables -D INPUT 9

Sometimes, you may need to remove all rules in a particular chain. Deleting them one by one isn’t practical, so there’s the -F switch which “flushes” a chain. For example, if you want to flush the filter table’s INPUT chain, you would run:

iptables -F INPUT

Inserting and replacing rules

So far, we have seen examples of appending and deleting rules. It turns out that you can also insert rules at a given position! This is useful in a number of cases. We’ll use the previous example in the Listing rules section. While you’re seeing attacks from 59.45.175.0/24, assume that you need to whitelist 59.45.175.10.

Since iptables evaluates rules in the chains one-by-one, you simply need to add a rule to “accept” traffic from this IP above the rule blocking 59.45.175.0/24. So, if you run the command:

iptables -I INPUT 1 -s 59.45.175.10 -j ACCEPT

This rule is inserted at the first line, and it makes the rule blocking 59.45.175.0/24 come to the second line. You can verify this by listing the rules:

Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT all -- 59.45.175.10 0.0.0.0/0
2 DROP all -- 59.45.175.0/24 0.0.0.0/0
3 DROP all -- 221.192.0.0/20 0.0.0.0/0
4 DROP all -- 91.197.232.104/29 0.0.0.0/0

You can also replace rules with the -R switch. As an example, perhaps you whitelisted the wrong IP, and typed in 59.45.175.12 instead of 59.45.175.10. Since the new rule is on the first line, you can replace it with the correct rule like so:

iptables -R INPUT 1 -s 59.45.175.10 -j ACCEPT

Protocols and modules

So far, we’ve seen examples of blocking IPs. However, you can do a lot more, by using modules and protocol based matching.

We’ll begin with a simple example. Say, you want to block all incoming TCP traffic. This isn’t too useful, but it’s an useful starting point. You simply need to specify the protocol with -p like so:

iptables -A INPUT -p tcp -j DROP

You can also specify other protocols such as udp or icmp. To work with ICMP traffic on ip6tables, you’ll need to use ipv6-icmp as the protocol.

Let us consider a more useful example this time. Say, you need to block SSH access for an IP range. You have to first match all TCP traffic, like we did in the example above. Then, to check the destination port, you should first load the tcp module with -m. Next, you can check if the traffic is intended to the SSH destination port by using --dport. Thus, the entire command would be:

iptables -A INPUT -p tcp -m tcp --dport 22 -s 59.45.175.0/24 -j DROP

Now, perhaps you want to block SSH and VNC access for the IP range. While you can’t specify multiple ports with the tcp module, you can do so with the multiport module. Then, you can specify the port numbers with --dports. The final command would be:

iptables -A INPUT -p tcp -m multiport --dports 22,5901 -s 59.45.175.0/24 -j DROP

Let us consider another example. Say, you want to block ICMP address mask requests (type 17). First, you should match ICMP traffic, and then you should match the traffic type by using icmp-type in the icmp module:

iptables -A INPUT -p icmp -m icmp --icmp-type 17 -j DROP

In the next section, we’re going to look at a very important module: the connection tracking module. In addition, we’re also going to look at other modules later in this article.

The connection tracking module

If you’ve tried blocking certain IPs on the INPUT chain, you might have noticed an interesting caveat — you can’t access the services hosted on those IPs either! You might think that rules in the INPUT chain are somehow affecting traffic on the OUTPUT chain, but that isn’t the case. The packets from your system do reach the server. However, the packets that the server sends to your system get rejected. (See the next section for an additional example.)

What we really need here is a way to tell iptables to not touch packets that are part of an existing connection. Fortunately, iptables is a stateful firewall, and it provides a connection tracking module named “conntrack” for this purpose. Connections tracked by this module will be in one of the following states:

NEW: This state represents the very first packet of a connection.
ESTABLISHED: This state is used for packets that are part of an existing connection. For a connection to be in this state, it should have received a reply from the other host.
RELATED: This state is used for connections that are related to another ESTABLISHED connection. An example of this is a FTP data connection — they’re “related” to the already “established” control connection.
INVALID: This state means the packet doesn’t have a proper state. This may be due to several reasons, such as the system running out of memory or due to some types of ICMP traffic.
UNTRACKED: Any packets exempted from connection tracking in the raw table with the NOTRACK target end up in this state.
DNAT: This is a virtual state used to represent packets whose destination address was changed by rules in the nat table.
SNAT: Like DNAT, this state represents packets whose source address was changed.

Thus, you need to place a rule like the one below, usually at the very top. (If this isn’t the first rule, use -I to place it at the top.) The --ctstate switch sets the states. On some older kernels, this module is named state and the switch is named --state instead of --ctstate.

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

In addition, it’s generally a good idea to drop any packets in INVALID state. You can place it just below the position where you placed the above rule.

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

Changing the default policy

Previously, we’ve explained the concept of a default policy. By default, the default chains have a default policy of accepting all traffic. This is also something that you can see in the list of rules:

Chain INPUT (policy ACCEPT)
...
Chain FORWARD (policy ACCEPT)
...
Chain OUTPUT (policy ACCEPT)
..

You can change the default policy with the -P switch. Perhaps you’re configuring iptables for your home computer, and you don’t want any local programs to receive inbound connections. Once you’ve set iptables to accept packets from related and established connections, you can switch the INPUT chain’s policy to DROP with:

iptables -P INPUT DROP

Keep in mind that you should first accept packets from established and related connections before using this rule! If you don’t, you’ll find out that you can’t use any internet based applications, becuase the responses coming in through the INPUT chain will be dropped.

Selecting interfaces

Since iptables matches packets to every rule in a chain, things can get really slow when there are a lot of rules. In such cases, it’s useful to exempt certain kinds of traffic.

For example, on a typical Nginx/PHP-FPM stack, Nginx communicates with PHP over localhost, which uses the loopback interface. Similarly, PHP may connect to a database server like Redis using the loopback interface. It’s useless to filter these kinds of traffic, so you can allow it. The loopback interface is typically named lo and you can add a rule like this at the top of the INPUT chain:

iptables -A INPUT -i lo -j ACCEPT

The -i flag specifies the input interface. Here, we’ve assumed that the INPUT chain is empty and you’re adding the first rule. If that isn’t the case, you need to use the -I switch to add it at the top.

For the OUTPUT chain, you’ll need to use the -o flag, which stands for “output interface”. As an example, say you want to block the IP range 121.18.238.0/29, only when you’re on WiFi. If the WiFi interface is named wlan0, you can add a rule like the one below:

iptables -A OUTPUT -o wlan0 -d 121.18.238.0/29 -j DROP

Negating conditions

At this point, we’ve seen many examples on how to match packets based on certain conditions. However, in some cases, you may need to negate these condition checks. iptables provides the negation operator ( !) for this purpose.

A very common way to run node.js or Go web applications on a server is to place a server such as nginx in front of them. Once you’ve configured nginx to forward traffic to the app, there’s no need for it to be directly accessible. In other words, you simply need to accept TCP traffic intended for HTTP, HTTPS and SSH — and you could drop the rest:

iptables -A INPUT -p tcp -m multiport ! --dports 22,80,443 -j DROP

However, you should first accept packets from established and related connections before using this rule! If you don’t, you’ll find out that you can’t use any TCP based applications. This is because legitimate TCP traffic would be dropped, too.

Blocking invalid TCP packets with the tcp module

Now that we know quite a bit about iptables, let us design some rules to block invalid TCP packets. The tcp module has a --tcp-flags switch, and you can use it to check individual TCP flags. This switch takes in two arguments: a “mask” and a set of “compared flags”. The mask selects the flags that should be checked, while the “compared flags” selects the flags that should be set in the packet.

Now, say for example, you want to block Christmas tree packets. So, you need to check “all” the flags, but only FIN, PSH and URG will be set. So, you can write a rule like so:

iptables -A INPUT -p tcp -m tcp --tcp-flags ALL FIN,PSH,URG -j DROP

In addition, there are many other types of invalid packets that you could reject too. For example, a packet with SYN and FIN set is invalid. In this case, you need to only check for these flags and verify if they’re set. So, the rule would be:

iptables -A INPUT -p tcp -m tcp --tcp-flags SYN,FIN SYN,FIN -j DROP

Next, we’ll consider another kind of invalid packet — a “new” connection that doesn’t begin with a SYN. Here, you simply need to check the FIN, RST, ACK and SYN flags; however only SYN should be set. Then, you can negate this condition. Finally, you can use conntrack to verify if the connection is new. Thus, the rule is:

iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP

Limiting packets: the limit module

In this section, we’re going to look at another module: limit. The job of this module is to place a limit to the number of packets passing through.

In order to understand how this module works, we’ll use an analogy. Suppose, you’re given a bucket containing 5 tokens. Whenever a packet comes in, you should throw out a token. Also, you can’t allow a packet if the bucket is empty. In addition, you can add back tokens at the rate of 3 in an hour (or 1 in 20 minutes). Technically, the number of tokens is the “limit-burst” value, and the rate at which you can refill it is the “limit” value.

Now, suppose 10 packets arrive on your system. In order to accept the first five packets, you throw out the first five tokens. Because your bucket is now empty, you have to discard the rest. In fact, you can’t accept any more packets for the next 20 minutes, because your bucket will be empty till that time. Now, suppose 45 minutes pass and three packets arrive. You can accept just the first two since your bucket now has just two tokens. If 100 minutes had passed without you receiving a packet, the bucket would have been completely refilled.

As a practical example, suppose you want to ratelimit ICMP packets. A rule like this can do the job:

iptables -A INPUT -p icmp -m limit --limit 1/sec --limit-burst 1 -j ACCEPT

As long as the traffic is within the given limits, packets will be accepted. However, as soon as the flow of packets exceed this limit, the traffic passes through this rule over to the other rules. Thus, you should set a default policy of DROP on the INPUT chain for this to work.

Per-IP packet limits: the recent module

Unfortunately, the limit module can’t handle a dynamic, per IP restriction. This is where the recent module comes in.

Perhaps you’ve been facing some brute force attacks on your SSH server. Usually, attackers try to make many connections to speed up their attack. So, you can place a per-IP restriction like so, which will slow down the attackers:

iptables -A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSHLIMIT --rsource
iptables -A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSHLIMIT --update --seconds 180 --hitcount 5 --name SSH --rsource -j DROP

We’ve set a name for the limit module by which it can keep track of things — in our case, it’s “SSHLIMIT”. The first line adds the source IP to the list that the recent module maintains. If the IP is already on this list, then the entry for this IP is updated. In the next line, we check whether the counter has hit the value of 5 in 180 seconds. If that’s indeed the case, we drop the packet. Thus, this allows 4 new SSH connections from an IP in 3 minutes.

Some recent kernels also have a --mask parameter. This allows you to put restrictions on IP ranges. So for example, if you want to do restrict an entire /24, you can run:

iptables ... -m recent ... --mask 255.255.255.0

Since the recent module provides based on the name, it’s possible to do quite a few tricks. For example, if you remove the conditions to match SSH from the second line, it would drop all traffic from the attacker. You can use this to block attackers completely from your system, if you wish.

The owner module

On a home computer, it’s often useful to block traffic on a per user basis. The owner module can help you with this job.

As an example, suppose you use a shared computer at your home. You’d like to block a particular website which as the IP 31.13.78.35 for your child. Assume that your child uses an account with the username bobby. So, the iptables rule will be:

iptables -A OUTPUT -d 31.13.78.35 -m owner --uid-owner bobby -j DROP

You can also use a numeric user ID or a range (such as 1000-1006) for the argument. Similarly, you can match packets of a group by using the --gid-owner packet.

The owner module is valid only in the OUTPUT and POSTROUTING chains.

Custom chains

Sometimes, you may need to do some complex processing on the same packet over and over. For example, say you want to allow SSH access just for a couple of IP ranges. With the types of rules we’ve seen so far, you’d have to do something like:

iptables -A INPUT -p tcp -m tcp --dport 22 -s 18.130.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -s 18.11.0.0/16 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 -j DROP

Unfortunately, this is a bit unwieldy and inefficient. A better way to organize these rules would be to use custom chains. First, you need to make a custom chain. We’ll name ours ssh-rules:

iptables -N ssh-rules

Then, you can add the rules for the IPs in the new chain. Of course, we aren’t limited to matching IPs — you can do just about anything here. However, since custom chains don’t have a default policy, make sure you end up doing something to the packet. Here, we’ve added a last line that drops everything else. There’s also the RETURN target, which allows you to return to the parent chain and match the other rules there — it’s similar to a non-terminating target.

iptables -A ssh-rules -s 18.130.0.0/16 -j ACCEPT
iptables -A ssh-rules -s 18.11.0.0/16 -j ACCEPT
iptables -A ssh-rules -j DROP

Now, you should put a rule in the INPUT chain that refers to it:

iptables -A INPUT -p tcp -m tcp --dport 22 -j ssh-rules

Using a custom chain carries many advantages. For example, you can entirely manage this chain through a script, and you don’t have to worry about interfering with the rest of the chain.

If you want to delete this chain, you should first delete any rules that reference to it. Then, you can remove the chain with:

iptables -X ssh-rules

Logging packets: the LOG target

We’ve worked with a number of terminating targets. We’ll now shift our focus to a commonly used non-terminating target: LOG. As its name suggests, it logs the nature of the packet matched in the kernel logs. The location of the log depends on the distribution, but it’s usually in /var/log/syslog or /var/log/messages.

This target is fairly easy to use. As an example, say you want to log invalid TCP packets before dropping them. You should first log the packet, and then drop it:

iptables -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j LOG
iptables -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP

The LOG target also takes a --log-prefix option, and you can use this so that you can search the log easily later:

iptables -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j LOG --log-prefix=iptables:

iptables-save and iptables-restore

The iptables/ip6tables commands provide a way to insert, delete and manage rules. However, the workflow isn’t particularly nice. First, you have to first list the existing rules. Next, you need to figure out where a new rule should go, and then write a command to insert the rule. Fortunately, iptables also comes with two commands, iptables-save and iptables-restore. (The IPv6 equivalents are ip6tables-save and ip6tables-restore). These commands dump rules from all chains and filters into standard output. You can redirect it to a file like so:

iptables-save > iptables.rules

Now, you can edit this file comfortably with a text editor. When you’re done, you can apply these rules with:

iptables-restore < iptables.rules

You can also use them to persist iptables rules, as we’ll see in the next section.

Preserving iptables rules across reboots

Unfortunately, it turns out that iptables rules aren’t persistent — they’re lost when you reboot your system. On distributions such as RHEL and CentOS, there’s a package named “iptables-services” which you can install with:

sudo yum install iptables-services

On Debian and Ubuntu, you can install the “iptables-persistent” package that does the same thing.

sudo apt install iptables-persistent

Internally, both of these packages run the iptables-save/restore commands to save iptables configuration to a file.

If your distribution doesn’t have a similar package, you can simply write a service file that loads iptables rules from a file when it starts up, and saves it when it stops.

Conclusion

A firewall is an important security tool for network administrators. As we’ve seen, iptables provides a very powerful firewall that you can use to secure your systems. We’ve covered a number of examples here that should be helpful to design iptables rules for the systems you manage.

Also, if case you’re willing to read more about iptables, this is a good resource (if a bit long). iptables-extensions’man page and the netfilter extension documentation also covers a few other modules we haven’t covered here.

How to Install WordPress on Windows with XAMPP

Mon, 05 Jun 2017 11:24:58 +0000

WordPress is a great Content Management System. It can be used to create nearly all kinds of websites ranging from blogs to e-commerce websites. In order to use Wordpress, we also need to install and configure a web server and a database. The Apache Friends project provides a software package named XAMPP, which comes with Apache, PHP and MySQL pre-configured out of the box.

In this post, you will learn how to install Wordpress on a Windows computer using XAMPP.

Install XAMPP

The first step is to install the XAMPP package. Go to the Apache Friends website and download the latest Windows version.

Once the download is complete, run the executable installer file. You will be prompted to check the components that you wish to install. To run WordPress, the minimum components we need PHP, MySQL, Perl and the Apache web server. If you want, you may uncheck the rest of the components.

Complete the rest of the installation steps by following through all the steps in the installer. The default settings suggested are fine. Once the installation is complete, you should launch the “XAMPP Control Panel”.

The control panel allows you to start and stop the web server, change configuration settings, and view server logs. You can launch the XAMPP Control Panel through the following steps:

Press the Windows + R keys to bring up the “Run” dialog box.
Enter C:\xampp\xampp-control.exe in the dialog box, and hit Enter.

If you have changed the XAMPP installation path when installing it, you would need to replace C:\xampp with the path where you installed it.

The XAMPP Control Panel will start up, and prompt for the language of your choice. Select a language of your choice and click “Save”.

The control panel allows you to can start and stop the web server, change configuration settings and view server logs.

Now, we’ll check whether the server is working. Click on the start button for Apache and MySQL on the control panel. The Windows firewall (or any other antivirus installed on your system) may warn you at this point. Be sure to click “Allow” (or a similar option) so that the web server works properly.

Next, open your favourite browser and navigate to localhost, which tells your browser to connect to the website hosted on your own computer. If the setup was successful, you should be able to see the default web page, as shown in the screenshot below:

If you can see this page, this means you were able to install XAMPP successfully. Now, before we move on to installing WordPress, we need to set up MySQL, since WordPress stores your website’s pages in MySQL.

Setting up the database

Setting up the database consists of two steps — securing MySQL’s “root” account and then setting up a user and a database for Wordpress.

Securing the database

MySQL comes with a database administrator account called “root”. However, by default, XAMPP doesn’t use a password to protect this account. This is bad, because if a hacker gained access to your website, they can log in as the “root” user (with no password to protect it) and change or delete databases.

To secure the MySQL database, click on the “Shell” button in the XAMPP Control Panel, and type in:

mysql\bin\mysql.exe -u root

You will be logged into MySQL, as shown below:

Next, decide on a root password since we’ll use it to protect MySQL. You should also note down this password, as you may need it when installing other applications on your XAMPP server.

We’ll assume that the password you chose is myrootpassword. Run the following command to set the root password:

alter user root@localhost identified by 'myrootpassword';

Don’t just close the shell window yet — we’re going to need it in the next section.

Setting up users and databases

Now, we need to create a database into which WordPress stores all its data, and a separate MySQL user for WordPress. WordPress will use our newly created user to talk to MySQL and store information in it.

First, we’ll create the database with the following command:

create database wordpress;

Next, we’ll create an user account for Wordpress to connect to the MySQL database. Here, we’ve used an username wordpress and password of wordpressdatabasepassword. This password shouldn’t be the same as the root password.

create user wordpress@localhost identified by 'wordpressdatabasepassword';

Finally, we need to give this user to operate on the wordpress database:

grant all privileges on wordpress . * to wordpress@localhost;

Now that our database setup is complete, type exit and close the shell window.

Installing WordPress

Head over to the WordPress download page and download the ZIP file.

Then, head over to XAMPP’s “htdocs” folder. You can do this easily by pressing Windows + R to bring up the Run button, and then typing in C:\xampp\htdocs and hitting Enter.

Once you’re in the “htdocs” folder, remove all the existing files, and extract the WordPress ZIP file into this folder. You will end up with a single folder named “wordpress” inside the “htdocs” folder.

Now, go inside the “wordpress” folder, and “cut” all the files and folders. You can do so by pressing Ctrl+A followed by Ctrl +X. Then, go back to the “htdocs” folder and paste the files that you had cut previously. You can paste by pressing Ctrl+V. Then, you can delete the old WordPress folder, which is now empty since we’ve removed all the files from it.

You will end up with all the WordPress files directly under the “htdocs” folder, like so:

Now, point your browser to http://localhost/ to begin the setup. Set your preferred language and click on Continue. Then, you will be asked to enter your database configuration. Enter the database name and username as wordpress. In the password field, enter the password you’ve set for the MySQL wordpress user. Then, click on the Submit button to continue.

In the next page, you will be asked to give your site title, email, username and password. You need to remember them as these are the credentials that you will need to login to the WordPress admin panel.

Finally, click on “Install WordPress”. You will be redirected to a page confirming that WordPress has been installed.

Now click on Log in and enter the username and password that you just set in the previous step. Alternatively, you may visit http://localhost/wp-admin to open the login screen.

Voila! You have successfully installed WordPress locally using XAMPP. To view the site, go to http://localhost/. The default website can look different for you, as WordPress ships new themes with latest versions of WordPress.

At this point, you have a fully working WordPress installation. You may want to experiment with various themes and plugins, play around with the various settings, and build your website.

A Guide to Using the Caddy Server on Linux

Mon, 29 May 2017 04:42:02 +0000

Caddy is a web server that’s extremely easy to configure and use. It provides HTTP/2 support, automatic SSL configuration with Let’s Encrypt, and an intuitive configuration syntax. It is cross-platform, and can run across all major operating systems, such as Linux, macOS and Windows.

In this article, we’re going to install Caddy on a Linux system and configure it to serve a website. These instructions should work on most popular distributions, and they were tested on Ubuntu 18.04/20.04, Debian 9/10, and CentOS 7/8.

Prerequisites

First of all, we will assume that you have root access on your system through sudo. However, if you can log in to the system directly as the root user, then you can simply remove the sudo from the beginning of the command before executing it.

If you are logged in as a non-root user, but do not have root access through sudo, then you can use the su command to obtain a root shell. Then, you can remove the sudo from the beginning of the commands, as mentioned earlier.

In addition, before you follow this article to install Caddy, there are a few packages that you need to install — nss-tools, setcap, wget and tar.

On Debian/Ubuntu, use the following command to install these packages:

sudo apt update
sudo apt install libcap2-bin libnss3-tools wget tar

On CentOS, use the following command to install these packages:

sudo yum install libcap nss-tools wget tar

Installing the Caddy binary

Head over to the Caddy releases page, which offers binaries for a variety of process architectures. To determine the processor architecture for your system, run the following command:

uname -m

Most often, servers and desktop systems run on Intel 64-bit processors. The output of the above command on such systems would be x86_64, which is also known as “amd64”. Therefore, you should download the .tar.gz file for Linux amd64:

wget https://github.com/caddyserver/caddy/releases/download/v2.0.0/caddy_2.0.0_linux_amd64.tar.gz

ARM based systems such as the Raspberry Pi, the output of the uname -m command is armv6l or armv7l. You can also find .tar.gz files for both these architectures on the releases page, and download them as shown above.

Once you have downloaded the file, extract the caddy binary to the /usr/local/bin directory by running:

sudo tar -C /usr/local/bin/ -xf caddy_2.0.0_linux_amd64.tar.gz caddy

To confirm that the binary was installed, run the following command:

caddy

You should see Caddy’s help text, as shown in the screenshot below. If you see this text, this would indicate that the binary was installed successfully.

Setting up user accounts and directories

Web servers such as Caddy are exposed to the internet. It is a best practice to run any internet facing software as a restricted user.

Therefore, we’ll set up a restricted user named caddy with the home directory of /etc/caddy by running the following command:

sudo useradd --shell /bin/false --home-dir /etc/caddy --system caddy

Under the home directory, Caddy stores configuration under the .config directory, and certificates under the .local directory. So, run the following command to set up these two directories:

sudo mkdir -p /etc/caddy/.config /etc/caddy/.local

We’ll also need a directory in which we can tell Caddy to store access logs. For this purpose, we’ll create a directory /var/log/caddy with the following command:

sudo mkdir -p /var/log/caddy

Next, change the ownership of these directories to the caddy user:

sudo chown -R caddy: /etc/caddy /var/log/caddy

In addition, when you run software as a non-root user, Linux prohibits these processes from listening on to port numbers which are less than 1024. To bypass this restriction and run Caddy securely as a non-root user, add the cap_net_bind_service capability to the caddy binary:

sudo setcap cap_net_bind_service+ep /usr/local/bin/caddy

Auto-starting Caddy

Now that we’ve created the user and directories, we’ll set up Caddy as a system service. By doing this step, the Caddy web server will automatically start when you boot (or reboot) your system. You’ll also be able to start, stop, or force Caddy to reload its configuration by using the systemctl command. We’ll also define the working directory for Caddy through this file; which will set the default directory where Caddy will look for files to serve.

First, run the following command to determine the systemd version:

systemctl --version

This will list the version of systemd, as shown below:

Note down the systemd version, since you need to create a different file based on the systemd version.

Next, use your favourite text editor to create the file /etc/systemd/system/caddy.service. Most distributions have the vi editor installed, so you can run:

sudo vi /etc/systemd/system/caddy.service

In the vi editor, press i to enter the editing mode.

If the systemd version is greater than 229, paste or type in the following content into the editor:

[Unit]
Description=Caddy web server
After=network-online.target

[Service]
User=caddy
Group=caddy
Type=exec
WorkingDirectory=/var/caddy/html

ExecStart=/usr/local/bin/caddy run -config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload -config /etc/caddy/Caddyfile
ExecStop=/usr/local/bin/caddy stop

LimitNOFILE=1048576
LimitNPROC=512

PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/etc/caddy/.local /etc/caddy/.config /var/log

CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

However, if the systemd version is less than 229, paste or type in the contents shown below. Since systemd versions less than 229 do not have the feature of adding security restrictions to the process at runtime, we’ve omitted those here.

[Unit]
Description=Caddy web server
After=network-online.target

[Service]
User=caddy
Group=caddy
Type=simple
WorkingDirectory=/var/caddy/html

ExecStart=/usr/local/bin/caddy run -config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload -config /etc/caddy/Caddyfile
ExecStop=/usr/local/bin/caddy stop

LimitNOFILE=1048576
LimitNPROC=512

[Install]
WantedBy=multi-user.target

Exit the editing mode by pressing “Esc”, and then type :wq to save the file.

Configuring Caddy to serve a website

The final step in configuring Caddy is to set up a Caddyfile, which tells Caddy about what to do when a HTTP request arrives. To create the Caddyfile, use:

sudo vi /etc/caddy/Caddyfile

Now, enter the editing mode by pressing “i”, and then enter the following content:

http:// {
 encode gzip
 file_server
}

This will instruct Caddy to serve files present in its current working directory, and compress HTTP response bodies with the gzip algorithm. Now, what’s the working directory for Caddy? As we mentioned previously, we have set up the working directory when we created the service file, as you can see in the line WorkingDirectory=/var/caddy/html.

Now, after typing in the above contents in the Caddyfile, press “Esc” to exit the editing mode, and then type :wq to save it.

Then, we will create the /var/caddy/html directory and put an index.html file inside it. Run the following commands to do so:

sudo mkdir -p /var/caddy/html
sudo bash -c 'echo "<h1>Hello from Caddy web server</h1>" > /var/caddy/html/index.html'

After making these changes, run the following command to start Caddy:

sudo systemctl start caddy

Now, visit the URL http://<server ip address>/, replacing the <server ip address> with the server’s public IP address. You should see a page like this:

If you can see this web page, this means that your setup is working as expected. However, if you receive a connection error or a timeout when connecting to your website, it could indicate the server firewall is blocking traffic, or that Caddy was not configured properly.

To troubleshoot any Caddy configuration issues, you can run the following command to view errors:

sudo systemctl status caddy

Finally, you would also want to set up Caddy to automatically start when your system is restarted. Use this command to set the Caddy service for automatic reboots:

sudo systemctl enable caddy

Setting up a website with HTTPS

Now that you have a website up and running, you can configure Caddy to automatically add HTTPS to it.

To follow the steps in this section, you should have your own domain name. First, go to the DNS settings of your domain’s DNS provider. Create a subdomain, or edit the apex domain’s DNS entry so that it points to the IP address of the server.

After you have made the changes in DNS, edit the file /etc/caddy/Caddyfile so that the configuration looks like this instead:

{
 email <YOUR EMAIL ADDRESS HERE>
}

<YOUR DOMAIN NAME HERE> {
 encode gzip
 file_server
}

Make sure to replace the placeholders <YOUR EMAIL ADDRESS HERE> and <YOUR DOMAIN NAME HERE> with your email address and domain name respectively. There’s no need to specify the protocol “https://” when specifying the domain name. When you don’t specify a domain name, Caddy automatically obtains certificates. Then, it applies HTTPS to your website, and also sets up redirection from HTTP to HTTPS.

After making the above changes, reload the Caddy configuration by running:

sudo systemctl reload caddy

Wait for a few moments, and then run:

sudo systemctl status caddy

If all went well, the “status” command should show output stating that it’s obtained the certificates:

Now, using a browser, visit your domain name. The site will appear as HTTPS-protected, as shown below:

Hosting multiple websites (virtual hosting)

Often, you want to serve requests for multiple domains from the same server. This concept is sometimes known as “virtual hosting”. You can add additional blocks to your Caddyfile to perform virtual hosting.

As an example you have three domains, example1.com, example2.com and example3.com, where each domain would serve different content to the user. The first step is to create directories for these domains, such as /var/caddy/example1.com, /var/caddy/example2.com and /var/caddy/example3.com, and store the required files in these directories.

Once you have the content in place, edit your Caddyfile like so:

{
 email <YOUR EMAIL ADDRESS HERE>
}

example1.com {
 root * /var/caddy/example1.com
 encode gzip
 file_server
}

example2.com {
 root * /var/caddy/example2.com
 encode gzip
 file_server
}

example3.com {
 root * /var/caddy/example3.com
 encode gzip
 file_server
}

After making the above changes, reload the Caddy configuration by running:

sudo systemctl reload caddy

Caddy will set up HTTPS on these three domains and serve the content from the respective directories.

If, for some reason, you want that the sites should be HTTP-only, you can add a http:// before each site’s name, like so:

http://example1.com {
 # Caddyfile directives for example1.com
}

Logging requests

By default, Caddy does not log requests. However, you can configure Caddy to log details through the “log” directive. By default, Caddy stores its logs as lines of JSON, although other formats are available too.

As we’ve seen in the previous sections, the Caddyfile consists of sections in which you add the configuration for every website. To configure logging, edit the Caddyfile so that every block looks like this:

<YOUR DOMAIN NAME HERE, OR SPECIFY "http://"> {
 encode gzip
 file_server
 log {
 output file /var/log/caddy/access.log
 }
}

Once you’ve saved the Caddyfile, reload the configuration with:

sudo systemctl reload caddy

Now, visit the website in a browser, in order to generate a few log entries. Once you’ve done that, run the following command to view the logs:

sudo cat /var/log/caddy/access.log

Here’s an example of how Caddy’s logs look like:

You can also configure “log rotation” with Caddy. When the size of your logs exceeds a certain threshold, Caddy moves the old log entries in a new file, and this is known as “rotation”. If the rotated log files stay there for extended period of time (that you specify), Caddy deletes the rotated logs.

By default, Caddy rotates log files when they reach 100 MB. The rotated log files are deleted after 90 days, or when there are more than 10 rotated logs. If you want to customize these parameters, you can do it like so:

log {
 output file /var/log/caddy/access.log {
 roll_size 10MB
 roll_keep 5
 roll_keep_for 240h
 }
}

In this example, the logs are rotated after they exceed the 10MB limit. These logs are deleted after 10 days (240 hours), or when there are more than 5 rotated logs.

You can also specify different log files for different websites, as shown below:

example1.com {
 # other Caddyfile directives
 log {
 output file /var/log/caddy/example1.log
 }
}

example1.com {
 # other Caddyfile directives
 log {
 output file /var/log/caddy/example1.log
 }
}

Conclusion

In this article, we took a brief overview of Caddy’s functionality and how to configure and use it on a Linux system. If you want to learn more, you should begin by reading about the Caddyfile.

A real website will also often use dynamic content in addition to static content. If you have a PHP based website, you should take a look at the “php_fastcgi” directive. On the other hand, if you have a web app written in a language like Go, NodeJS or Rust that starts up its own HTTP server, use the “proxy” directive.

A Guide to SSH Port Forwarding/Tunnelling

Mon, 22 May 2017 05:26:05 +0000

SSH is a widely used protocol for system administration and file transfer. In addition, it has a feature called SSH tunnelling (or SSH port forwarding). It creates an encrypted connection between a local computer and a remote computer through which you can relay traffic. It is very useful, and you can use it to securely access unencrypted protocols such as VNC or firewalled resources like database servers. You can also use it as a form of proxy/VPN and get around restrictive, firewalled networks.

In this article, we’re going to take a look at using SSH port forwarding.

Prerequisites

Before you can begin, you need to check if forwarding is allowed on the SSH server you’ll connect to.

If you’re using the OpenSSH server, open /etc/ssh/sshd_config in a text editor. If you find AllowTcpForwarding is set to No, change them to Yes. In addition, if you’re going to use remote port forwarding (discussed later in this article), you also have to set GatewayPorts to Yes. Then, you need to restart the server for the change to take effect.

If you’re on Linux, depending upon the init system used by your distribution, run:

sudo systemctl restart sshd
sudo service sshd restart

Again, depending on your distribution, you may find that the service is named ssh instead of sshd.

If you’re on a Mac, you can restart the server like so:

sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

If you’re on Windows and want to set up a SSH server, have a look at MSYS2 or Cygwin.

You also need to have a SSH client on the computer you’re working on. On most Unix-like systems, it’s already installed by default. If you’re on Windows 10 and you use Bash on Windows, you can install OpenSSH in it the way you would on a regular Ubuntu system. However, you may need to add the -4 switch, as IPv6 is not supported properly there.

For other versions of Windows, you can use the OpenSSH package from MSYS2 or Cygwin. If you’re not willing to bring an entire Unix-like system on your computer, try Putty.

(If you’re using Putty as your SSH client, simply replace the ssh command with putty in the examples below.)

Local port forwarding

Local port forwarding allows you to forward traffic on a port of your local computer to the SSH server, which is forwarded to a destination server.

That was quite a mouthful, so let’s look at a few examples. Say, you want to connect to a database server running at port 3306 on your server. The port for the database server is firewalled to protect it from external attackers. Although you can’t directly access the database server, you can do through SSH.

On your local computer, type in:

ssh -L 4000:127.0.0.1:3306 user@example.com

Now, SSH will bind to port 4000 on your computer. Any traffic that comes to this port is sent to the SSH server. Then, the traffic received is sent to port 3306 of 127.0.0.1, which is the server itself.

Now, you can connect to the database. You simply need to set the database client to use 127.0.0.1 as the host and 4000 as the port.

Moreover, you can forward multiple sets of ports in a single ssh command:

ssh -L 5901:127.0.0.1:5901 -L 4000:127.0.0.1:3306 user@example.com

In addition to forwarding the local port 4000 to 127.0.0.1:3306, we’ve also forwarded the local port 5901 to 127.0.0.1:5901.

Note that the destination to which the SSH server forwards the traffic is from the perspective of the server itself. This has some interesting implications. Say for example, there’s a single server having public SSH access in your company. You want to connect to the database server, which is running on server003.local in the internal network, on port 3306. Now, you can forward ports like so:

ssh -L 4000:server003.local:3306 user@example.com

Now, point your database client to 127.0.0.1:3306 and bam — you’re connected to the database server!

Remote port forwarding

Remote port forwarding is the exact opposite of local port forwarding. It forwards traffic coming to a port on your server to your local computer, and then it is sent to a destination. Again, let’s take a look at an example.

Suppose, you’re developing a web application that’s running on port 8000 of your local computer. Other people can’t access it directly because you’re sitting behind a NAT network without a public IP. Now, you would like to show a customer how the application looks like. Fortunately, remote forwarding can help you with this. Type in:

ssh -R 7000:127.0.0.1:8000 user@example.com

When you run this command, the SSH server binds to the 7000 port on example.com. Any traffic that it receives on this port is sent to the SSH client on your local computer, which in turn forwards it to port 8000 on 127.0.0.1. Now, the customer can open http://example.com:7000 in a browser, and they can use your application!

Again, as with remote port forwarding, you can also use a different destination. Say for example, you want a friend to help you with configuring a router. Since they can’t access your router directly, you can use remote port forwarding like so:

ssh -R 8080:192.168.100.1:8000 user@example.com

Dynamic port forwarding

So far, we’ve seen how to forward local ports with SSH. Although it comes very close to acting like a proxy, but you can’t use it as-is. This is because, you’ll need to specify different ports for every destination and every service you want to access, which simply isn’t practical.

Fortunately, there’s another type of forwarding called “dynamic port forwarding” which can be used for this purpose. In order to set up dynamic port forwarding, type the following command:

ssh -D 4000 user@example.com

The SSH client creates a SOCKS proxy at port 4000 on your local computer. Any traffic sent to this port is sent to its destination through the SSH server.

Next, you’ll need to configure your applications to use this server. The “Settings” section of most web browsers allow you to use a SOCKS proxy. Moreover, there are some extensions like Proxy Helper(Chrome) and FoxyProxy Standard(Firefox) that allow you to set up and switch proxies quickly.

For other applications, the Arch Linux wiki has some great instructions on using SOCKS proxies.

A few tips

If you’ve tried the above examples, you may have noticed that running the command also opens up a shell. If you don’t need the shell, you can disable it with the -N switch:

ssh -N ... user@example.com

If you want SSH to go into the background, you can do so with the -f flag.

ssh -f -N ... user@example.com

If you’re using SSH as a proxy, it’s best not to run a HTTP server on the remote machine. This is because many services will refuse registrations and transactions under the assumption that you’re using a web proxy.

Conclusion

In this article, we’ve seen how to use SSH forwarding, and how its useful in a variety of situations. If you have questions, feel free to ask in the comments below!

(BTW, if you’re using any of this at your workplace, you should first see if it’s okay with management — they might not be cool with it, after all.)

Adding HTTPS to Your Website with Let's Encrypt

Mon, 15 May 2017 04:53:03 +0000

HTTPS is an important part of securing websites. In our previous article, we’ve discussed how to add HTTPS to your website with Cloudflare. Unfortunately, for all intents and purposes, CloudFlare’s HTTPS is just an intercepting proxy sitting between your server and the client. In addition, it won’t work at all if you’re serving a website with a dynamic DNS service.

Let’s Encrypt is a new and completely automated Certificate Authority (CA) which provides Domain Validated (DV) certificates for your domain for free. In this post, we will show you how you can obtain and deploy Let’s Encrypt certificates and deploy them for servers such as Apache and nginx. These instructions should work on any Linux distribution, including popular ones such as Ubuntu, Debian and CentOS.

Installing Certbot

The first step with any Let’s Encrypt deployment is to install a client that verifies that you own the domain and sets your server up to serve files. There are many clients for this, but we’re going to use the official client, Certbot.

Unfortunately, most distros don’t carry Certbot or carry old copies of it. We’re going to use the latest version that’s available on EFF’s website. Download it and make it executable like so:

wget https://dl.eff.org/certbot-auto
chmod +x certbot-auto

Then, move it to a standard location from which it will be accessible, such as /usr/bin:

mv certbot-auto /usr/bin

That’s it — you’ve installed Certbot! It can automatically bootstrap itself and install any required dependencies.

Deploying HTTPS on Apache and nginx

Now that we’ve installed certbot, we’ll take a look at how to deploy HTTPS. Fortunately, Certbot has plugins for Apache and nginx, making it a breeze to set things up.

Depending on the webserver you’re using, you simply need to run its plugin. So, for Apache, you should run:

sudo certbot-auto --apache

For nginx, you should run:

sudo certbot-auto --nginx

certbot-auto will now bootstrap itself by installing the necessary packages and downloading some necessary code. When it’s done, you will be asked for your email address and to agree to the terms of service.

Then, you’ll be asked to select a website to deploy HTTPS for. (Certbot gets this list by looking at your server configuration.) Leave the field blank if you want to deploy for all of them, or you can deploy it for a specific set of sites by entering the numbers from the list.

If Certbot can’t find the name of your website, it’ll ask you to type it in.

After you type in the server names, Certbot will now complete a challenge given by the server to prove that you indeed own the domain. Once it is complete, Certbot will make the necessary changes to your server configuration to deploy HTTPS. You can find a list of files changed in the output:

Finally, you need to select if you want to redirect all traffic to HTTPS by default. Select the option that suits your needs best, and hit Enter.

Now, you can visit your website from a browser. Depending on the option you’ve selected before, you may need to add a https:// before your site’s address. If everything worked fine, you’ll find that your website now has HTTPS.

Deploying HTTPS on other web servers

It’s fairly easy to deploy HTTPS on Apache and nginx. However, what if you’re using something less common like HAProxy or cloud-torrent? Fortunately, that’s easy too. Before starting, however, be sure to stop any web servers that you’re running.

Then, run certbot-auto with the “certonly” option. It simply gets certificates but doesn’t configure your server.

sudo certbot-auto certonly

You’ll be asked to select an authentication method. The standalone method works in all cases, so select it and hit Enter.

The rest of the setup instructions are the same as with the Apache/nginx instructions. After the setup, you can start any web servers back up. The certificate will be stored in /etc/letsencrypt/live/<your_domain>/fullchain.pem and the private key will be stored in /etc/letsencrypt/live/<your_domain>/privkey.pem. Now, you simply need to modify your server configuration to use these two files.

Renewing Let’s Encrypt certificates

Let’s Encrypt certificates are valid for 90 days, and you need to renew them before they expire. You can manually renew them with:

sudo certbot-auto renew

If you’re using the “certonly” method, Certbot won’t be aware of the software you’re using. So, it won’t have any idea about how to restart your server after a renewal. However, you can manually specify commands to be executed after a renewal. For example, if you use HAProxy, and you want to restart it after installing the certificate, you can run:

sudo certbot-auto renew --renew-hook "systemctl restart haproxy"

However, having to renew them by hand again and again would get boring quickly. So, you should set up a cron job to take care of renewals automatically.

Open the crontab file for the root user with:

sudo crontab -e

The crontab file will open in a text editor. Here, you need to add this line:

30 12,6 * * * /usr/bin/certbot-auto renew 2>>/var/log/cert-renew.log >>/var/log/cert-renew.log

This command will run the certbot renew command at 6:30 and 12:30 every day. The command ignores any request to update certificates before its close to the renewal date, so running it everyday isn’t a problem. The output of the command will get logged to /var/log/cert-renew.log.

To install the modified crontab file, save it and exit the editor.

The staging environment

To prevent abuse of their system, Let’s Encrypt has limits on the rate at which they issue certificate. Thus, if you frequently obtain new certificates for testing purposes, you may be unable to obtain a certificate. In order to prevent this issue, you can use the “staging environment”. Although it gives out fake certificates that aren’t trusted by a browser, this is fine if you are writing some scripts to automate your website deployment process.

Suppose you want to automatically configure Apache with a certificate from the staging environment. Simply tack on --test-cert at the end along with the original command, like so:

sudo certbot-auto --apache --test-cert

If your website deployment process involves some tests to check if your website is working correctly, these may start failing. This is because, as we previously mentioned, these certificates are not trusted by a browser. In order to work around this problem, you can import the “Fake LE Root X1” certificate available on the staging environment page.

Removing Let’s Encrypt certificates

Sometimes, you might need to remove a Let’s Encrypt certificate from a server. This may be due to a server compromise (which means your private key has possibly been stolen), or perhaps you don’t need to operate that site any more. In these cases, you can remove the certificate with the following instructions.

The first step is to “revoke” the certificate, which marks the certificate as invalid. You can do this by using the “revoke” command with the full path of the “cert.pem” file:

sudo certbot-auto revoke --cert-path /etc/letsencrypt/live/<certificate_name>/cert.pem

Next, you can delete the certificate with:

sudo certbot-auto delete --cert-name <your_domain>

You might wonder how revocations work, given that you are not being issued new certificates. Each Let’s Encrypt certificate contains an URL where the revocation status of certificates are maintained. Before a browser (or any other client) exchanges data over SSL/TLS, it checks this URL to see if the certificate is valid.

Conclusion

In this post, we successfully obtained a Domain Validated free SSL certificate using Let’s Encrypt. Serving your website over HTTPS provides security (and SEO!) benefits and it’s a no-brainer these days. If you have any questions, do ask us in the comments below.

How to Access Your Mac over SSH with Remote Login

Fri, 05 May 2017 16:25:36 +0000

Remote Login is a feature in macOS that allows you to connect to your Mac from another computer. Enabling this feature starts up a SSH server, and it allows you to access your Mac through a command line interface and transfer files.

SSH is an encrypted and authenticated protocol. Therefore, it protects you against man-in-the-middle attacks, unlike traditionally used protocols such as telnet and FTP.

In this article, we’re going to take a look at how to enable Remote Login and use SSH to manage your Mac and transfer files.

Enabling Remote Login

Enabling Remote Login is quite easy. You simply need to:

Open “System Preferences” from the Apple menu and click on “Sharing”.
Then, enable Remote Login by clicking on the checkbox.

As soon as you click on the checkbox, the SSH server is started in the background.

If you want to restrict SSH logins to certain users or groups of users, select “Only these users” and add some users to the list. Otherwise, you can select “All users”.

Now, you might want to verify whether the SSH server working as intended. Open a Terminal window and type in:

ssh <your_username>@localhost

This command will show you the server key and ask you to accept it. Type in “yes” to verify, and then type in your password. You’ll now be logged in to your Mac over SSH. Now, you can execute various commands. For example, in the screenshot below we’ve run a few commands:

Since we now know that everything is working, type in “logout” and hit Enter.

Connecting to your Mac remotely

In order to connect to your Mac from a remote computer, you should first know its IP address. Fortunately, you can find it in the “Sharing” settings window. However, if your Mac is behind a router, you’ll see a local IP address like “10.0.2.15” and while you can still SSH into your computer from any other device on the same network, you can’t access it from the internet. You’ll need to configure port forwarding on your router so that any inbound traffic on port 22 is sent to your Mac.

After enabling Remote Login and configuring your network (if needed), you can now connect to your Mac from another computer over SSH.

On Linux, macOS or an Unix variant, you can fire up a terminal window and type in:

ssh <your_username>@<your_mac's_ip_address>

If you’re using Bash on Windows, open the “Bash on Ubuntu on Windows” shortcut to get a bash shell and type the above command there. Cygwin and MSYS2 also have the OpenSSH package in their repositories and you can connect with the same command in their shells.

If you aren’t using Bash on Windows, you can use PuTTY to connect to your Mac.

If you’re on a mobile device, check out Termius, ConnectBot and JuiceSSH.

Transferring files remotely

SSH allows you to transfer files with methods such as SFTP, rsync and SCP. In this section, we’re going to take a look at using the first two protocols to transfer files.

SFTP

When it comes to SFTP, there are a variety of clients you can use. There’s the command-line based sftp program, as well as GUI based clients such as Cyberduck(macOS) and WinSCP(Windows). For example, if you’re using Cyberduck, select SFTP from the dropdown and fill in the server details:

Next, click on “Connect”. Cyberduck will log in and show you a list of files. Now, you can modify, remove, upload and download files from your Mac.

rsync

Another way to transfer files is rsync. It has the same semantics of cp. Let us assume that you’re the user john and your Mac is located at 11.22.33.44. Say, you want to copy a file named “report.txt” in your home folder onto the current computer you’re working on. You can do it with:

rsync -v john@11.22.33.44:~/report.txt ./

The -v flag is to enable verbose logging, and it is useful to see the percentage, speed and the list of files transferred.

If you want to transfer a directory on your current computer to the home folder on your Mac, you can run:

rsync -rv ~/Movies john@11.22.33.44:~

Just like cp, the -r flag allows you to copy directories.

rsync has a lot of options, such as preserving file permissions, partial transfers and compression. If you’re interested, check out the man page.

Enabling Remote Login from the Command Line

If you’re setting up your Mac with scripts, you can enable Remote Login from the command line. On a terminal window, type:

sudo systemsetup -setremotelogin on

If you want to disable it, you can do so with:

sudo systemsetup -setremotelogin off

Changing SSH server preferences

Sometimes, you may need to change the preferences for the SSH server. For example, you may want to run SSH on a different port to thwart attackers, or you may want to disable password authentication. Open up a terminal window and type in:

sudo nano /etc/ssh/sshd_config

This will open up the SSH server configuration file in the nano text editor:

Once you’re finished editing the file, press Ctrl + O to save the file. Then, press Ctrl + X to exit the editor.

Next, you should restart the server for your changes to take effect. In the terminal window, run these commands:

sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

Any existing SSH sessions will get terminated, so don’t restart the server if you’re working on something through SSH at the same time.

Conclusion

SSH is a pretty nifty thing to have on your system. Fortunately, macOS makes it a breeze to set up a SSH server, allowing you to control your Mac and transfer files.

If you have any questions, feel free to ask in the comments!

How to Get an Unix/Linux Environment on Windows with MSYS2

Sat, 29 Apr 2017 11:30:01 +0000

If you’re a Unix/Linux user who also has to work with Windows systems, you’ve probably wished that Windows was more Unix-like. An Unix-like environment provides many advantages — the powerful text processing tools, using SSH to remotely manage systems, being able to control every aspect of a system, the list goes on.

While Windows has been lacking in this regard, other people have created similar tools emulate the Unix environment. In this article, we’re going to look at how to set-up an Unix-like environment for Windows, named MSYS2.

So, what is MSYS2?

MSYS2 is a port of a collection of standard Unix/Linux utilities to Windows. It provides shells, development tools and version control system programs. Because it provides tools like gcc, it is also useful to compile Windows versions of Unix-like programs!

In addition, MSYS2 comes with pacman, the package manager from Arch Linux.

I already use Bash On Windows/Cygwin, how is it better?

If you already know about or use Bash On Windows/Cygwin, you might be thinking “why should I use MSYS2”? Probably one of the biggest arguments is MSYS2 handles file paths correctly. Both Cygwin and Bash on Windows have problems when you try to invoke normal Windows programs with Unix-like paths. On the other hand, MSYS2 handles things like this without a hitch:

start notepad++ /c/Users/John/file.txt

You get to keep that Unix-ish style to your commands, even when working with Windows programs!

Both Cygwin and MSYS2 can be used to compile Windows version of Unix/Linux programs. However, Cygwin uses a compatibility layer that tries to simulate Linux better, and it comes at a huge performance cost. MSYS2 compromises better compatibility for performance, which means MSYS2 should be significantly faster.

If you’re using Bash on Windows, one of the disadvantages is you won’t have the huge choice of packages that Ubuntu provides. This shouldn’t be a problem though — most software for Linux also has a Windows version which you can download separately.

Installation

You can find the MSYS2 downloads here. It’s available in both 32-bit(i686) and 64-bit(x86-64) variants. Download and run the installer and follow through the steps — at the end, you’ll end up with MSYS2 installed in C:\msys32 or C:\msys64 depending on whether your system is 32- or 64-bit. (In this article, we’ll assume that it was installed in C:\msys32.)

You’ll also find a folder named “MSYS2 32/64 bit” in your start menu, which contains shortcuts to launch a MSYS2 shell. Most often, you’ll need the “MSYS2 MSYS” shell. (We’ll see the purpose of the other “MSYS2 MinGW” shells in a minute.) If you fire up the shortcut, you’ll find a Bash shell starting up. The shortcut doesn’t use Windows’ default console window; it uses a terminal emulator named mintty.

Launch the “MSYS2 MSYS” shortcut, and run pacman -Syu to update all packages and the locally cached repository information. MSYS2 should now be ready for your use, and you can install other applications like Python and Ruby (or something else) through pacman.

Adding MSYS2 to your PATH variable

The MSYS2 tools (and that includes tools like grep) won’t be available if you don’t launch them through the Start Menu shortcut. To make them available everywhere, you need to add them to the “Path” variable like so:

Open the Run box by pressing Windows + R, and type in systempropertiesadvanced.
Click on the “Environment Variables” button.
In the “System Variables” section, scroll down and double-click on the “Path” variable.
If you’re on Windows 7/8, add the text C:\msys32\usr\bin;C:\msys32\usr\local\bin; to the beginning of the variable value. If you’re on Windows 10, add the C:\msys32\usr\bin and C:\msys32\usr\local\bin folders to the list, and move these two entries to the top.

You’ll now be able to access MSYS2 tools everywhere, including the Windows Command Prompt and the Run box. This means you can do stuff like this in the Command Prompt:

C:\Users\Admin>ipconfig /all | grep 'IPv4 Address'
   IPv4 Address. . . . . . . . . . . : 192.168.56.10(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.122.203.7(Preferred)

The difference between MinGW32/64 and MSYS2

As we’ve already said, MSYS2 can be used to compile Unix programs for Windows. MSYS2 also comes with a file named msys-2.0.dll that provides a lightweight compatibility layer. This layer is why you can work with Unix paths on MSYS2 programs that work seamlessly for Windows programs — behind the scenes, it converts these paths. However, even this layer has some overhead; and if you’re building Unix programs for people who don’t care about Unix compatibility, it’s even more pointless.

The MinGW shells allows you to use compilers and libraries that don’t depend on this layer. In fact, the number of library packages provided in the MinGW repositories is far higher than the number of MSYS2 packages.

Let’s say, you’ve been looking for the MinGW version of gcc. First, you need to locate the package:

$ pacman -Ss gcc
mingw32/mingw-w64-i686-gcc 6.3.0-2 (mingw-w64-i686-toolchain)
    GNU Compiler Collection (C,C++,OpenMP) for MinGW-w64
mingw64/mingw-w64-x86_64-gcc 6.3.0-2 (mingw-w64-x86_64-toolchain)
    GNU Compiler Collection (C,C++,OpenMP) for MinGW-w64
msys/gcc 6.3.0-1 (msys2-devel)
    The GNU Compiler Collection - C and C++ frontends

Say, you want to use the 32-bit MinGW version of gcc. You can install it with:

$ pacman -Ss mingw-w64-i686-gcc

When you’re compiling a program with this library, you should start up the “MSYS MinGW 32 bit” shortcut and work on that shell. It will automatically configure the shell to use the MinGW packages.

The “behind the scenes” view

If you’re an intermediate/advanced user, you probably want to know what’s going on behind the scenes when you use these different shortcuts. The easiest way to know this is to use declare.

From the MinGW shell, run:

$ declare > mingw-config

Again, run this from the MSYS2 shell:

$ declare > msys2-config

Now, you can diff these files to see the differences in environment variables. Here we’ve only shown what has changed; the actual diff is a whole lot longer.

+ACLOCAL_PATH=/mingw32/share/aclocal:/usr/share/aclocal
-CONFIG_SITE=/etc/config.site
+CONFIG_SITE=/mingw32/etc/config.site
-MANPATH=/usr/local/man:/usr/share/man:/usr/man:/share/man
+MANPATH=/mingw32/share/man:/usr/local/man:/usr/share/man:/usr/man:/share/man
+MINGW_CHOST=i686-w64-mingw32
+MINGW_MOUNT_POINT=/mingw32
+MINGW_PACKAGE_PREFIX=mingw-w64-i686
+MINGW_PREFIX=/mingw32
-MSYSTEM=MSYS
+MSYSTEM=MINGW32
-MSYSTEM_CHOST=x86_64-pc-msys
-MSYSTEM_PREFIX=/usr
+MSYSTEM_CHOST=i686-w64-mingw32
+MSYSTEM_PREFIX=/mingw32
-PATH=/usr/local/bin:/usr/bin:/bin:/opt/bin:[other paths]
+PATH=/mingw32/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:[other paths]
-PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/share/pkgconfig:/lib/pkgconfig
+PKG_CONFIG_PATH=/mingw32/lib/pkgconfig:/mingw32/share/pkgconfig

Setting up SSH

One of the best parts about an Unix-like system is SSH. It is a secure text-based protocol that allows you to work on remote computers. You can even use it for transferring files and use it as a proxy server.

Fortunately, setting up a SSH server is quite easy. First, we need to install the openssh package:

$ pacman -S openssh

Unlike most Unix-like systems, simply installing the package won’t enable the server. You need to create keys for the SSH server to use:

$ ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
$ ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
$ ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
$ ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

Now, open /etc/ssh/sshd_config (In our case, it’s located in C:\msys32\etc\ssh\sshd_config). Then, set the following preferences in the file:

UsePrivilegeSeparation no
PasswordAuthentication yes

The openssh server generally needs a “privilege seperation user” to provide better security against attacks. However, since it’s difficult to set up such an user on Windows, we disable it by setting UsePrivilegeSeparation to No. (This doesn’t reduce the security of the server; running as a different user is a defense-in-depth measure.)

We’ll also enable password authentication in SSH by setting PasswordAuthentication to yes. For the purposes of this article, we won’t set up key based authentication in SSH.

Now, try running the server with:

$ /usr/bin/sshd

At this point, you’ll find a warning from Windows Firewall. Depending upon how you want to access your computer, select the option to allow access on public and private networks.

The SSHD process should detach from the terminal. You can check if the server is working by trying to connect from it locally with:

$ ssh -l <your_windows_username> localhost

If your Windows username contains spaces, you can write it in double quotes.

Starting the SSH server on system boot

Now that we have our SSH server up and running, we need a way to way to auto-start it when the system boots up. We can do this by adding a scheduled task:

Open the Run box by pressing Windows + R, and type in taskschd.msc.
Select “Task Scheduler Library” from the left panel. Right click on it and select “Create Task”.
In the “General” tab, give the task a name. Click on the “Change User or Group” button, and add “SYSTEM” as an object name. Then, click “OK” to save the user settings.
Select the “Run with highest privileges” checkbox.
Under the “Triggers” tab, add a trigger to begin the task “At startup”.
In the “Actions” tab, add a new action that starts up the sshd.exe program. In our case, the full path to it will be C:\msys32\usr\bin\sshd.exe.
Under the “Conditions” tab, uncheck all conditions.
Under the “Settings” tab, uncheck the “Stop the task if it runs longer than” option.
Finally, click on “OK” to create a scheduled task.

You’ll find that a task has been created in the Task Scheduler window:

Now, restart your system and try logging in locally. If everything went well, you should be able to log in locally. You can also verify if you can connect from another computer.

Conclusion

Although Windows doesn’t provide satisfactory Unix-like tooling by default, there are projects like MSYS2 which address this nicely. If you’ve followed through this article, you now have a system that allows you to work on an Unix-like system on Windows.

Have you run into any problems while setting this up? Do you know of other methods to set up an Unix-like system on Windows? Let us know in the comments below!

How to Run Android Apps on Your Computer With Android-x86

Mon, 17 Apr 2017 13:37:03 +0000

Android has turned out to be the most popular OS on mobile devices. However, desktop OSes such as Windows, Mac or Linux don’t support running Android apps natively. While there are emulators such as Bluestacks, they’re filled with ads and hogs your computer. In this article, we’re going to look at an alternative way of running Android apps on your computer — using Android-x86 and VirtualBox.

Step 1: The prerequisities

Before you begin, there’s a few things we need to do:

Install VirtualBox

VirtualBox is a program that helps you to run different OSes on top of your existing OS, such Windows, Mac or Linux. Since Android-x86 is a full-blown OS, you should install VirtualBox so that you can use it alongside your existing OS.

So go ahead, download VirtualBox and install it on your computer.

Download Android-x86

You can download an ISO file of Android-x86 here. It comes in both a 32-bit (x86) and a 64-bit(x86_64) variant. In this guide, we’re going to use the android-x86-6.0-r1.iso file, which is the latest 32-bit release.

Step 2: Setting up the virtual machine

Now, we will create a virtual machine(VM) inside VirtualBox. Open VirtualBox and click on the “New” icon. You’ll get a dialog box asking for the details of the VM. Give it a name (we’ll call ours “Android”), and use the following settings:

Operating system type: Linux
Operating system version: Linux 2.6/3.x/4.x (32-bit)
Memory size: 1024 MB or more
Hard disk: When you’re asked for hard disk settings, select “Create a virtual hard disk now” and opt for a “Dynamically allocated” hard disk file. Set the hard disk size to 10 GB or more.

At the end, you’ll find that a new VM named “Android” has been created.

Step 3: Installing Android-x86

Power on the VM by double-clicking on it. You’ll be asked to add an ISO file to boot from. Here, you can select the Android-x86 ISO file you’ve downloaded.

If this option doesn’t show up for you, hit F12 when the virtual machine starts to boot. Then, you can right click on the CD icon in the status bar to add an ISO file. Then, press the key shown on screen to boot from CD. (In our case, the key is “c”.)

When the VM boots up, select the installation option with the up/down arrows and press Enter to continue.

Now, you need to create a partition to install Android. Select the “Create/Modify Partitions” option. Then, select “No” to skip using GPT.

You’ll be taken to a text-based partition editing tool. Use the left/right arrow keys to select the “New” option, and hit Enter. Then, choose the “Primary” option and hit enter. You’ll be asked for the size of the partition. The default is fine, so hit Enter again.

Since we’ll be installing an OS to the partition, we have to mark it as bootable. Select the “Bootable” option and hit Enter. Finally, the partitioning scheme will look something like this:

Select the “Write” option to save your changes. You’ll have to type in “yes” to confirm. Finally, you can quit the partition editor using “Quit”.

Now, you have to select the partition where Android-x86 should be installed. The default is fine, so you can press Enter.

Select the “ext4” filesystem, and select “Yes” to format the disk.

Select “Yes” to install the GRUB bootloader. Then, select “No” to skip installing the EFI version of GRUB as we won’t need it.

Then, select “Yes” to install the /system partition as read-write. This is very useful for trying out root-based apps.

It will take a few minutes for the installation to complete. When it’s done, right click the CD icon in the status bar and select “Remove disk from virtual drive”. Then, select “Reboot” and press Enter. If you don’t do this, you’ll end up booting from the ISO again, instead of the hard disk. (If you’ve already booted into the ISO again, simply press Right Ctrl + R to force a reboot.)

When the reboot completes, Android should ask you for the initial setup, just like a regular smartphone or tablet.

Step 4: The initial setup

You might find a few quirks while trying to set up Android. For example, when you try to click within the window, you’ll find that the mouse pointer doesn’t show up. To fix this, you can disable mouse integration from the “Input” menu.

After you disable it, simply click once within the VirtualBox window to use the mouse. When you’re done, hit the Right Ctrl key to free the mouse pointer from the window.

If your screen is too small to fully show the Android screen, hit Right Ctrl + F to go full screen. You can exit from the full screen mode by pressing Right Ctrl + F again. If the full screen mode is not to your tastes, you can use the “Scaled Mode” by pressing Right Ctrl + C.

You can now complete the setup process by signing in to your Google account. During the setup process, you may get a warning saying that there’s no WiFi connection; you can ignore this safely.

At the end of the process, you’ll be taken to the home screen.

You can now install apps from the Play Store and surf the internet. While most applications work fine, there may be the occasional app which refuses to run inside Android-x86.

Conclusion

Using Android-x86 and VirtualBox is a great way to enjoy Android applications on your computer. Please leave a comment below to let us know of your results installing and running Android-x86!

Adding HTTPS to Your Site with Cloudflare

Mon, 10 Apr 2017 18:34:54 +0000

Much of the web works on HTTP; but it is insecure protocol as it provides no protection. It’s secure sibling, HTTPS provides protection against eavesdropping and tampering of data. Historically, HTTPS has been used for financial transactions and other situations involving transmitting highly sensitive information.

However, recently, there has been a push to move all websites to HTTPS; probably caused by egregious violations of privacy by ISPs and governments. In fact, Firefox and Chrome regard HTTP websites as insecure. Apart from the security benefits, Google uses HTTPS as a ranking signal. Thus, it has become very important for websites to use HTTPS.

In order to deploy HTTPS, you need a SSL certificate that’s signed by a Certification authority(CA). There are various CAs who can give you such certificates for free or for some amount of money. However, configuring HTTPS is often a pain. Instead, we’ll look at deploying HTTPS with a Content Delivery Network(CDN), Cloudflare. It is a very simple technique of adding HTTPS and it only takes a few minutes.

How Cloudflare’s HTTPS works

Simply put, Cloudflare manages all your DNS records by providing a nameserver. To use Cloudflare, you need to switch to Cloudflare’s nameservers instead of using the ones provided by your host. After setting it up, all traffic is sent to Cloudflare’s servers, and requests are proxied to your web server via Cloudflare. Similarly, the response is proxied from your server to Cloudflare’s servers, and then sent to the user. HTTPS connections are made between the user and Cloudflare’s server, and traffic is sent to your server in plain HTTP. Thus, it allows you to deploy HTTPS without the complexity of certificates.

As we mentioned in a previous article on CDNs, deploying Cloudflare will also help you reduce your server bandwidth usage and protecting it against attacks. Again, the protection is due to the proxying of requests we mentioned above.

The initial setup

Deploying HTTPS is easy with Cloudflare. If you don’t have an account, you should sign up first.

Now, log in to your account, and click on “Add site” link on top. This should bring up a web page like the one below:

Type in your domain name below (without a prefix like “www.”) and hit the “Begin Scan” button. Cloudflare will now try to detect the DNS settings for your subdomain. Once the scan finishes, click on the “Continue Setup” button.

Now, you’ll get a list of all the subdomains and other DNS settings that had been configured. In our example, we’ve used a domain which already had various subdomains configured. If you’re using a domain name with no subdomains configured in it, you can add them in this step.

The DNS entries with a grey cloud icon are the ones which bypass Cloudflare’s network, whereas the ones with an orange cloud icon use Cloudflare’s network. At this point, if some web-related domains are not autodetected, click on the grey icon to make it orange. Keep in mind that Cloudflare can work for HTTP traffic only, so if you want to use protocols such as FTP or SSH, add DNS entries such as “ftp” or “direct” to the DNS configuration, by adding it through the form below. Be sure to click on the orange cloud icon to make it gray, otherwise direct access won’t be possible.

If there are some entries which point to such services (like the “ftp” server example above), be sure to leave them grey clouded.

After setting up the DNS records, click on the green “Continue” button at the bottom of the page. You’ll be asked to select a Cloudflare plan. Select the free plan and click on “Continue”.

Now, you will be given a list of nameservers to use, like the example below:

Now, log in to your domain registrar’s control panel, and replace the existing nameservers with those provided by Cloudflare. The screenshot below is from the domain registrar we’ve used; it will look different for you depending upon who your domain registrar is.

Next, come back to the Cloudflare page and hit continue. You may get a warning like the one shown below. If this happens, then click on the “Recheck Nameservers” button.

As stated in the screenshot above, the process of traffic rolling over to the new nameservers may take some time (usually it takes around 10 minutes to 2-3 hours). This is because DNS records are cached by various DNS servers and it may take some time for the new DNS records to be propagated. During this time, your server will be accessible over HTTP but not with HTTPS. When the new nameservers come into effect, you will be able to access your site with HTTPS.

You can try opening https://<your_domain_name> to check if this process has already occured.

Fixing misconfigured server apps

When you tried to open the HTTPS version of your website, you may have seen that some images and scripts refused to load. This is because the applications on the server have been configured to use HTTP, but you’re using HTTPS.

The instructions to fix this issue depend upon the web application you’re using. For example, if you’re using WordPress, you should first switch back to the HTTP version of the site in your browser. Then, you can log in Wordpress and change it from the “General Settings” panel. Here, you should set the site URL to use “https” instead of “http”.

For most other web applications, the process should be similar. In other cases, you might have to change a configuration file by hand by logging into your server.

Fixing mixed content warnings

In addition to configuring your web application, you may find some external scripts and images which are accessed over HTTP that break your website. If you have a lot of content on your website, fixing it one-by-one is not a feasible solution. You can use nginx’s nginx_substitutions_filter or Apache’s mod_substitute to rewrite all the HTTP URLs in a web page to HTTPS.

Redirecting to the HTTPS version of your website

Now that you’ve sorted out these issues, you should redirect users to the HTTPS version of your website. Otherwise, visitors would still get the HTTP version when they visit your site. Thus, your users won’t the security benefits of HTTPS.

Fortunately, in this case, you need not deal with redirection rules for your server — Cloudflare allows you to set up page rules to do this. Head over to the “Page Rules” tab and click on “Create Page Rule”. A dialog box like the one below should come up:

You can add an URL pattern like http://*<your_website_name>* (for example, in our case, http://*xn--138h.ml* to match all URLs that can be accessed. Next, click on “Add a Setting” and select “Always use HTTPS” from the dropdown menu. Finally, click “Save and Deploy” in order to enable these redirections.

Conclusion

HTTPS is of prime importance in a day where the internet is mostly a hostile place with attackers trying to tamper and eavesdrop on your connections. Services such as Cloudflare make the painful process of deploying HTTPS simpler and cost-effective.

Building a Spam Filter Using Machine Learning

Mon, 27 Mar 2017 17:13:52 +0000

Machine learning is everywhere. From self driving cars to face recognition on Facebook, it is machine learning behind the scenes that drives all of it. If you’ve ever used GMail or Yahoo Mail, you must have seen a folder named “Spam” where all unwanted mail goes in. Have you ever wondered how that works? That’s machine learning at work, too!

In this article, we’re going to develop a simple spam filter in node.js using a machine learning technique named “Naive Bayes”. The filter will be able to determine whether an email is spam by looking at its content.

The basics of machine learning

The word “machine learning” has a certain aura around it. We’ll break it down for you — machine learning is a field of computer science where computers can learn to do something, without the need to explicitly program them for the task. First, the algorithm is made to look at a certain set of data, in order to train it for the task. Then, we give the algorithm data it has never seen before, and perform the task on this data.

Thus, a machine learning algorithm can be thought to have two phases: “training” and “prediction”. For each of these phases, we use various mathematical methods.

There are a wide variety of machine learning algorithms. Depending upon how these algorithms “learn”, they can be categorized as:

In supervised learning, the algorithm is provided with data, along with the correct answer for it. So, if we were to develop an algorithm to predict house prices, and you gave the size of the land and the price to the algorithm, it would fall into this category.
In unsupervised learning, the algorithm is provided with data, but the answers are not provided to it. It is upon the algorithm to find structure in the data, and figure out things from there. They are commonly used in places such as market segment analysis. We don’t know what kind of market segments are there for your product — and the algorithm must figure it out.

Again, based on the type of output that a machine learning algorithm produces, we can categorize them into two types:

Classification: These algorithms produce outputs that categorize the data. For example, an algorithm which takes in medical information about a patient and produces a diagnosis that may be only one of “no cancer”, “lung cancer” or “colon cancer” would be of this type.
Regression: In regression, the output types are continuous valued. For example, consider the previous example of predicting house prices. The predicted price would depend on the size of the land. Unlike regression, we don’t have outputs that nicely categorize the data.

As you’ll see later in this article, we’ll train our filter using a collection of spam and non-spam(aka “ham”) emails. So, we’ll provide right answers to train the filter, and later in the prediction phase, its output for a given message would be either “spam” or “ham”. So, this filter is an example of a supervised classification algorithm.

Now that you know the types of machine learning algorithms, let us see what we would need to build the filter.

The training phase

As we’ve already said, our filter would analyze the content of an email to tell if it’s spam. Let us now take a deeper look at the problem.

If you take a look at the typical spam email, you’d find words such as “replica”, “loans” and “singles”. These words are hard to come across in ham, though. Similarly, words such as “presentation” and “manager” is typical in ham, but hard to find in spam.

During the training phase, we’d tell our program to look at a set of spam or ham emails. For every distinct word in the email, it would note down the probability with which the word occurs in either spam or ham. In this article, we’ll refer to these values as “spammicity” and “hammicity”. If we were to define spammicity in a more verbose way, we could say it was the probability of a word occurring, given that the mail is spam. We’ll represent this as $P(W|S)$ . Similarly, we could represent hammicity as $P(W|H)$ .

So, for example, you have 10000 emails in your training dataset, out of which 6000 mails are spam and the rest is ham. The word “replica” occurs in 2000 spam emails and 10 ham emails. So, $P("replica"|S) = \frac{2000}{6000} = 0.333$ and $P("replica"|H) = \frac{10}{4000} = 0.0025$ .

You might be thinking as to why we need the hammicity value at all. If both your spam and ham emails contain the word “car” frequently, we’d have high spammicity and hammicity values. If we didn’t take the hammicity of the emails into consideration, then we’d classify every email containing “car” as spam.

Next, we’ll discuss how the prediction phase would work.

The prediction phase

We now know the spammicity and hammicity values for every word. We can also easily find out the probability of a message being spam or ham. Considering our previous example, the probability that of a message is spam is $\frac{6000}{10000} = 0.6$ and the probability of a message being ham is $\frac{4000}{10000} = 0.4$ . We’ll refer to these probabilities as $P(S)$ and $P(H)$ , respectively.

Now, let’s say, we’ve got a new email containing an arbitrary word $W$ in it. Our job is to find out whether the message is spam, given that it contains $W$ . This is an exact opposite situation of finding the spammicity, and can be represented as $P(S|W)$ .

Here’s where we can use Bayes Theorem, and calculate the probability as:

P(S|W) = \frac{P(W|S)P(S)}{P(W|S)P(S) + P(W|H)P(H)}

Continuing with the “replica” example, the probability of the message being spam would be:

P(S|"replica") = \frac{{0.333 \times 0.6}}{{0.333 \times 0.6 + 0.0025 \times 0.4}} = 0.9925

So far, we’ve considered a single word. An email consists of multiple words, and we have to use Bayes theorem once again to find out the overall probablility of it being spam. This model assumes all words are equally likely to appear at any given position in the message. However, due to the grammar rules of a language, this never happens; so this model is less than perfect. However, it is good enough for our purposes, and many spam filters use this model due to its simplicity.

It may so happen that the filter may encounter a new word when classifying an unknown email. We’d first discard these words, because there’s no way we can make predictions about them. After this, assume there are n distinct words in the email, $W_{1}, W_{2}, ..., W_{n}$ . Then, we find out $p_{1} = P(S|W_{1}), p_{2} = P(S|W_{2}), ..., p_{n} = P(S|W_{n})$ . We can then find out the overall probability of the email being spam like so:

p = \frac{p_{1}p_{2}...p_{n}}{p_{1}p_{2}...p_{n} + (1 - p_{1})(1 - p_{2})...(1 - p_{n})}

If p contains a sufficiently large value (say, > 0.5), we’ll assume the email to be spam.

A few practical considerations

Before we write the code, there are a few practical considerations to make so that our filter can work better.

In the formula for $P(S|W)$ , we’ve considered the values for $P(S)$ and $P(W)$ . However, in real situations, $P(S)$ can be high as 0.8, which would lead to misclassifications of ham emails as spam. Given a new email, we have no prior reason to suspect it may be spam, so we’ll use $P(S) = P(W) = 0.5$ . In other words, we assume it is equally probable for the message to be either spam or ham. Thus, the formula for $P(S|W)$ would get reduced to:

P(S|W) = \frac{P(W|S)}{P(W|S) + P(W|H)}

There’s another optimization we can make. In every email, you’d find some common words like “if”, “be”, “then” and so on. There’s no need to consider these words because they aren’t helpful to find out if the message is spam.

Now, take a look at the formula for calculating the overall probability the email being spam. Probability values always lie between 0 and 1, and computers don’t handle multiplying such small values well. As a result, to avoid errors creeping into the value of p, we rewrite the formula like so:

p = \frac{1}{1 + e^{\eta}}

where:

\eta = \sum_{i = 1}^{n}\left [ log_{e}(1 - p_{i}) - log_{e}(p_{i})\right ]

(You can read the derivation if you’re interested.)

Writing the classifier

Enough with the theory! We’ll now write the spam filter. We’ll begin off by writing:

#!/usr/bin/env node

"use strict";

const fs = require("fs");

let classifier = {
};

Here, we’ve used the fs module, which helps us to read from files and write to them. The rest of the code discussed in this section would go into the classifier object. This object will be responsible for learning from the training examples and for classifying new emails.

Now, we need to define a data structure to hold information about previously seen emails. For this purpose, we define a dataset object like so:

dataset: {
 total_spam: 0,
 total_ham: 0,
 spammicity: {},
 hammicity: {}
},

Initially, we have seen exactly zero spam and ham emails, and we don’t know anything about the words in an email. This is why total_spam and total_ham are initialized to zero, and the spammicity and hammicity objects are empty. After training, the spammicity and hammicity objects will contain words and their corresponding values like the example below:

spammicity: {
 "replica": 0.8831255,
 "watches": 0.910244
}

We would also need to load the training data from the disk and save it when we’re done. In order to do this, we define two functions, load() and save(). These functions save this data as JSON into training.json.

load: function() {
 try {
 this.dataset = JSON.parse(fs.readFileSync(`${__dirname}/training.json`, "utf-8"));
 }
 catch (e) {
 // nothing to do here.
 }
},

save: function() {
 fs.writeFileSync(`${__dirname}/training.json`, JSON.stringify(this.dataset));
},

Now, say, we’ve loaded the data of a spam email into a string. We have to ignore the common words in this string, and create a data structure consisting of all the distinct words inside the string. For this purpose, we’ve defined a function, createTable():

createTable: function(str) {
 let table = {}, result, word;
 let rgx = /\b([a-z]{2,}-)*[a-z]{3,}/gi;

 while ((result = rgx.exec(str)) !== null) {
 word = result[0].toLowerCase();

 if (!this.isCommonWord(word)) {
 table[word] = true;
 }
 }

 return table;
},

This function returns and object, which contains the list of words like so:

{
 "hello": true,
 "world": true
}

At this point, you might be thinking, why not use an array? Using an array would require checking to ensure there are no duplicate elements in it. Using the key-value association of Javascript objects is a much simpler and faster way of doing this, because duplicate keys cannot exist in an object.

The regular expression /\b([a-z]{2,}-)*[a-z]{3,}/gi fetches words with more than two letters and phrases (like “out-of-the-box”) from the string. Words with less than three letters are ignored because it would match words like “an”, “by” and “of” which are very common, and aren’t helpful for our purpose.

Despite this initial level of filtering, words such as “the” and “for” would still get through. Thus, we’ve defined another function, isCommonWord(), which filters out common names and some common words.

isCommonWord: function(str) {
 return /^(?:the|and|that|have|for|not|with|you|this|but|his|from|they|we|say|her|she|will|one|all|would|there|their|what|out|about|who|get|which|when|make|can|like|time|just|him|know|take|people|into|year|your|good|some|could|them|see|other|than|then|now|look|only|come|its|over|think|also|back|after|use|two|how|our|work|first|well|way|even|new|want|because|any|these|give|day|most|ever|among|stand|yet|often|hour|talk|might|start|turn|help|big|small|keep|old|out|high|low|ask|should|down|thing|aaron|adam|alan|albert|alice|amanda|amy|andrea|andrew|angela|ann|anna|anne|annie|anthony|antonio|arthur|ashley|barbara|benjamin|betty|beverly|billy|bobby|bonnie|brandon|brenda|brian|bruce|carl|carlos|carol|carolyn|catherine|charles|cheryl|chris|christina|christine|christopher|clarence|craig|cynthia|daniel|david|deborah|debra|denise|dennis|diana|diane|donald|donna|doris|dorothy|douglas|earl|edward|elizabeth|emily|eric|ernest|eugene|evelyn|frances|frank|fred|gary|george|gerald|gloria|gregory|harold|harry|heather|helen|henry|howard|irene|jack|jacqueline|james|jane|janet|janice|jason|jean|jeffrey|jennifer|jeremy|jerry|jesse|jessica|jimmy|joan|joe|john|johnny|jonathan|jose|joseph|joshua|joyce|juan|judith|judy|julia|julie|justin|karen|katherine|kathleen|kathryn|kathy|keith|kelly|kenneth|kevin|kimberly|larry|laura|lawrence|lillian|linda|lisa|lois|lori|louis|louise|margaret|maria|marie|marilyn|mark|martha|martin|mary|matthew|melissa|michael|michelle|mildred|nancy|nicholas|nicole|norma|pamela|patricia|patrick|paul|paula|peter|philip|phillip|phyllis|rachel|ralph|randy|raymond|rebecca|richard|robert|robin|roger|ronald|rose|roy|ruby|russell|ruth|ryan|samuel|sandra|sara|sarah|scott|sean|sharon|shawn|shirley|stephanie|stephen|steve|steven|susan|tammy|teresa|terry|theresa|thomas|timothy|tina|todd|victor|virginia|walter|wanda|wayne|william|willie)$/.test(str);
},

Learning from examples

It’s time to define the learnSpam() function. It takes the contents of an email as a string, gets the list of words with createTable(), and adjusts the dataset values.

learnSpam: function(str) {
 let table = this.createTable(str);
 let total = this.dataset.total_spam;

 for (let word in this.dataset.spammicity) {
 let old_spammicity = this.dataset.spammicity[word];

 if (table[word] === undefined) {
 this.dataset.spammicity[word] = (total * old_spammicity)/(total + 1);
 }
 else {
 this.dataset.spammicity[word] = (total * old_spammicity + 1)/(total + 1);
 delete table[word];
 }
 }

 for (let word in table) {
 this.dataset.spammicity[word] = 1/(total + 1);
 }

 this.dataset.total_spam = total + 1;
},

This function is a bit complex, so let’s take an example. Suppose, our classifier has seen five emails. Three of them contain “replica” and two of them contain “loans”. Thus, we would have:

P("loans"|S) = \frac{2}{5} = 0.4

P("replica"|S) = \frac{3}{5} = 0.6

Now, suppose we have an email which reads: “replica watches”. The value of total_spam and the denominators of all the fractions would be incremented by 1. Now:

The filter has never seen “watches” but it occurs in the new message. So, the numerator will be equal to one, because after the training, the filter would have seen it exactly once.
The filter has seen “loans”, but it does not occur in this message. So, the new numerator won’t increase; it will be equal to the previous numerator. However, since we didn’t store the old numerator in the dataset, we have to calculate it indirectly with total * old_sapmmicity.
The word “replica” occurs both in the new message and in the previously seen messages. Thus, we must add one to the old numerator.

We’ll get these new values, and you can manually verify that these are indeed correct:

P("watches"|S) = \frac{1}{5 + 1} = \frac{1}{6} = 0.167

P("loans"|S) = \frac{0.4\times 5}{5 + 1} = \frac{2}{6} = 0.333

P("replica"|S) = \frac{0.6\times 5 + 1}{5 + 1} = \frac{4}{6} = 0.666

The learnHam() function is similar, except for the fact that it works on hammicity values.

Making predictions

The predict() function creates a list of words with createTable(), and calculates $P(S|W)$ for each word. It then combines them with the alternative formula for p we discussed above. We indicate that the message is spam by returning true when the p > 0.5.

predict: function(str) {
 let table = this.createTable(str), word, p = {};

 for (word in table) {
 let spammicity = this.dataset.spammicity[word] || 0;
 let hammicity = this.dataset.hammicity[word] || 0;

 if (spammicity !== 0 && hammicity !== 0) {
 p[word] = spammicity/(spammicity + hammicity);
 }
 }

 let eta = 0;

 for (word in p) {
 eta += (Math.log(1 - p[word]) - Math.log(p[word]));
 }

 let p_final = 1/(1 + Math.pow(Math.E, eta));

 return (p_final > 0.5);
}

When a spammicity or hammicity entry cannot be found for a given word, we assume it to be zero. This might seem strange, but it is the right thing to do. For example, if the word “replica” never occurs in ham messages, there would be no entry in the hammicity table. By assuming a hammicity of zero, we get $P(S|"replica") = \frac{1}{1 + 0} = 1$ , which is the right answer.

If both of spammicity and hammicity values are zero, then, we’ll ignore the word as it has never been seen by the filter.

The driver program

This completes our classifier. Now, we need some code to read files and to call the functions defined in classifier. This code should be defined outside the classifier object and below it. The code is fairly self-explanatory, and we won’t describe it here.

let arg_length = process.argv.length;

if (arg_length === 2) {
 console.log(`Usage: ${process.argv[0]} ${__filename} {-s|-h|-p} {filename}`);

 process.exit(0);
}

let actions = ["-s", "-h", "-p"];
let type = actions.indexOf(process.argv[2]);

if (arg_length === 3 || type === -1) {
 console.log(`Incorrect arguments, please run "${process.argv[0]} ${__filename}" for help.`);
 process.exit(1);
}

classifier.load();

switch (type) {
 case 0:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 classifier.learnSpam(file_contents);
 }

 classifier.save();
 break;

 case 1:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 classifier.learnHam(file_contents);
 }

 classifier.save();
 break;

 case 2:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 console.log(process.argv[i] + ": " + classifier.predict(file_contents));
 }
 break;
}

The functions of the classifier can now be accessed with the help of the command line switches. Assuming that you saved the file as spamfilter.js, the following functions will be available:

node spamfilter.js -s <list of files> trains the filter to learn the contents of the given files as spam.
node spamfilter.js -h <list of files> trains the filter to learn the contents of the given files as ham.
node spamfilter.js -p <list of files> predicts if the content of each given file is spam or ham.

The entire program

The source for the entire program is as below.

#!/usr/bin/env node

"use strict";

const fs = require("fs");

let classifier = {
 dataset: {
 total_spam: 0,
 total_ham: 0,
 spammicity: {},
 hammicity: {}
 },

 load: function() {
 try {
 this.dataset = JSON.parse(fs.readFileSync(`${__dirname}/training.json`, "utf-8"));
 }
 catch (e) {
 // nothing to do here.
 }
 },

 save: function() {
 fs.writeFileSync(`${__dirname}/training.json`, JSON.stringify(this.dataset));
 },

 isCommonWord: function(str) {
 return /^(?:the|and|that|have|for|not|with|you|this|but|his|from|they|we|say|her|she|will|one|all|would|there|their|what|out|about|who|get|which|when|make|can|like|time|just|him|know|take|people|into|year|your|good|some|could|them|see|other|than|then|now|look|only|come|its|over|think|also|back|after|use|two|how|our|work|first|well|way|even|new|want|because|any|these|give|day|most|ever|among|stand|yet|often|hour|talk|might|start|turn|help|big|small|keep|old|out|high|low|ask|should|down|thing|aaron|adam|alan|albert|alice|amanda|amy|andrea|andrew|angela|ann|anna|anne|annie|anthony|antonio|arthur|ashley|barbara|benjamin|betty|beverly|billy|bobby|bonnie|brandon|brenda|brian|bruce|carl|carlos|carol|carolyn|catherine|charles|cheryl|chris|christina|christine|christopher|clarence|craig|cynthia|daniel|david|deborah|debra|denise|dennis|diana|diane|donald|donna|doris|dorothy|douglas|earl|edward|elizabeth|emily|eric|ernest|eugene|evelyn|frances|frank|fred|gary|george|gerald|gloria|gregory|harold|harry|heather|helen|henry|howard|irene|jack|jacqueline|james|jane|janet|janice|jason|jean|jeffrey|jennifer|jeremy|jerry|jesse|jessica|jimmy|joan|joe|john|johnny|jonathan|jose|joseph|joshua|joyce|juan|judith|judy|julia|julie|justin|karen|katherine|kathleen|kathryn|kathy|keith|kelly|kenneth|kevin|kimberly|larry|laura|lawrence|lillian|linda|lisa|lois|lori|louis|louise|margaret|maria|marie|marilyn|mark|martha|martin|mary|matthew|melissa|michael|michelle|mildred|nancy|nicholas|nicole|norma|pamela|patricia|patrick|paul|paula|peter|philip|phillip|phyllis|rachel|ralph|randy|raymond|rebecca|richard|robert|robin|roger|ronald|rose|roy|ruby|russell|ruth|ryan|samuel|sandra|sara|sarah|scott|sean|sharon|shawn|shirley|stephanie|stephen|steve|steven|susan|tammy|teresa|terry|theresa|thomas|timothy|tina|todd|victor|virginia|walter|wanda|wayne|william|willie)$/.test(str);
 },

 createTable: function(str) {
 let table = {}, result, word;
 let rgx = /\b([a-z]{2,}-)*[a-z]{3,}/gi;

 while ((result = rgx.exec(str)) !== null) {
 word = result[0].toLowerCase();

 if (!this.isCommonWord(word)) {
 table[word] = true;
 }
 }

 return table;
 },

 learnSpam: function(str) {
 let table = this.createTable(str);
 let total = this.dataset.total_spam;

 for (let word in this.dataset.spammicity) {
 let old_spammicity = this.dataset.spammicity[word];

 if (table[word] === undefined) {
 this.dataset.spammicity[word] = (total * old_spammicity)/(total + 1);
 }
 else {
 this.dataset.spammicity[word] = (total * old_spammicity + 1)/(total + 1);
 delete table[word];
 }
 }

 for (let word in table) {
 this.dataset.spammicity[word] = 1/(total + 1);
 }

 this.dataset.total_spam = total + 1;
 },

 learnHam: function(str) {
 let table = this.createTable(str);
 let total = this.dataset.total_ham;

 for (let word in this.dataset.hammicity) {
 let old_hammicity = this.dataset.hammicity[word];

 if (table[word] === undefined) {
 this.dataset.hammicity[word] = (total * old_hammicity)/(total + 1);
 }
 else {
 this.dataset.hammicity[word] = (total * old_hammicity + 1)/(total + 1);
 delete table[word];
 }
 }

 for (let word in table) {
 this.dataset.hammicity[word] = 1/(total + 1);
 }

 this.dataset.total_ham = total + 1;
 },

 predict: function(str) {
 let table = this.createTable(str), word, p = {};

 for (word in table) {
 let spammicity = this.dataset.spammicity[word] || 0;
 let hammicity = this.dataset.hammicity[word] || 0;

 if (spammicity !== 0 && hammicity !== 0) {
 p[word] = spammicity/(spammicity + hammicity);
 }
 }

 let eta = 0;

 for (word in p) {
 eta += (Math.log(1 - p[word]) - Math.log(p[word]));
 }

 let p_final = 1/(1 + Math.pow(Math.E, eta));

 return (p_final > 0.5);
 }
};

let arg_length = process.argv.length;

if (arg_length === 2) {
 console.log(`Usage: ${process.argv[0]} ${__filename} {-s|-h|-p} {filename}`);

 process.exit(0);
}

let actions = ["-s", "-h", "-p"];
let type = actions.indexOf(process.argv[2]);

if (arg_length === 3 || type === -1) {
 console.log(`Incorrect arguments, please run "${process.argv[0]} ${__filename}" for help.`);
 process.exit(1);
}

classifier.load();

switch (type) {
 case 0:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 classifier.learnSpam(file_contents);
 }

 classifier.save();
 break;

 case 1:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 classifier.learnHam(file_contents);
 }

 classifier.save();
 break;

 case 2:
 for (let i = 3; i < arg_length; i++) {
 let file_contents = fs.readFileSync(process.argv[i], "utf-8");
 console.log(process.argv[i] + ": " + classifier.predict(file_contents));
 }
 break;
}

Training the filter

Now that we’ve developed our filter, we now need to train it. Fortunately, there are already many email datasets available for this purpose. We’ll be using the CSDMC2010 dataset in this article.

Download the ZIP archive attached to the file and extract it. We’ll do this through the command line, although you’re free to do this through a GUI if that’s what you’re comfortable with.

$ wget -O CSDMC2010_SPAM.zip "http://csmining.org/index.php/spam-email-datasets-.html?file=tl_files/Project_Datasets/task2/CSDMC2010_SPAM.zip"
$ unzip CSDMC2010_SPAM.zip

The dataset contains a set of training files and labels (spam/ham) and a set of test files containing emails. These emails are contained in EML files, and we need to extract the bodies of these emails. In order to do this, you should pull in the mailparser module with:

$ npm install mailparser

Next, save the script below as eml2txt.js. This script takes in the name of a directory containing EML files, decodes them and saves them in another directory.

#!/usr/bin/env node

"use strict";

let arg_length = process.argv.length;

if (arg_length === 2) {
 console.log(`Usage ${process.argv[0]} ${__filename} {eml-dir} {txt-dir}`);
 process.exit(0);
}

if (arg_length !== 4) {
 console.log(`Incorrect arguments, run "${process.argv[0]} ${__filename}" for help.`);
 process.exit(1);
}

const fs = require("fs");
const path = require("path");
const simpleParser = require('mailparser').simpleParser;

try {
 let res = fs.statSync(process.argv[3]);

 if (!res.isDirectory()) {
 console.log(`${process.argv[3]} is not a directory.`);
 process.exit(1);
 }
}
catch (e) {
 fs.mkdirSync(process.argv[3]);
}

fs.readdir(process.argv[2], function(err, files) {
 if (err) {
 console.log(`Failed to open input directory: ${process.argv[2]}`);
 process.exit(1);
 }

 let l = files.length;

 for (let i = 0; i < files.length; i++) {
 let file = files[i];
 let abs_file = path.join(process.argv[2], file);

 fs.stat(abs_file, function(err, stat) {
 if (err) {
 console.log(`Error reading file: ${abs_file}`);
 return;
 }

 if (stat.isFile(abs_file)) {
 fs.readFile(abs_file, function(err, content) {
 if (err) {
 console.log(`Error reading file: ${abs_file}`);
 return;
 }

 simpleParser(content, function(err, mail) {
 fs.writeFile(path.join(process.argv[3], file), mail.html || mail.text, function(err) {
 if (err) {
 console.log(`Error writing file: ${file}`);
 }
 });
 });
 });
 }
 });
 }
});

Let us now convert the training and test EML files to text files.

$ node eml2txt.js CSDMC2010_SPAM/CSDMC2010_SPAM/TESTING/ testing/
$ node eml2txt.js CSDMC2010_SPAM/CSDMC2010_SPAM/TRAINING/ training/

You’ll find two new directories, training and testing. These folders contain the decoded files. Now, we can train our filter by using the labels from the SPAMTrain.label file provided. Files with a label of 0 are spam, while those with a label of 1 are ham.

$ sed -nr 's#^0 (.*)#training/\1#p' CSDMC2010_SPAM/CSDMC2010_SPAM/SPAMTrain.label | xargs node spamfilter.js -s
$ sed -nr 's#^1 (.*)#training/\1#p' CSDMC2010_SPAM/CSDMC2010_SPAM/SPAMTrain.label | xargs node spamfilter.js -h

The training might take some time. When it finishes, you’ll find a file, training.json, which contains the dataset object, saved by the call to classifier.save().

Testing the filter

Now that you’ve trained the filter, it’s now time to test it. You can test it out on random emails with:

$ node spamfilter.js -p $(printf "testing/TEST_%05d.eml" $((RANDOM % 4292)))

Feel free to test this on a few messages. You’ll find that the filter gets it right most of the time. If we train it on a more comprehensive dataset, the results would improve. In addition, you can also play around with the threshold value of 0.5 in predict() and see if it improves things.

Improving the design

To keep things easy to understand, we’ve kept our spam filter simple. However, we could add a variety of bells and whistles to make it more accurate. For example, in an actual implementation, we could assume a higher value of $P(S)$ if the email originates from an IP address known for sending out spam prolifically.

Again, our filter is not resilient to letter substitutions that a spammer could make. For example, the spammer could write “replica” as “rεplica”. Humans can still read it, but our implementation would have trouble detecting this as a word. However, even with a filter that can detect it as a word, “replica” might still be considered different than “rεplica”. An actual implementation could detect words with such mixed character sets and automatically assign them high spammicity values, even though the modified word has never been seen before.

There are many more ways a determined spammer could work their way around our filter. For example, they could write their spam messages like so:

<p><!-- A short spam message --></p>
<div style="display: none;"><!-- The entire Wikipedia article about horses.--></div>

The <div> containing the article about horses is hidden from the user, but it can be still seen by the filter. Unfortunately, due to the large amount of legitimate text, the value of p would be low, and the message would be marked as ham. A robust spam filter would probably have its own HTML and CSS parser, remove invisible regions from the text, and find out p for the remaining text.

Conclusion

This was a really long article. If you’ve made it this far: congratulations on building your first machine learning based spam filter! Let that sink in — you designed an algorithm and showed it examples of spam and ham messages. After a bit of training, the algorithm has learned how to distinguish between them!

It is no wonder thus that machine learning is making inroads everywhere. Performing tasks without the need for programming things explicitly is what makes machine learning so powerful.

8 Critical Security Issues to Avoid in Your Website

Mon, 20 Mar 2017 18:28:18 +0000

A web application, being exposed to the web, is far more prone to be attacked than other kinds of applications. Building a secure web application is of prime importance — a hacked website costs money and time, impacts your business prospects and harms your customers.

Building a secure web application is not difficult at all. The security issues that arise are often due to simple mistakes that can be easily avoided.

In this article, we’re going to discuss the most common vulnerabilities found in web applications, and how to prevent them.

Issue #1: Cross site scripting (XSS)

In XSS, the web application sends back data entered by the user without any filtering. Thus, an attacker can enter JavaScript into the input parameters, causing their JavaScript to run on the user’s browser. This can be used to steal cookies (allowing the attacker to log in as the authenticated user), redirect the user on a malicious page, and more.

Let’s have a look at a simple demonstration of XSS attack. Let’s say, we have a file named demo.php with the following content:

Hello <?php echo $_GET['name']; ?>

Note how the script displays user input without filtering it. You can inject arbitrary scripts by navigating to an URL like:

http://your-website.example/demo.php?name=<script>alert("Hacked");</script>

Typing the above URL in a browser would result in:

Obviously, a real attacker would do something more malicious, such as injecting a script which steals the user’s cookies. The attacker could then send it to users, who are compromised by opening the link.

This attack could appear in other forms too. Consider a comment form on a website: an attacker could submit a script as a comment. This script will now be executed by anyone who happens to visit the page.

The proper way to prevent XSS is to convert characters such as < and > to their respective character entities, such as < and >. If you’re storing the data as with the comment form example, you should only do this while displaying the page. You should not store the data in the entity-encoded form.

If you’re using PHP, you can use htmlspecialchars() for this purpose. If we rewrite our above PHP script like so:

Hello <?php echo htmlspecialchars($_GET['name']); ?>

The attack will no longer work, and instead display:

Hello, <script>alert("Hacked");</script>

If you’re using a templating engine or a framework, this is already handled for you.

The XSS prevention cheat sheet by OWASP covers this in great detail, including cases where conversion to HTML entities may not be enough.

Issue #2: SQL injection

SQL injection occurs when data entered by the user is put into an SQL statement directly. The user-entered data could contain SQL statements, thus allowing the attacker to bypass authentication, steal information and modify your databases.

Consider the login form of a website. To check for successful authentication, a website could do something along the lines of:

result = db.query("SELECT * FROM users WHERE username = '" + request.post("username") + "' AND password = '" + request.post("password") + "'")

For a normal user, the SQL query would look like:

SELECT * FROM users WHERE username = 'admin' AND password = '$tr0ngpa$$w0rd';

However, an attacker could enter an username such as hacker' OR 1 = 1 --. This results in the following query being executed:

SELECT * FROM users WHERE username = 'hacker' OR 1 = 1 -- AND password = 'password';

In this case, the attacker is able to bypass authentication entirely, by changing the query that was executed.

The proper way to prevent a SQL injection attack is to use parameterized queries. It is a way of executing database queries where the data and the query are kept separate, thus preventing this problem entirely. Most database drivers have this feature, so you should switch to it already if you’ve been using string concatenation instead.

In addition, if you use an ORM (Object-Relational Mapper) for accessing the database, you should also be fine. Most ORMs use parameterized queries to generate their underlying queries.

Issue #3: Problems in authentication and session management

Developing your own authentication system could give rise to a number of security issues. While discussing all of them would be outside of the scope of this article, here are the most common problems:

Not preventing username enumeration: When there is an authentication failure, the error should not suggest whether the username or password was incorrect — instead, simply say something like “Incorrect username or password.” This prevents attackers from finding out which users have accounts on your service.
Not regenerating sessions: When an user is logged in or logged out, you should regenerate the session identifier associated with the user, in order to prevent a “session fixation” attack. Suppose, an attacker logs into a website from a public computer, notes the session identifier, and then logs out. When another user logs in from the same computer, the same identifier is being reused. Thus, the attacker can send requests from another computer with the same identifier to determine if another user has logged in. This allows for a complete compromise of the user’s account.
Allowing unlimited login attempts: You should place limits on the number of times an user can try logging in within a certain period of time. Otherwise, your web application would be vulnerable to brute force attempts. The most user-friendly way is to require users to fill in a CAPTCHA before logging in if they’ve already tried doing so too many times. In other high-security applications, you could lock out IPs or user accounts if you’ve had too many attempts.
Not checking the use of insecure passwords: Many web applications do not have rules in place to prevent users from keeping insecure passwords, such as the infamous password or 123456. While most users find complicated password rules frustrating, you can add simple checks like rejecting passwords that match the 1000 most common ones, or passwords that are the same as the user’s username or email address.

Issue #4: Proper storage of passwords

Storing sensitive information such as passwords in plaintext is one of the worst mistakes that a developer can make. This is because if your database is stolen (such as through SQL injection or by an insider), the passwords will be served on a silver platter to the attacker. Since there are many users who reuse passwords across websites, this could lead to all of their accounts being hacked.

Many websites suggest using a cryptographic hashing function, however, even this is insecure — they are too fast to provide sufficient resistance to an attacker who is trying to iterate across millions of passwords to see if one of them matches the hashed password.

The proper way to store passwords is by using a strong password hashing function, such as bcrypt, along with a “salt” added to it. A “salt” is a random value that is added while hashing the password. When the password is stored, the hash is stored along with it, in plaintext. The salt simply serves to change the hash, so that even if two users chose the same password, the hashed password would vary. Thus, an attacker would be unable to tell if two users passwords are the same, and make educated guesses as to what the password may be.

For more details, take a look at this video Marty Weiner presenting at QCon:

Issue #5: Cross Site Request Forgery (CSRF/XSRF)

A CSRF attack involves making an user perform actions on a website without their knowledge.

Consider a banking website where people can make money transfers. An attacker notices the fact that money transfers make use of the following URL:

https://bank.com/moneytransfer?to=alice&amount=300

Let’s say, the attacker, Eve wants to steal money from a person, Bob. Bob makes financial transactions all the time, and he has the bank’s website open most of the time. Eve sends a mail to Bob, which contains a link containing pictures of cute cats. However, in addition, to all those pictures, the page also contains the following HTML:

<img src="https://bank.com/moneytransfer?username=eve&amount=10000">

When the browser tries to fetch this URL, it ends up performing the money transfer, with Bob being none the wiser.

The easiest way to prevent CSRF attacks is by checking the Referer/Origin headers of the request. If they don’t match the name of your website, deny the request.

Another popular prevention technique is to initiate sessions for every user who happens to visit the website. On the server, a long, random string of characters is generated for every session, which is called the “CSRF token”. When a form is presented to the user that performs a sensitive action, this token is embedded as a hidden field:

<input type="hidden" name="csrf_token" value="cas11L2bcutotsSjnp2a5mOveGI">

When the form is submitted, the value of the hidden field and the value stored on the server is compared. If they do not match, the request is rejected.

If you’re using a framework, this should be automatically handled for you with just a few lines of code.

Issue #6: Failure to restrict URL access

There are some URLs in a web application which should be restricted to a certain set of users. For example, only administrators should be able to access administration pages; users should not be able to change other’s profile information, and so on. If these pages can be accessed by other categories of users, then the website is vulnerable.

When building a web application, you must be careful so as not to allow unauthorised users access to resources that are not meant for them.

Issue #7: Unvalidated redirects

Redirections are commonplace on websites. If the redirection occurs on the basis of a parameter present in the HTTP request, it could be misused for various purposes.

For example, there are many websites which redirect an user to a web page after performing an action. For example, many websites allow the user to return to the same page after logging in — and in order to do so, they use URLs of the form:

http://example.com/login?redirect=http://example.com/videos/

An attacker could replace the redirect parameter with something like:

http://example.com/login?redirect=http://otherwebsite.com/

While this seems far more innocuous than the other vulnerabilities, it can still be useful for deceiving users. For example, it could be used to redirect the user to a phishing website after logging in. Most users would have a hard time detecting it. After all, they did enter their username and password on the original website!

Preventing this attack is easy. If the redirection parameter contains an URL which doesn’t point to your website, don’t redirect them to that page.

Issue #8: Uploading server-executable scripts

If users can upload files through your web application, you must take care so as to not allow uploading of PHP and CGI scripts. An attacker could simply upload a PHP script, and then access it, thereby gaining access to your server.

You can prevent this attack from happening in a variety of ways. Two of them are as follows:

Restricting the types of files that the user can upload: Say, you’re building a website for uploading images. You can check the MIME-type of the uploaded file. You should not rely upon the file extension or the MIME-type value provided by the browser. Instead, you should check the actual structure of the file to see if it’s an image. In PHP, you can use finfo_file(). If it isn’t an image, you can reject it.
Disabling scripts in the upload directory: You can configure your server to exclude the upload directory from executing CGI and PHP scripts. This will allow users to even upload such scripts. When the user tries to visit the URL to the uploaded script, they will be able to see the contents as-is, just like a text file.

However, if you’re using something like NodeJS or Django to build your web applications, and/or you don’t have PHP or CGI enabled on your server, you won’t need to check for this.

Conclusion

Security is an aspect which if ignored can cause serious repercussions. If you’ve read through all these steps and taken care of the issues involved, your web application is now more secure!

Customizing and coloring the bash prompt

Mon, 13 Mar 2017 10:21:00 +0000

If you’re already not familiar with bash, the prompt is the little blurb of text displayed when it asks for a command. It capable of communicating a lot, but most people don’t use it quite fully. Do you want to add some color to the rather bland CLI interface? Or do you need to get information like the current git branch you’re on? This article will show you how to spruce up bash’s shell prompt.

A very basic prompt

To keep things simple, let us write out a very simple prompt – we will display >>> instead of the usual prompt. Type in the following at the very end of your ~/.bashrc file:

PS1='>>> '

To see the changes in action, fire up a new terminal window, or you can type in source ~/.bashrc in an existing terminal window.

You will now see something like this:

>>> _

Congratulations, you’ve just customized your bash prompt. So, how does that work? The bash prompt is controlled by a variable named PS1, and we’ve just set it to »>. Setting the PS1 variable is all you need to do to customize your prompt.

By the way, if you want to make these changes available for all users on the system, you need to use /etc/bash.bashrc (on Debian and Ubuntu) or /etc/bashrc (on other distributions) instead of ~/.bashrc.

Having that sorted out, let us see how to make more useful custom prompts.

Escape sequences

Bash allows you to use some shortcuts to retrieve details, such as the user name, the machine name, the current time, and so on. These shortcuts are called escape sequences.

Let’s say, you want to display the user’s name and the current directory, followed by a $. It can be retrieved with the \u and \w escape sequences respectively. Thus, our PS1 would look like:

PS1='\u \w $ '

This ends up looking like this:

john ~ $ _

If you wanted to display the current time in addition to the other details, and display the $ on another line entirely, you could do something like so:

PS1='[\t] \u \w\n$'

And sure enough, our prompt has it:

[01:13:55] john ~
$ _

The \t shows the current time, and the \n moves everything after it (in this case, the $) to the next line.

Here’s a list of the most commonly used escape sequences: \uThe current user\wThe current working directory\WThe last fragment of the current working directory. For example, if you are currently in /usr/local/bin, this will give you bin.\hThe name of the computer, upto a dot( .). For example, if your computer is named ubuntu.pc, this gives you ubuntu.\HThe full hostname\dThe date in “Weekday Month Date” format (e.g.“Tue 21 July”)\tThe current time in 24 hour HH:MM:SS format\TThe current time in 12 hour HH:MM:SS format\@The current time in 12-hour AM/PM format\nMove on to the next line.

Adding color and styling

Most people would like some color on the rather dull shell prompt. This can be achieved with the help of ANSI escape sequences in the PS1 variable. These escape sequences need to be enclosed in \[ and \] in order to work properly.

Suppose, you want to show the username in red and the directory path in cyan, followed by a yellow, bold $ symbol. The escape sequences for red is \e[31m , for cyan, it is \e[36m, and for yellow, it is \e[33m. For bold text, we need \e[1m. In addition, we need the reset ANSI escape sequence, which prevents styles from affecting the rest of the text in the shell. The reset sequence is \e[0m.

The PS1 for this case would look like this:

PS1='\[\e[32m\u\] \[\e[36m\w\] \[\e[33m\]\[\e[1m\]$ \[\e[0m\]'

The prompt ends up looking like this:

john ~ $ _

There are a lot of styles and colors you can come up with. Have a look at the entire list of ANSI escape sequences to see what’s possible.

Advanced customization with command substitution

The customization discussed so far are pretty basic in nature. What if we want to do more advanced things?

It turns out that we can assign values of PS1 conditionally by using the command substitution construct. Text enclosed within a $(...) is evaluated as a command, and then, its output is put directly its place.

A pretty common use case would be to display the current git branch you’re on in your project directory. You can easily get the current branch with:

git branch 2>/dev/null | sed -n "s/* \(.*\)/\1/p"

If you’re already familiar with command substitution, you might be tempted to write something like this:

PS1="\u \w $(git branch 2>/dev/null | sed -n "s/* \(.*\)/\1 /p")$ "

However, this won’t work. This is because, the command substitution construct is evaluated at the time of reading .bashrc. When you move across directories or change git branches, the command substitution isn’t re-evaluated.

However, it turns out that if the $(...) is present as-is inside the PS1 variable, it will be re-evaluated every time the bash prompt is shown. If you’re using single quotes, it’s easy to do this:

PS1='\u \w $(git branch 2>/dev/null | sed -n "s/* \(.*\)/\1 /p")$ '

When you’re using double quotes, you need to escape the $ symbol like so:

PS1="\u \w \$(git branch 2>/dev/null | sed -n "s/* \(.*\)/\1 /p")$ "

Phew! So now that we’ve sorted out how to get things working, let us see how things actually look like. When you are in a directory that is not a git repository, the prompt will appear as:

john ~ $ _

However, when you are in a directory containing a git repository, the prompt will change:

john ~/my-awesome-project master $ _

That was a rather simple example. However, it is possible to take this concept further and design more complex prompts.

Example: smileys on the prompt

Say, you want to display a green :) if the previous command succeeded, and a red :( if it failed. The $? variable holds the return value of the previous program — it contains a 0 if it succeeded, and a non-zero value if it failed. You could do something like:

PS1='$(if [[ $? == 0 ]]; then echo "\[\e[32m\]:)"; else echo "\[\e[31m\]:("; fi)\[\e[0m\] \u \w $ '

The $(if ... fi) block contains the logic for displaying the smileys.

The prompt does exactly what we want:

:) john ~ $ touch a.txt
:) john ~ $ rm b.txt
rm: cannot remove 'b.txt': No such file or directory
:( john ~ $

Unicode characters

Unicode is a standard way of representing text in a compatible way across various platforms. Don’t let the bit about “text” fool you though — it can even represent symbols and emojis!

To display a Unicode character in the prompt, you should first know its hexadecimal (hex) representation. Given a particular Unicode character, you can get its hex representation by using xxd or hexdump -C like so:

$ printf '✓' | xxd
00000000: e29c 93                                  ...

$ printf '✓' | hexdump -C
00000000  e2 9c 93                                          |...|
00000003

As you might have guessed from the output, the hex representation of ‘✓’ is \xe2\x9c\x93.

Now that you know the hex representation, you can display it with:

PS1=$'\xe2\x9c\x93 \u \w $ '

This will give you a prompt that looks like:

✓ john ~ $ _

In the above example, note the $ followed by the single quotes. If you leave out the $ or put the PS1 content in a different type of quote, it won’t work.

If you want to display them conditionally in a command substitution construct, using printf would be the way to go:

PS1='$(if condition; then printf "\xe2\x9c\x93"; fi)'

We’ll have a look at a practical example of this.

Example: we ❤ emoji

We’ll, who doesn’t love emoji? Since emojis are Unicode characters, they could be placed in the prompt too!

Let us make an emoji variant of our smiley prompt. is represented as \xf0\x9f\x98\x83, whereas is represented as \xf0\x9f\x98\x9c. Here’s the code for it:

PS1='$(if [[ $? == 0 ]]; then printf "\xf0\x9f\x98\x83"; else printf "\xf0\x9f\x98\x9c"; fi)\[\e[0m\] \u \w $ '

You can now bask in the glory of emoji on your terminal:

 john ~ $ touch a.txt
 john ~ $ rm b.txt
rm: cannot remove 'b.txt': No such file or directory
 john ~ $

However, depending upon the terminal emulator and the font used, you may see monochrome emoji or even garbled characters.

Customizing other prompts

Here’s something that most people don’t know — bash has other prompts, and they can be customized too! We have already seen how we can control the main prompt by setting the PS1 variable. Similarly, the PS2 variable controls the prompt that’s shown while entering a multi-line command.

The default multi-line prompt is simply a >, and you can see this by typing:

$ ls \
> -lR

Try modifying PS2 and see it in action:

$ PS2='==> '
$ ls \
==> -lR

Armed with these tricks, you can now fully customize your bash prompt!

If you’re looking for inspiration for customizing your prompt, have a look at this old revision on prompt customization and another user’s notes from the Arch Linux wiki. If you have interesting bash prompts to share, post them in the comments below!

Protecting SSH with Fail2ban

Mon, 06 Mar 2017 12:17:06 +0000

SSH allows you to log in to a remote computer or server and control it through a command-line interface. However, because SSH is exposed to the internet; attackers can try to log in by trying various username and password combinations.

A good way to protect SSH would be to ban an IP address from logging in if there are too many failed login attempts. You can use a package called “fail2ban” for this purpose, and it works with minimal configuration. In addition, you can even configure Fail2ban to protect other applications, like web servers.

We’ll cover how to protect SSH with Fail2ban in this post. These instructions should work on modern versions of Ubuntu (16.04 and later), Debian and CentOS 7.

Installing Fail2ban

On Debian and Ubuntu, you can install Fail2ban with:

sudo apt update
sudo apt install fail2ban

On CentOS, you should first enable the EPEL repository and install Fail2ban.

sudo yum -y install epel-release
sudo yum -y install fail2ban

Now, create /etc/fail2ban/jail.local with a text editor such as nano:

sudo nano /etc/fail2ban/jail.local

In the text editor, type in the following text:

[sshd]
enabled = true
banaction = iptables-multiport

Save the file contents and exit the text editor. Then, enable the Fail2ban service and start it up with:

sudo systemctl enable fail2ban
sudo systemctl restart fail2ban

That’s it! With this minimal configuration, Fail2ban will block an IP for 10 minutes if it notices five failed logins occurring in a 10-minute period.

The basics of Fail2ban

So how does that work? Put simply, Fail2ban is a daemon that monitors logs and takes actions based on their contents. It is driven by three types of configuration:

Filters specify certain patterns of text that Fail2ban should recognize in log files.
Actions are things Fail2ban can do.
Jails tell Fail2ban to match a filter on some logs. When the number of matches goes beyond a certain limit specified in the jail, Fail2ban takes an action specified in the jail.

Fail2ban comes with a jail instructing it to look at system logs and take actions against attacks on SSH. The default action (which is discussed later in detail) adds iptables rules to block out attackers.

In the previous section, we set the “enabled” setting of the “sshd” jail to true, so that Fail2ban can block these attacks. In the next section, we will see how you can configure some aspects of Fail2ban.

Configuring the jail

While the defaults will suffice for most, sometimes you may need to change things so that it suits your needs better.

Say, you want to block IP addresses for a day, if they have tried to make 10 login attempts within 12 hours. In order to do this, open the /etc/fail2ban/jail.local file. Previously, we had set the “enabled” setting in this file. Now, we will add the following settings to the end of the file:

maxretry = 10
findtime = 43200
bantime = 86400

The file would end up looking like this:

[sshd]
enabled = true
banaction = iptables-multiport
maxretry = 10
findtime = 43200
bantime = 86400

Here:

maxretry controls the maximum number of allowed retries.
findtime specifies the time window (in seconds) which should be considered for banning an IP. (43200 seconds is 12 hours)
bantime specifies the time window (in seconds) for which the IP address will be banned (86400 seconds is 24 hours).

If you have moved SSH to a different port, you can also specify the port number with port = <port_number> in this file.

Restart Fail2ban for these changes to come into effect with the following command:

sudo systemctl restart fail2ban

Configuring the action

In the “Installing Fail2ban” section, we set up Fail2ban to use the “iptables-multiport” action. By default, it rejects packets with a “port unreachable” message. If you want to drop packets instead of rejecting them, create /etc/fail2ban/action.d/iptables-common.local and type in the following:

[Init]
blocktype = DROP

If you want to take an entirely custom action, you have to first define an action file in /etc/fail2ban/action.d/my-action.local. (You can name the file anything you want, as long as it has the .local extension.) A typical action file looks like this:

[Definition]
actionstart = ...
actionstop = ...
actioncheck = ...
actionban = ...
actionunban = ...

As their names suggest, the actionstart and actionstop lets you specify initialization commands that will be run when Fail2ban starts up and shuts down. actionban and actionunban specify the commands which will be run when Fail2ban wants to block an IP. In addition, Fail2ban may also need to verify whether an IP is already blocked, and the command specified in actioncheck helps it to do so.

To see a practical example of this, have a look at the /etc/fail2ban/action.d/iptables-multiport.conf file.

When specifying these commands, you can use the <ip> placeholder to get the IP address of the attacker.

After you’ve written your action file, in the SSH jail defined in /etc/fail2ban/jail.local, you can set the action by setting the following:

banaction = my-action

Blocking repeat offenders with multiple jails

As a server administrator, you may have noticed that there are some IPs that keep brute forcing over and over, despite the bans. You could work around this by setting a low value of maxretry and findtime and a high value of bantime, but this risks locking out legitimate users. Fortunately, you can block these attacks by setting up multiple jails.

For our example, we’re going to define a jail named sshlongterm. It will block an IP for a week if there are 35 failed attempts login attempts over 3 days.

However, before we get started, we need the configuration of the default jail. Open the /etc/fail2ban/jail.conf and look for a line containing only [sshd]. The default configuration is mentioned below this line. On Ubuntu 16.04, the default configuration looks like this:

[sshd]
port = ssh
logpath = %(sshd_log)s

Copy these rules over to /etc/fail2ban/jail.local, under the definition of the sshlongterm jail. The file should end up looking like this:

[sshd]
; rules for the default SSH jail

[sshlongterm]
port = ssh
logpath = %(sshd_log)s

Now, you need to set up the primary rules for the jail. This involves setting up the filter type, as well as the values for maxretry, findtime and bantime. Add the following lines:

banaction = iptables-multiport
maxretry = 35
findtime = 259200
bantime = 608400
enabled = true
filter = sshd

These settings allow up to 35 failed logins over 3 days before blocking the IP for a week.

The complete jail would end up looking like this:

[sshlongterm]
port = ssh
logpath = %(sshd_log)s
banaction = iptables-multiport
maxretry = 35
findtime = 259200
bantime = 608400
enabled = true
filter = sshd

Restart fail2ban for these changes to take effect.

The default jail as well as the sshlongterm jail should now work in conjunction. Short term attacks will be handled by the default jail, and the long term attacks handled by our own jail.

Conclusion

As we’ve seen, Fail2ban protects SSH right away. With a little bit of configuration, it can make massive brute force attacks a trivial problem.

What is a CDN and How to Use It

Mon, 27 Feb 2017 06:18:30 +0000

You may have come across the buzzword called CDN and may have probably wondered what it is. In this article, we are going to demystify this term and explain how to use it for a variety of benefits on your website.

Content Delivery Network (CDN) is a network of servers distributed across many places around the globe. Their job is to serve content to visitors. But, the website’s server does that too - " Serving Content". So what is the difference?

A CDN stores a copy of your website at multiple places around the globe so that the nearest server to the visitor serves the content. This reduces the latency to reach a very distant server considerably improving the performance. It can be understood by a simple example: Suppose your origin web server is in Singapore and someone at Paris wants to access your website. The data packets have to go through several layers across different territories before reaching the destination. So, the visitor has to wait for a long time before the page starts to display. This can be very frustrating, and often visitors leave if the delay is more than 3 seconds. Now, in the case of a CDN, the website can be served from the server that is nearest to Paris. So, the data packets need not go through multiple networks and can be served directly.

The concept has been demonstrated above in the cover photo. On the left, we have a conventional system where one server serves to all the visitors around the world and on the right, there is a distributed approach where the host connects to the nearest CDN server.

Infographic by KeyCDN

Benefits of a CDN:

Using a CDN has numerous benefits other than speed. Let’s look at some of them

Speed and Performance - We already discussed how a CDN could drastically increase the rendering time of a website. In short, the CDN servers store a cached copy of the site. The visitors connect to their nearest CDN server and fetch the content rather than going all the way up to the origin server. For the sites which have dynamic content, like online shopping websites, social networking or microblogging sites, caching is done using various TCP tweaks like cache control headers or edge side includes. Therefore, a CDN ensures a super fast web experience to the end users.
Security - The ever rising cyber crimes and DDoS attacks are costing organisations millions of dollars. Most of the leading CDNs provide protection against these attacks. They provide web application firewall, SSL certificates, rate limiting and more tools to secure the websites.
Availability - Whenever the origin server goes down or becomes unavailable for some reason, the CDN will serve cached static pages in the meantime. It will also act as a load balancer when your website’s traffic is very high which would not have been possible by only one origin server. The CDN can handle tonnes of terabytes of data per second which is also the reason they can handle massive DDoS attacks.
IPV6 compatible - You can make your website IPV6 compatible even if your origin server doesn’t support IPV6. It has a host of benefits like more efficient routing and better packet processing than the conventional IPV4
HTTP/2 enabled - Nearly all the CDNs today have enabled HTTP2 by default. It is a worthy upgrade over the previous version of the protocol which offers features like multiplexing, header compression and prioritization. The following video by Akamai explains it all.

How to use a CDN:

Setting up a CDN is fairly simple for simple websites like blogs, product/company pages or other sites which do not have very high dynamic content. You just need to change the nameservers of your site to the one that is provided by the CDN company. Below is a short video showing the process of signing up in CloudFlare.

embed

https://www.cloudflare.com/video/cloudflare-setup.mp4

/embed

For dynamic sites like e-commerce, the process is a bit tedious. You need to specify which resources you want to cache and which ones to serve from the origin. Also, you may want to host your images and videos on the CDN server. Do note that this can affect your SEO since the URLs will change. To solve it, you need to set up CNAME records for better-looking links.

CDN companies:

There are various CDN companies to choose from and it all depends on your requirements. These are some of the popular CDN companies:

Cloudflare - This is the most beginner friendly CDN and is great for all types of websites. The best thing about Cloudflare is that it offers a free plan that includes a free shared SSL certificate too. Cloudflare ranks #1 in terms of popularity among CDN companies. They have a massive network comprising of 100+ data centers spread across the world. Setting up Cloudflare is also very easy, and people with no experience can also do it. You can start off with the free plan and later upgrade to the $20/month Pro plan which offers web application firewall and other extra features.
Incapsula - It is also a relatively popular CDN and also includes a free plan to start with. The free plan does not include free SSL, unlike CloudFlare.
BunnyCDN - It is a new player in the market, and not many people know about it yet. BunnyCDN offers the lowest rate in the industry for premium services - $0.01-0.03/GB of traffic. You can literally serve a terabyte of traffic for only $10. This can be ideal for people who want premium features but are budget constraint.
Stackpath (Previously MaxCDN) - It offers premium services and is in the market for a lot of years. They have an ever growing network of data centers around the world with reasonable and clear pricing. Stackpath uses Anycast routing for one-to-nearest content delivery over multiple connections.
Akamai - It is a CDN designed for enterprise organisations. According to Wikipedia, their network is one of the world’s largest distributed computing platforms, responsible for serving between 15 and 30 percent of all web traffic. They have customised plans for each client and are not available publicly.

Some of the other popular CDNs include Amazon’s CloudFront, KeyCDN, fastly, EdgeCast, CDN77, Azure Content Delivery Network and CacheFly.

Test your CDN

After having set up your CDN, you can see how well it performs at various locations across the globe. Head over to Pingdom or webpagetest.org and enter your website’s URL. Then test it across the different locations given. You should see a notable difference in latency.

Conclusion

Having a fast site is very crucial for keeping the visitors happy. CDN plays a key role in speeding up a website and securing it. In addition to CDN, other technologies like Accelerated Mobile Pages should also be used for an array of benefits among mobile users. Which CDN are you using and what are your experiences? Do let us know in the comments down below!

Downloading Torrents over the Web with cloud-torrent

Mon, 20 Feb 2017 07:08:25 +0000

BitTorrent is a peer to peer protocol for sharing files. There are many local clients that allow you to download files via torrents. However, there are times where you may need to download them over the web because you can’t use a local torrent client.

cloud-torrent is a web based torrent-client. It allows you to search torrent trackers, add torrent files and magnet links, and download them on your computer.

In this article we will look at installing cloud-torrent on a server running Linux. You can even run it on your own computer if it is publicly accessible with an IP address.

A word of caution though: if you’re planning to run torrent applications on your server, you may want to confirm whether your provider allows this.

Download and run cloud-torrent

Download the correct version of cloud-torrent from the releases page. Binaries are available for Intel 32- and 64-bit variants, as well as for ARM. In the example below, we have downloaded the 64-bit Linux version:

wget https://github.com/jpillora/cloud-torrent/releases/download/0.8.24/cloud-torrent_linux_amd64.gz

Having downloaded the archive, you have to unpack it, rename the binary and make it executable. To do that, run the following commands:

gzip -d cloud-torrent*.gz
mv cloud-torrent* cloud-torrent
chmod +x cloud-torrent

Then, run the binary:

./cloud-torrent

Now, visit http://your_server_name:3000/ with a browser. If you can see the web interface, that’s great! However, if it’s not working for you, check that:

You’re not blocking traffic on port 3000 – you need to change your firewall configuration in that case.
You’re not running an existing service on port 3000. If this is the case, cloud-torrent will exit with an error listen tcp 0.0.0.0:3000: bind: address already in use. You can run cloud-torrent on a different port, like so:
```
./cloud-torrent -p <custom_port_number>
```
The port number must be anywhere between 1025 and 65535.

To stop cloud-torrent, simply press Ctrl+C.

Setting up cloud-torrent

While cloud-torrent is already working, you probably want it to be started automatically on system startup like a regular system service. You can do this quite easily.

First, you need to set up a separate user for cloud-torrent to run. While it is not strictly necessary to do so, it is generally recommended for better security.

sudo useradd -rm cloud-torrent

Next, you have to move the cloud-torrent binary to the home directory of the new user.

sudo mv cloud-torrent /home/cloud-torrent/cloud-torrent

Now, you need to configure the system to start it automatically. In order to do so, you need to find out the init system that’s in use. To do this run (source):

if [[ `/sbin/init --version` =~ upstart ]]; then echo upstart;
elif [[ `systemctl` =~ -\.mount ]]; then echo systemd;
elif [[ -f /etc/init.d/cron && ! -h /etc/init.d/cron ]]; then echo using sysvinit;
else echo cannot tell; fi

If the output is upstart or sysvinit, you should use the instructions for sysvinit. If the output is systemd, you should use the systemd instructions.

systemd-based systems

With a systemd-based system, this process is quite easy. Create /etc/systemd/system/cloud-torrent.service with the following contents:

[Unit]
Description=torrent web server

[Service]
Type=simple
User=cloud-torrent
WorkingDirectory=/home/cloud-torrent
ExecStart=/home/cloud-torrent/cloud-torrent

[Install]
WantedBy=multi-user.target

If you need to use a different port, you need to put in ExecStart=/home/cloud-torrent/cloud-torrent -p <port_number> instead.

Then, you need to tell systemd that you’ve created a new service file and also set it to run at boot:

sudo systemctl daemon-reload
sudo systemctl enable cloud-torrent.service

Finally, you should start the service immediately so that you can access cloud-torrent:

sudo systemctl start cloud-torrent.service

sysvinit/upstart-based systems

For sysvinit or upstart based systems, you need to add a shell script with the following code in /etc/init.d/cloud-torrent:

#!/bin/sh
### BEGIN INIT INFO
# Provides: cloud-torrent
# Required-Start: $local_fs $network $named $time $syslog
# Required-Stop: $local_fs $network $named $time $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the cloud-torrent web server
# Description: starts cloud-torrent using start-stop-daemon
### END INIT INFO

DESC="torrent web server"
NAME=cloud-torrent
DAEMON=/home/cloud-torrent/cloud-torrent

DAEMONUSER=cloud-torrent
PIDFILE=/var/run/$NAME.pid

STOP_SCHEDULE="${STOP_SCHEDULE:-QUIT/5/TERM/5/KILL/5}"

test -x $DAEMON || exit 0

start() {
 start-stop-daemon --start --quiet --make-pidfile --pidfile "$PIDFILE" \
 -d "/home/$DAEMONUSER" --background --chuid "$DAEMONUSER" --oknodo --exec "$DAEMON"
}

stop() {
 start-stop-daemon --stop --quiet --pidfile "$PIDFILE" --retry="$STOP_SCHEDULE" \
 --name "$NAME" --oknodo
 rm -f "$PIDFILE"
}

status() {
 if [ -f "$PIDFILE" ]; then
 if kill -0 $(cat "$PIDFILE"); then
 echo "$NAME is running"
 else
 echo "$NAME process is dead, but pidfile exists"
 fi
 else
 echo "$NAME is not running"
 fi
}

case "$1" in
 start)
 echo "Starting $NAME $DESC..."
 start
 ;;
 stop)
 echo "Stopping $NAME $DESC..."
 stop
 ;;
 restart)
 echo "Restarting $NAME $DESC..."
 stop
 start
 ;;
 status)
 status
 ;;
 *)
 echo "Usage: $0 {start|stop|restart|status}"
 exit 2
 ;;
esac

exit 0

If you need to run cloud-torrent on a different port, simply modify start() in the above shell script like so:

start-stop-daemon --start --quiet --make-pidfile --pidfile "$PIDFILE" \
 -d "/home/$DAEMONUSER" --background --chuid "$DAEMONUSER" --oknodo --exec "$DAEMON" -- -p <port_number>

Then, you need to make the script executable:

sudo chmod +x /etc/init.d/cloud-torrent

Then, you should enable the service to run at boot. We will also start the service immediately. If you use Ubuntu or Debian, run:

sudo update-rc.d cloud-torrent defaults
sudo service cloud-torrent start

However, if you are using CentOS or Fedora, it is a bit more involved. First, you have to install the start-stop-daemon program:

wget https://raw.githubusercontent.com/boolean-world/misc-tools/master/install-start-stop-daemon.sh
sudo bash install-start-stop-daemon.sh

Now, you can enable the service at boot with the following commands:

sudo chkconfig --add cloud-torrent
sudo chkconfig --level 2345 cloud-torrent on

Finally, you can start the service immediately with:

sudo service cloud-torrent start

Exploring cloud-torrent’s interface

If you have followed the steps above, cloud-torrent should be up and running! You can visit http://<your_server_IP>:3000/ to verify this. Now, you can download torrents by pasting magnet links or torrent file links in the text box. In addition, you can enter a search term to search for torrents on common torrenting websites.

You can configure seeding and encryption settings by clicking on the three bars icon. (See the screenshot below.)

Password protection

The current setup has a problem — anyone who visits it can use it. You can prevent this by setting a password by specifying an environment variable named AUTH.

If you’re using systemd, edit /etc/systemd/system/cloud-torrent.service and put this line in the [Service] section:

Environment="AUTH=username:password"

Then, restart the cloud-torrent service with:

sudo systemctl daemon-reload
sudo systemctl restart cloud-torrent

For sysvinit based systems, simply export the AUTH variable like you would do in a shell script:

# [previous part of shell script]
DAEMONUSER=cloud-torrent
PIDFILE=/var/run/$NAME.pid
export AUTH="username:password"
# [rest of the shell script]

Replace username and password with your own values, and then restart with:

sudo service cloud-torrent restart

Configuring HTTPS

To protect your privacy, you may want to configure HTTPS on cloud-torrent.

First, you need to obtain a certificate. If you don’t have one, you can obtain one through Let’s Encrypt. In the segments below, you should replace /path/to/key.pem and /path/to/cert.pem with the correct paths. If you’re using Let’s Encrypt, you can find these files under the /etc/letsencrypt directory.

For systemd users this will be easy. In /etc/systemd/system/cloud-torrent.service, modify the line containing ExecStart= like so:

ExecStart=/home/cloud-torrent/cloud-torrent -k /path/to/key.pem -r /path/to/cert.pem

Again, for sysvinit/upstart users, it is a bit more involved. Open the /etc/init.d/cloud-torrent file and change the start-stop-daemon command in start() like so:

start-stop-daemon --start --quiet --make-pidfile --pidfile "$PIDFILE" \
 -d "/home/$DAEMONUSER" --background --chuid "$DAEMONUSER" --oknodo --exec "$DAEMON" -- -k /path/to/key.pem -r /path/to/cert.pem

After making these changes, restart the cloud-torrent service as described above. At this point, cloud-torrent should be set up over HTTPS. For more configurable options, be sure to check out the --help switch.

How to Speed up Your Website up to 85% Using AMP

Sun, 12 Feb 2017 13:26:16 +0000

We all have the experience of loading resource intensive sites in patchy networks. The truth is, the pages take forever to load! This has been a big problem for web developers especially if the target country of the website had slow overall internet speeds. The engineers at Google came up with a unique solution called Accelerated Mobile Pages (AMP). Using AMP, pages load just in an instant. In this article, we will see how to enable AMP for your site but first, let’s see how AMP works.

Working of AMP:

The following points illustrate the mechanism behind working of AMP pages.

Block native JS of the page- Javascript is the main culprit for lingering the DOM of the website. AMP modifies this with its custom asynchronous scripts that do not slow down the rendering.
Extension mechanisms can’t block page rendering- extensions like lightboxes, Instagram embeds, tweets, etc. require additional HTTP requests. AMP has its custom script tags for these extensions like amp-iframe so that the DOM is aware that there is an iframe in the page and create an iframe box even before the iframe loads.
Pre-defined size for external resources - Many style recalculations take place when a page is rendered as the CSS dynamically changes as it loads. To avoid this, sizes of the external resources such as ads, images, videos are explicitly defined before they are downloaded.
Sandboxing third-party external javascript- AMP grants access to third party JS only in sandboxed iframes so that they don’t interfere with the main DOM.
Inline CSS - Having external CSS files increases the HTTP requests which slows down rendering. AMP only allows inline CSS to minimise the number of HTTP requests from the critical rendering path, thereby increasing the speed considerably.
Font Optimisations - In general, the font is loaded after most of the CSS and JS files are loaded. AMP makes the font request as the first HTTP request before any other request as soon the page starts loading. So the font loads before the other scripts.
Handling Animations - Animations are restricted to only two methods: transform and opacity. This ensures that the browser does not require page layouts, and the animations are handled by the GPU itself thereby increasing the performance.
The three Ps - Preconnect, Prerender and Prefetch are used in a smart way by AMP to save bandwidth and CPU cycles by only downloading resources above the fold. Using this, a page is rendered before the user visits the page. So when the user visits the page, it instantly loads up!
Progressive loading of resources - AMP loads only the resources that are needed eliminating many unnecessary roadblocks. AMP also prefetches lazy loaded resources when the network and CPU are idle. This makes the content available to the users very quickly and efficiently.

The above points are beautifully presented by Malte Ubl at Google IO 2016 in the following video.

Enough of the working principle. Now let’s see how to use AMP for your website.

The post is split into two parts: The first part will show how to use AMP in WordPress sites and in the second part we will see how to write an AMP friendly website from scratch.

Part 1: AMP for WordPress websites

Using AMP in WordPress is easy as pie. Here are the steps:

Install AMP plugin by Automattic.
(Optional) Install Glue for Yoast SEO & AMP if you are using Yoast SEO plugin. It also has other features like customising the default color options for AMP pages.

That’s it! You have set up AMP on your website. To check if it is working, navigate to any post in your site and add /amp at the end of the URL.

For e.g., If your post has URL like https://example.com/blog/article then replace it with https://example.com/blog/article/amp It should show the AMP version of the page.

For validation see step 3 of part 2 below.

NOTE: The AMP page will not work for the root domain of your WordPress site. It will only work for the post pages since AMP is designed for blogs or news websites which contain a lot of content and not intended to be used for promotional pages featuring products, company websites, or personal portfolio websites.

AMP version of a Boolean World article

Part 2: Building an AMP friendly website from scratch

We will be following along the official Google guide. For more details, you can visit the link given. You must be familiar with HTML. Knowledge of CSS and Javascript is also helpful.

Step 1: Create an HTML file and add the following code to it. This forms the base for our AMP website.
Step 2: To add images to your site, use the following tag: <amp-img src="image.jpg" alt="alternate-text" height="500" width="800"></amp-img> You may have noticed that the conventional <img> tag has been replaced with tag. Many of the conventional tags are replaced with custom amp tags, and many are not allowed in AMP. The reason for this is preloading and prioritising resources For a list of tags that are replaced or disallowed visit here.
Step 3: Validate your code - Now, that the HTML file is created, we need to see if there are any errors in the code. Copy the entire code and visit https://validator.ampproject.org. Remove the contents of the text area and paste your code there. Then click on Validate, and it will immediately show the details for any errors. Alternatively, if you have already published your page you can directly enter the URL in the field given.

Step 4: Discovery of AMP pages - This is one of the most crucial steps that enables Google bot to discover the AMP pages and show them in the search results.
- If you have one AMP and one non-AMP version of the same page, you need to be link them together. In the non-AMP version of the page add the following line of code: <link rel="amphtml" href="https://www.example.com/ampversion.html"> Replace the example.com URL to add the address of your non-AMP version of your page.Now, in the AMP version of your page add the following line of code: <link rel="canonical" href="https://www.example.com/fullversion.html"> Similarly, replace the example.com URL to add the address of your non-AMP version of your page.
- If you have made only one version of the page and that is an AMP version, add the following line of code: <link rel="canonical" href="https://www.example.com/url/to/amp/document.html"> Now perform the previous step of validation again to check for any errors. The AMP page is now ready to publish and should be discoverable by Google.

For further customizations, visit the official guides.

AMP pages in Google search results

Conclusion

Every second in website load time matters. User engagement significantly increases if the loading time is under 3 seconds. In early tests, Google had reported performance improvements from 15% up to 85% (source). It even leads to increased mobile visibility as Google displays AMP results in organic search listings and makes them look appealing in contrast to the non-AMP listings. It has some restrictions, but the advantages extensively override them. Optimising page loads with AMP must surely be on the checklist of every content publisher.

Blocking bad bots with Fail2ban

Sun, 05 Feb 2017 10:57:21 +0000

Fail2ban is a versatile security tool. While it is primarily used for preventing brute-force attacks against SSH, it can also be used for protecting other services.

There are bots which go around scanning the internet and send thousands of requests to web servers in hopes of finding vulnerabilities. This post discusses blocking such bots with Fail2ban.

We assume that you are using Apache as a web server. However, these instructions can be easily adjusted for nginx or any other web server.

However, you should keep in mind that Fail2ban is not a Web Application Firewall (WAF) and cannot fend off malicious requests as they come in through. This is because fail2ban takes actions by monitoring logs; so there must be at least one malicious attempt which gets logged before Fail2ban can take an action.

What is a bad bot, anyway?

In this post, we will focus on blocking bots that do one of the following things:

Scans the website for an open proxy
Sends a GET request with parameters containing SQL injection payloads
Sends a GET request with parameters containing Shellshock payloads

Of course, you can block other kinds of attacks as well. However, we will restrict ourselves to the above three cases for this article.

Installing Fail2ban

Fail2ban is available in the repository of most distributions.

To install it on Debian/Ubuntu, run the following:

sudo apt-get update
sudo apt-get install fail2ban

On CentOS, you should first enable the EPEL repository; then, you should enable and start it.

sudo yum -y install epel-release
sudo yum -y install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Fail2ban basics

At the heart of the working mechanism of Fail2ban, there are a set of jails. Put simply, a jail tells Fail2ban to look at a set of logs, and to apply a filter on it each time the log changes. If the number of matches for the filter equals the maximum number of matches allowed by the jail, then an action specified in the jail is taken.

Thus, you need to define two things: a filter, and a jail. The jail will be configured to look at Apache’s logs to detect malicious requests.

Defining the filters

A filter is simply a collection of Python regular expressions that are matched against a log. Here, we’d need to define filters for the criteria we described above.

But first, let us have a look at an entry in Apache logs:

66.249.79.189 - - [17/Jan/2017:14:10:41 +0000] "GET /robots.txt HTTP/1.1" 200 3494 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:
//www.google.com/bot.html)"

Notice that the request header GET /robots.txt HTTP/1.1 is enclosed in double quotes. While designing such rules yourself, you should take sufficient care to ensure that only the request header matches. Otherwise, you risk blocking legitimate users.

SQL injection payloads generally contain strings of the form union select(...) or select concat (...). Thus, you could try to match this pattern with the following regular expression:

(?i)^<HOST> -.*"[^"]+(?:union[^"]+select[^"]*|select[^"]+concat[^"]*)(?:%%2[8C]|[,(])

The <HOST> part defines the location of the IP address in the log entry, and the (?i) states that the regular expression is case insensitive.

The [^"] in the regex ensures that the matched text is enclosed in double quotes. This helps to ensure that the regular expression matches just the request header and nothing else. The (?:%%2[8C]|[,(]) specifies that the union select or select concat matched is followed by a comma( ,) or a parenthesis( (), either directly or in their percent-encoded form.

Bots scanning for open proxies often send requests of the form of:

101.33.59.9 - - [17/Jan/2017:14:10:41 +0000] "GET http://google.com/ HTTP/1.1" 400 3494 "-" "Mozilla"
101.33.59.9 - - [17/Jan/2017:14:10:44 +0000] "CONNECT yahoo.com:80" 400 3499 "-" "Mozilla"

A regular expression such as the one below can match these easily:

(?i)^<HOST> -.*"(?:(?:GET|POST|HEAD) https?:|CONNECT [a-z0-9.-]+:[0-9]+)

The regular expression (?:(?:GET|POST|HEAD) https?: matches requests of the first type, whereas the regular expression CONNECT [a-z0-9.-]+:[0-9]+ matches requests of the second type.

Bots scanning for Shellshock often send out requests like:

10.11.12.13 - - [17/Jan/2016:16:00:00 +0000] "GET /cgi-bin/printenv.cgi HTTP/1.0" 200 1 "-" "() { test;};echo \"Content-type: text/plain\"; echo; echo; /bin/rm -rf /var/www/"

A regular expression like this will match the pattern for Shellshock:

<HOST> -.*"\(\)\s*\{[^;"]+[^}"]+}\s*;

Here, we are checking for the () { <command>; } pattern, and the \s accounts for whitespaces that may be present in the malicious request.

Combining them, we can now write our filter:

[Definition]

failregex = <HOST> -.*"\(\)\s*\{[^;"]+[^}"]+}\s*;
 (?i)^<HOST> -.*"[^"]+(?:union[^"]+select[^"]*|select[^"]+concat[^"]*)(?:%%2[8C]|[,(])
 (?i)^<HOST> -.*"(?:(?:GET|POST|HEAD) https?:|CONNECT [a-z0-9.-]+:[0-9]+)

ignoreregex =

The ignoreregex allows you to whitelist entries. You can add ignored regexes just like the way it has been done for failregex.

Save the above filter into /etc/fail2ban/filter.d/badbot.local

Defining the jail

Having defined the filter, it’s now time to define the jail. Here, we block the IP address for six minutes, if it sends three such requests within a span of six minutes.

Add this to your /etc/fail2ban/jail.local file:

[badbot]

enabled = true
port = http,https
filter = badbot
logpath = /var/log/apache*/*access.log
maxretry = 3
banaction = iptables-multiport
findtime = 360
bantime = 360

Having configured the jail, you should restart fail2ban for these changes to take effect. Depending on your distribution, one of these commands would be needed to restart it:

sudo systemctl restart fail2ban
sudo service fail2ban restart

fail2ban will now block all attempts made by bad bots trying to attack your web server. You can also extend these rules for your another web server, or for other kinds of attacks that you might see.