A multi automated python project

This article would be an in-depth guide to my project here. The project I undertook is a menu driven python program with various utilities that if dived in deep could be a decent project in themselves. I would be giving an overview of the functionality and also a brief introduction of the technology implemented.

Aim of the project

Using the app the user can:

  1. Perform linux operations
  2. Setup the hadoop
  3. Perform action on AWS
  4. use docker commands
  5. Perform the partitions
  6. Perform LVM partitions
  7. Create and manage an Apache Web Server
  8. One click yum configuration

The given app runs commands based on Redhat Linux. So some features are prone to error when used on some other operating system.

The Setup

1. AWS environment

Since the app is menu driven the user gets a prompt to select the type of service he wants to run. I would be explaining all the options and a certain service of each technology.

Let’s begin with the AWS commands.

Before beginning I would like to discuss about what this is all about. AWS has a dedicated website which can be used to do take advantage of all the services provided by Amazon. However the process of using the WebUI is very slow and tedious hence there is a CLI alternative to it. However when beginning the AWS CLI can be a little overwhelming with so many commands to perform. In this app we use the AWS CLI in the backend but on the front the user only gets to select the options and those are implemented with the speed of CLI.

Let’s say we want to create a new S3 bucket.

We just have to name the bucket and select the region where we want to place the bucket.

Let’s see if this bucket was actually created or not using the website.

We can see that the bucket was created. In the same way the other services can also be implemented.

Let’s try one more. How about setting up a new EC2 instance

So you can see the instance was launched in my account. Similarly the other services can be added too.

2. Hadoop Setup

Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation.

When we have to use the HDFS cluster in hadoop, we have to go through and setup two files and then launch the node in the cluster. Well this task can take some time and becomes hectic when we have to do the same task in so many different systems. Here I created a program that sets up the files according to the need of the user without any hassle in a second. Making the whole process faster.

These are the initial files of the node. We are going to configure this node as datanode and connect it to the namenode without performing any hadoop related command directly.

We can clearly see that right now there is no datanode connected to the namenode.

Here we first updated the HDFS file and then the core-site file, then we moved on to start the datanode. The program creates a folder in the directory for sharing the storage of datanode to the namenode and then finally connects to it. We can check the dfsadmin report for the same.

The similar setup can be done on the namenode as well. A few clicks is all it takes.

3. Docker Setup

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. It eases the use of containers. In a way docker container behaves like virtual machines however they are very light weight and can be launched almost instantly.

Docker containers are isolated from the host system and hence the host is completely safe in case an unwanted bug corrupts the container. All we need to do is shutdown the system and launch a new container.

Our program provides an easy interface to create and launch new containers and view the containers.

Before using the above commands the docker should be setup in the system, however this can be done using configuration which I have explained a little later in this article.

4. Linux Commands

There is an option to run command on the local machine or to run it after logging in into a remote system.

There are various commands available which we use commanly including checking the RAM usage or enabling and disabling firewalls. Disabling firewall might not be the best of choices to make but there are always some use cases when it can come in handy.

5. Partitioning

A single drive having all their resources along with the OS installed. This might not be the best idea to get by.

The major issue that occurs in such a condition is that if the OS crashes it will take down the whole drive and hence it his highly likely that all the sensitive data might be compromised. This is why partitioning and planning is a nice idea when using your system.

In this program I tried to ease this work of partitioning by allowing the user to just select a few options to choose what they wanna do and the partitioning commands are run in the background.

6. Performing LVM partitions

The user is prompted to create a physical volume and a volume group. It is as easy as just entering the name of the folder and files without having to actually implement them through commands.

Then the user can just select the folder in which the storage is to be mounted. I chose a folder by the name of “mounter” and it is allocated 200 MB as I specified as an input.

Before beginning the commands the program implicitly yum installs all the dependencies of using the LVM storage. So a configured yum repository is extremely necessary.

So even that can be configured as per the need as there is provision to add that as well.

7. Configure yum and other dependencies

All the configuration and status check of the commands can be done with just one click . This was a primary need because before using any features having yum configured is extremely necessary.

8. Configuring the webserver

The webserver is needed when we want to host a website or program. I used the apache server to configure it.

The process involved for successfully hosting a server is of 3 steps

  1. Download the software
  2. Configure the server
  3. start the services

Firstly to download the dependencies we will setup the server.

Now we can directly start the services using 2.

To check whether the services have started I will use

systemctl status httpd

Clearly we can see that the services are up and running.

In this article I gave a brief working of the project I have undertaken. You can try forking the project from here if you find it interesting. I have thought of automating it further as I keep learning new skills.

You can catch me up at this location as I am always up for a good healthy talk.

Thanks.

Tech enthusiast