A multi automated python project
This article would be an in-depth guide to my project here. The project I undertook is a menu driven python program with various utilities that if dived in deep could be a decent project in themselves. I would be giving an overview of the functionality and also a brief introduction of the technology implemented.
Aim of the project
The key focus area of the application is to provide a simple interface for the user to interact and perform actions while having little to no knowledge of the setup being created. For example the user can create an instance on AWS he/she without any prior knowledge of AWS CLI or setup hadoop datanode with just a click of a button. The project has an aim to help users achieve more in lesser time.
Using the app the user can:
- Perform linux operations
- Setup the hadoop
- Perform action on AWS
- use docker commands
- Perform the partitions
- Perform LVM partitions
- Create and manage an Apache Web Server
- One click yum configuration
The given app runs commands based on Redhat Linux. So some features are prone to error when used on some other operating system.
The Setup
1. AWS environment
If you have a running OS. All you need to do is install the prerequisite softwares for the technologies we are going to implement and that is gonna be all.
Since the app is menu driven the user gets a prompt to select the type of service he wants to run. I would be explaining all the options and a certain service of each technology.
Let’s begin with the AWS commands.
Before beginning I would like to discuss about what this is all about. AWS has a dedicated website which can be used to do take advantage of all the services provided by Amazon. However the process of using the WebUI is very slow and tedious hence there is a CLI alternative to it. However when beginning the AWS CLI can be a little overwhelming with so many commands to perform. In this app we use the AWS CLI in the backend but on the front the user only gets to select the options and those are implemented with the speed of CLI.
Let’s say we want to create a new S3 bucket.
We just have to name the bucket and select the region where we want to place the bucket.
Let’s see if this bucket was actually created or not using the website.
We can see that the bucket was created. In the same way the other services can also be implemented.
Let’s try one more. How about setting up a new EC2 instance
So you can see the instance was launched in my account. Similarly the other services can be added too.
2. Hadoop Setup
Let’s move on to some hadoop setup.
Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation.
When we have to use the HDFS cluster in hadoop, we have to go through and setup two files and then launch the node in the cluster. Well this task can take some time and becomes hectic when we have to do the same task in so many different systems. Here I created a program that sets up the files according to the need of the user without any hassle in a second. Making the whole process faster.
These are the initial files of the node. We are going to configure this node as datanode and connect it to the namenode without performing any hadoop related command directly.
We can clearly see that right now there is no datanode connected to the namenode.
Here we first updated the HDFS file and then the core-site file, then we moved on to start the datanode. The program creates a folder in the directory for sharing the storage of datanode to the namenode and then finally connects to it. We can check the dfsadmin report for the same.
The similar setup can be done on the namenode as well. A few clicks is all it takes.
3. Docker Setup
Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. It eases the use of containers. In a way docker container behaves like virtual machines however they are very light weight and can be launched almost instantly.
Docker containers are isolated from the host system and hence the host is completely safe in case an unwanted bug corrupts the container. All we need to do is shutdown the system and launch a new container.
Our program provides an easy interface to create and launch new containers and view the containers.
Before using the above commands the docker should be setup in the system, however this can be done using configuration which I have explained a little later in this article.
4. Linux Commands
A small module of the program includes performing linux commands by pressing the button rather than typing the command itself.
There is an option to run command on the local machine or to run it after logging in into a remote system.
There are various commands available which we use commanly including checking the RAM usage or enabling and disabling firewalls. Disabling firewall might not be the best of choices to make but there are always some use cases when it can come in handy.
5. Partitioning
Now let’s move on to partitions. Partitions helps us to plan our usage. I have seen many users whose storage looks something like this,
A single drive having all their resources along with the OS installed. This might not be the best idea to get by.
The major issue that occurs in such a condition is that if the OS crashes it will take down the whole drive and hence it his highly likely that all the sensitive data might be compromised. This is why partitioning and planning is a nice idea when using your system.
In this program I tried to ease this work of partitioning by allowing the user to just select a few options to choose what they wanna do and the partitioning commands are run in the background.
6. Performing LVM partitions
The CLI application we decided to work on is still under development and we would be expanding it’s usability and functionality. Currently we are working on Ansible. The integration of ansible might help to further automate the processes so that the process of downloading the software is also eliminated so that the user can just launch the program and the program takes care of downloading and installing the dependencies as well.
The user is prompted to create a physical volume and a volume group. It is as easy as just entering the name of the folder and files without having to actually implement them through commands.
Then the user can just select the folder in which the storage is to be mounted. I chose a folder by the name of “mounter” and it is allocated 200 MB as I specified as an input.
Before beginning the commands the program implicitly yum installs all the dependencies of using the LVM storage. So a configured yum repository is extremely necessary.
So even that can be configured as per the need as there is provision to add that as well.
7. Configure yum and other dependencies
All the configuration and status check of the commands can be done with just one click . This was a primary need because before using any features having yum configured is extremely necessary.
8. Configuring the webserver
The webserver is needed when we want to host a website or program. I used the apache server to configure it.
The process involved for successfully hosting a server is of 3 steps
- Download the software
- Configure the server
- start the services
Firstly to download the dependencies we will setup the server.
Now we can directly start the services using 2.
To check whether the services have started I will use
systemctl status httpd
Clearly we can see that the services are up and running.
In this article I gave a brief working of the project I have undertaken. You can try forking the project from here if you find it interesting. I have thought of automating it further as I keep learning new skills.
You can catch me up at this location as I am always up for a good healthy talk.
Thanks.
