Merging 'WIP: Do Not Merge This Branch' Into Master

README.md

@@ -1,5 +1,4 @@
-
-# SecDep (Still under development, hardening and docker deployment missing)
+# SecDep
 
 Automated secure docker services deployment and virtual machine management using (mainly) the [libcloud](https://github.com/apache/libcloud) library to connect to AWS, Azure and GCE.
 
@@ -9,6 +8,26 @@ Have you ever wanted to instantly deploy a simple virtual machine (or multiple)
 
 That's where SecDep comes in. With SecDep, you can manage your virtual machines across all three cloud providers with ease without having to think of where to go and how to get there to do what you want.
 
+# Features and roadmap
+
+- [x] Virtual machine creation
+    - [x] GCE
+    - [x] Azure
+    - [x] AWS
+- [x] Virtual machine actions (delete, start, stop, reboot)
+    - [x] GCE
+    - [x] Azure
+    - [x] AWS
+- [x] Virtual machine simple ssh connection
+- [x] Interactive `.env` file editing
+- [x] Provider image, location, size and instance listing
+    - [x] GCE
+    - [x] Azure
+    - [x] AWS
+- [x] Choice to perform above actions and instance listing on a single or all providers
+- [ ] Hardening during creation
+- [ ] Docker deployment during hardening
+
 # Prerequisites
 
 Even though we want to avoid using the tools or the web portals we actually have to do it just once to have everything set up to go.
@@ -26,3 +45,55 @@ For Azure you need to be a part of an organization that has an Azure subscriptio
 
 Then follow the instructions [here](assets/pages/azure/Azure_setup.md) to set up your Azure account.
 Instructions based on [libcloud's](https://libcloud.readthedocs.io/en/stable/compute/drivers/azure_arm.html#connecting-to-azure) documentation.
+
+## AWS
+
+For AWS you firstly need an account and then to follow the instructions [here](assets/pages/aws/AWS_setup.md)
+
+# Dependencies
+
+We would have 3 external dependecies but due to missing functionality in libcloud when it comes to Azure management we need 6.
+
+1. [libcloud](https://github.com/apache/libcloud)
+2. [python-dotenv](https://github.com/theskumar/python-dotenv)
+3. [paramiko](https://github.com/paramiko/paramiko) which is actually needed for the deployment of libcloud
+4. [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) from which we need
+   1. [azure-identity](https://pypi.org/project/azure-identity/)
+   2. [azure-mgmt-resource](https://pypi.org/project/azure-mgmt-resource/
+   3. [azure-mgmt-network](https://pypi.org/project/azure-mgmt-network/)
+
+# Installation
+
+## Cloning the repository
+
+You can clone the repository by running `git clone https://git.konsthol.eu/konsthol/SecDep.git` in the directory you want the files to reside.
+
+It goes without saying that you need to have git installed and that you need python to run it since it is a python script afterall.
+
+## Installing the dependencies
+
+The simplest way to install them is to run `pip install -r requirements.txt` in the root directory of the project.
+
+If you don't have pip installed you can install it by running `sudo apt install python3-pip` on Debian based systems or `sudo dnf install python3-pip` on Fedora based systems.
+
+For other systems you can check [here](https://pip.pypa.io/en/stable/installation/) for more information.
+
+# Walkthrough
+
+Assuming you have set up your accounts and have the dependencies installed, you can now start using SecDep.
+
+Further instructions [here](assets/pages/usage/usage.md)
+
+# Connecting to the virtual machines
+
+If you want to connect using your ssh client you will be able to run `ssh -i ~/Path/To/SecDep/secdep secdep@<ip>`.
+
+To use putty you will need to probably add the key to the program.
+
+The ip will be outputted after the creation of the virtual machine.
+
+If for any reason you delete either the public or the provate key they will both be regenerated and you should probably delete the virtual machines that use it or find another way to log in and manually put the secdep.pub key to the authorized_keys file of the secdep user.
+
+# License
+
+[GPLv3](https://www.gnu.org/licenses/gpl-3.0.txt)

assets/pages/aws/AWS_setup.md

@@ -0,0 +1,16 @@
+# AWS  Authentication
+
+To authenticate with AWS, the simplest thing you need is an access key.
+
+1. Go to the IAM section of the AWS console
+
+![IAM 1](../../images/aws/aws-1.png)
+
+2. Head over to manage access keys and create a new access key
+
+![IAM 2](../../images/aws/aws-2.png)
+
+The needed values are:
+
+* AWS_ACCESS_KEY -> the Access key ID
+* AWS_SECRET_KEY -> it's value

assets/pages/usage/usage.md

@@ -0,0 +1,160 @@
+# First run
+
+What will happen on your system on first run? well, SecDep will create a `.env` file in the root directory of the project. This file is needed to store the authentication values needed to access the cloud providers. You will be asked for every needed value for every supported provider and if you don't need one you should just press enter to leave it's respecting values empty.
+
+Should you ever change your mind you can always edit the `.env` file manually or run `python3 secdep.py -v` or `python3 secdep.py --values` to change the values you want.
+
+Then a public and a private ssh key will be generated and stored in the root of the directory. The public key will be added to the `~/.ssh/authorized_keys` file of a secdep user that will be created on the virtual machines. The private key will be used to connect to the virtual machines.
+
+# Creating a virtual machine
+
+To create a virtual machine you can run `python3 secdep.py -P <provider> -c` or `python3 secdep.py --provider <provider> --create` where `<provider>` is either `gce`, `azure` or `aws`.
+
+You will be asked to enter some information regarding the virtual machine you want to create such as
+
+1. The name
+2. The size
+3. The image
+4. The location
+
+and then you will be shown the chosen values and asked to confirm the creation of the virtual machine.
+
+# Skipping the confirmation and some options
+
+If you want to skip the confirmation or some of the asked values you can specify them using command line flags as so
+
+1. `-n <name>` or `--name <name>` to specify the name
+2. `-s <size>` or `--size <size>` to specify the size
+3. `-i <image>` or `--image <image>` to specify the image
+4. `-g <region>` or `--region <region>` to specify the location
+5. `-y` or `--yes` to skip the confirmation
+
+You will be asked for the values you didn't specify.
+
+A note on the `-g` or `--region` flag. It is used for Azure and GCE and it is used to specify the region. For AWS it is derived from the image because they are region specific.
+
+A full example looks like this
+
+`python3 secdep.py -P <provider> -c -n <name> -s <size> -i <image> -g <region> -y`
+
+or with specific values
+
+## GCE
+
+`python3 secdep.py -P gce -c -n test-node -g us-central1-a -s f1-micro -i debian-10-buster-v20230306 -y`
+
+## Azure
+
+`python3 secdep.py -P azure -c -n test-node -s Standard_B1ls -i Debian:debian-11:11:0.20221219.1234 -g eastus -y`
+
+## AWS
+
+`python3 secdep.py -P aws -c -n test-node -s t3.micro -i ami-0246e87085c5c98e3 -y`
+
+# Finding valid values
+
+If you want to find valid values for the image you can run `python3 secdep.py -P <provider> -I -p` or `python3 secdep.py --provider <provider> --listimages --print` where `<provider>` is either `gce`, `azure` or `aws`.
+
+Same goes for the `-S -p` or `--listsizes --print` flag for the sizes and the `-G -p` or `--listlocations --print` flag for the location.
+
+## AWS image example output
+
+`python3 secdep.py -P aws -I -p`
+
+`<NodeImage: id=ami-0eb2c4104acb437b2, name=debian-10-amd64-20221224-1239, driver=Amazon EC2  ...>`
+
+You can see that the image id is `ami-0eb2c4104acb437b2` and the name is `debian-10-amd64-20221224-1239`.
+
+What you'll need to specify is the image id.
+
+## GCE image example output
+
+`python3 secdep.py -P gce -I -p`
+
+`<NodeImage: id=992739542793186627, name=debian-10-buster-v20230306, driver=Google Compute Engine  ...>`
+
+Here you will need the name of the image.
+
+## Azure image example output
+
+`python3 secdep.py -P azure -I -p`
+
+`<AzureImage: id=Debian:debian-11:11:0.20221219.1234, name=Debian debian-11 11 0.20221219.1234, location=eastus>`
+
+Here you will need the id of the image.
+
+## AWS size example output
+
+`python3 secdep.py -P aws -S -p`
+
+`<NodeSize: id=t3.micro, name=t3.micro, ram=1024, disk=0, bandwidth=0, price=0.0204, driver=Amazon EC2  ...>`
+
+Here you will need the id of the size.
+
+## GCE size example output
+
+`python3 secdep.py -P gce -S -p`
+
+`<NodeSize: id=1000, name=f1-micro, ram=614 disk=0 bandwidth=0 price=None driver=Google Compute Engine ...>`
+
+Here you will need the name of the size.
+
+## Azure size example output
+
+`python3 secdep.py -P azure -S -p`
+
+`<NodeSize: id=Standard_B1ls, name=Standard_B1ls, ram=512 disk=4.0 bandwidth=0 price=0 driver=Azure Virtual machines ...>`
+
+Here you will need the id of the size.
+
+## AWS location example output
+
+`python3 secdep.py -P aws -G -p`
+
+`<EC2NodeLocation: id=0, name=us-east-1a, country=USA, availability_zone=<ExEC2AvailabilityZone: name=us-east-1a, zone_state=available, region_name=us-east-1> driver=Amazon EC2>`
+
+That is actually here for demonstration purposes because AWS doesn't have locations. The location is derived from the image.
+
+## GCE location example output
+
+`python3 secdep.py -P gce -G -p`
+
+`<NodeLocation: id=2000, name=us-central1-a, country=us, driver=Google Compute Engine>`
+
+We get the name of the location.
+
+## Azure location example output
+
+`python3 secdep.py -P azure -G -p`
+
+`<NodeLocation: id=eastus, name=East US, country=Virginia, USA, driver=Azure Virtual machines>`
+
+We get the id of the location.
+
+# Listing virtual machines
+
+To list the virtual machines you can run `python3 secdep.py -P <provider> -l` or `python3 secdep.py --provider <provider> --list`.
+
+You can also do the same without the provider flag to list all the virtual machines from all the providers.
+
+The print flag also works with the list flag like this `python3 secdep.py -P <provider> -l -p` or `python3 secdep.py --provider <provider> --list --print`.
+
+# Deleting a virtual machine
+
+To delete a virtual machine you perform an action with the -a or --action flag.
+
+When the action is delete you choose one of the virtual machines from the list and it will be deleted. Compined with the provider flag it lists all the virtual machines from that one provider.
+
+`python3 secdep.py -P <provider> -a delete`
+
+or
+
+`python3 secdep.py -a delete`
+
+If you want to delete them all in either case you can use the deleteall action instead of the delete
+
+Same goes for stop, start and reboot.
+
+# SSHing into a virtual machine
+
+To SSH into a virtual machine you get the choice from `python3 secdep.py -ssh` or `python3 secdep.py -P gce -ssh` to only choose from the gce instances.