Monthly Archives: February 2020

How to use the cut command on Linux Bash

Often, you will have to deal with strings in your scripts. For instance, parsing Syslog files to find out and graph very specific information. And when it comes to string manipulation, the cut command can be very useful!

Cutting by byte position

This mode will allow you to cut from a position to another, let say between the first and the fourth-byte position. If your string includes a special char that uses more a byte, cutting with this mode can be tricky. As a result, I don’t use that mode, yet it’s good to know for very specific usage:

# This will return M
echo 'My first string' | cut -b 1 

# This will return Mf
echo 'My first string' | cut -b 1,4

# This will return My first
echo 'My first string' | cut -b 1-8

Cutting by character

This is the one I use the most, that is the same command but it doesn’t take into account the size of each character, only their position, so no bad surprise:

# This will return M
echo 'My first string' | cut -c 1 

# This will return Mf
echo 'My first string' | cut -c 1,4

# This will return My first
echo 'My first string' | cut -c 1-8

Cutting by delimiter

Here comes the real power of cut, using a delimiter made from a single character will allow you to do great things. In addition to that delimiter, you need to provide the field number, see the following to better understand.

Let’s take a more complex example to show you what is possible. Imagine you want to list all the bash used by the users on your system. This information can be found on /etc/password.

First, let see what looks like an entry located in /etc/passwd :

smar:*:101:1:Seb Mar:/home/smar:/usr/bin/zsh

We have 7 fields, the one we want is the last one. All those fields are separated by a semicolon. So, it’s pretty simple “:” will be our delimiter and we will ask for the 7th field:

# This will show you the list of all the shell defined in /etc/passwd
# With redundancy though
cat /etc/passwd | cut -d ':' -f 7

To finish properly our little exercise, let’s filter that to avoid duplicate using sort:

# This will show you the list of all the shell defined in /etc/passwd
# Without duplicates!
cat /etc/passwd | cut -d ':' -f 7 | sort -u

Please note that extracting more than a single field will return the selected fields separated by the same delimiter. For instance:

# The command below will return smar:/usr/bin/zsh
cat "smar:*:101:1:Seb Mar:/home/smar:/usr/bin/zsh" | cut -d ':' -f 1,7

However, if you want to replace this delimiter, you can use the –output-delimiter directive:

# The command below will return smar,/usr/bin/zsh
cat "smar:*:101:1:Seb Mar:/home/smar:/usr/bin/zsh" | cut -d ':' -f 1,7 --output-delimiter=','

Using complement to reverse the result

If you need the opposite result of your filter, cut provide the –complement directive to achieve that:

# While this will return My first
echo 'My first string' | cut -c 1-8

# This will return string
echo 'My first string' | cut -c 1-8 --complement

I hope those tips will help you scripting more efficiently!

Using Terraform with Azure

This post is a sticky note about how to configure your workstation in order to use Terraform with Microsoft Azure.

Configure Azure CLI on your station

First of all, you need to install this client to be able to connect to your Azure subscription. To do so, download the software from the Microsoft website. For Windows users, it’s a simple MSI package to install, then run a PowerShell console to run the login command as follow:

az login

This command will open your browser, and ask you to log in to your Azure account.

Once’s validated, you will be able to see your subscriptions within the PowerShell console. Take note of the tenant ID and the ID of the subscription you want to use further if you have many subscriptions as below:

az account list                                   [
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "xxx-xxx-xxx-xxx-xxx",
    "id": "xxx-xxx-xxx-xxx-xxx",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "xxx-xxx-xxx-xxx-xxx",
    "user": {
      "name": "xxx@xxx.xxx",
      "type": "user"
    }
  },
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "xxx-xxx-xxx-xxx-xxx",
    "id": "xxx-xxx-xxx-xxx-xxx",
    "isDefault": false,
    "managedByTenants": [],
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "xxx-xxx-xxx-xxx-xxx",
    "user": {
      "name": "xxx@xxx.xxx",
      "type": "user"
    }
  }
]

For information, you can use the command below to see your subscriptions:

az account list

That’s all for the Azure client, pretty simple, isn’t it?

Configuring Terraform on your station

This step is easy too, really, just browse Terraform web page to get the version that suits you. Keep in mind that, depending on the cloud provider (Azure in that very case), a version can or cannot be compatible with the provider. At this time, I’m using Terraform v0.12.21

OK, so unpack the archive to the folder of your choice. From here you can use the previous PowerShell console and browse to this folder. This is optional, but if you want to be able to run terraform from everywhere, you will need to deal with your environment variables. I won’t describe the process in this post though.

Open the Terraform folder with your file browser, then create inside a file called main.tf This file will contain the very basic information in order to connect to Azure:

provider "azurerm" {
  version = "=1.44.0"

  subscription_id = "xxx-xxx-xxx-xxx-xxx"
  tenant_id       = "xxx-xxx-xxx-xxx-xxx"
  skip_provider_registration = true
}

resource "azurerm_resource_group" "my-ressource-group" {
  name     = "my-ressource-group"
  location = "US West"
}

The first declaration is about the provider required to connect and manage Azure cloud services. Terraform strongly recommend using it to pin the version of the Provider being used. So this value will change in the future. Then set the subscription_id and tenant_id variables with the information gathered earlier. The skip_provider_registration is set to true here due to some restricted permissions I have, that might not be the case for you though.

If you have already created a resource group, you can also add the following declaration to your file:

resource "azurerm_resource_group" "my-resource-group" {
  name     = "my-resource-group"
  location = "US West"
}

Just keep in mind that Terraform, sadly, won’t import the resources you defined automatically. You will have to do that manually.

Now, run the following command to initialize Terraform, making it downloading the provider defined and creating the first local state file.

terraform.exe init

terraform.exe init

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.44.0...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Storing Terraform State in an Azure storage

You should see in the Terraform folder a file called terraform.tfstate which contains a representation of what you have in Azure.

If you are planning to work with a team, you should use a common file shared across your people. For that, we are going to create a storage account and a container to put this terraform.tfstate file in it. Note that Azure is handling will manage the lock on that file automatically, avoiding multiple access at the same time from your coworkers. To do so, create a storage-accounts.tf file (choose the name that suits you the best):

resource "azurerm_storage_account" "my-storage" {
  name                     = "my-storage"
  resource_group_name      = azurerm_resource_group.my-resource-group.name
  location                 = azurerm_resource_group.my-resource-group.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "my-storage" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.my-storage.name
  container_access_type = "private"
}

Here, we have defined a storage account and a container within, I let you check the documentation about the replication type.

Now, let’s create those resources with the simple command below:

terraform.exe apply

This command will check on Azure to see which change (update, creation or deletion) it has to do. In the end, it will show you the plan, asking you for a yes in order to apply. Type yes and see the magic happens.

Now, you have a container to put your tf file in, however, we need to update our main.tf file in order to use it. Edit the file by adding the following content after the provider definition:

terraform {
  backend "azurerm" {
    resource_group_name   = "my-resource-group"
    storage_account_name  = "my-storage"
    container_name        = "tfstate"
    key                   = "terraform.tfstate"
  }
}

The key parameter represents the file in your local Terraform folder. Run a new init this time, to apply those change:

terraform.exe init

Now you’re being serious, you can work from different places on the same Azure subscription, that’s a good start.

From there, you can go on the Terraform’s documentation page and see what the Azure provider can do for you.

The following script are just reminders for me, you might not be interrested.

Virtual network, route table, and subnet

resource "azurerm_virtual_network" "my-network" {
  name                = "my-network"
  address_space       = ["10.62.34.128/25"]
  location            = azurerm_resource_group.my-resource-group.location
  resource_group_name = azurerm_resource_group.my-resource-group.name
}

resource "azurerm_route_table" "my-route-table" {
  name                          = "my-route-table"
  location                      = azurerm_resource_group.my-resource-group.location
  resource_group_name           = azurerm_resource_group.my-resource-group.name
}

resource "azurerm_subnet" "my-subnet" {
  name                 = "my-subnet"
  resource_group_name  = azurerm_resource_group.my-resource-group.name
  virtual_network_name = azurerm_virtual_network.my-network.name
  address_prefix       = "10.62.34.128/27"
  route_table_id       = azurerm_route_table.my-route-table.id
}

A basic Linux machine

resource "azurerm_network_interface" "my-interface" {
  name                = "my-interface"
  location            = azurerm_resource_group.my-resource-group.location
  resource_group_name = azurerm_resource_group.my-resource-group.name

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = azurerm_subnet.my-subnet.id
    private_ip_address_allocation = "Static"
	private_ip_address 			  = "10.62.34.136"
  }
}

resource "azurerm_virtual_machine" "my-linux-vm" {
  name                  = "my-linux-vm"
  location              = azurerm_resource_group.my-resource-group.location
  resource_group_name   = azurerm_resource_group.my-resource-group.name
  network_interface_ids = [azurerm_network_interface.my-interface.id]
  vm_size               = "Standard_DS1_v2"

  # Uncomment this line to delete the OS disk automatically when deleting the VM
  delete_os_disk_on_termination = true

  # Uncomment this line to delete the data disks automatically when deleting the VM
  delete_data_disks_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }
  storage_os_disk {
    name              = "my-linux-vm-dsk01"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
	disk_size_gb	  = 50
  }
  os_profile {
    computer_name  = "my-linux-vm"
    admin_username = "myUser"
    admin_password = "ASup3r53cr3t!!@"
  }
  os_profile_linux_config {
    disable_password_authentication = false
  }
  tags = {
    type = "A tag I want"
  }
}

How to find a Linux server public IP address, from its terminal

Today, I would like to summarize some of the commands I usually use when I want to check the public IP address used by a server when connected through SSH. It’s a very simple post, yet I think that can help people at some point:

Using curl, the easiest way in my opinion:

curl ifconfig.me

Using dig and OpenDNS, way too complicated if you ask me:

dig +short myip.opendns.com @resolver1.opendns.com

Using wget, if curl is not available (that should never happen though), the file will be downloaded (index.html). You might use a command with pipe in order to get the IP address from the downloaded content:

wget http://monip.org

Configure a virtual Fortigate in Microsoft Azure

Before moving further, at this time, there is no way to configure your Azure network to push another gateway that the Azure default to your machine, you can create route for the next-hop, but those routes can’t override the default gateway. Long story short, you will have to manually configure the default gateway of your servers to make them use your Fortigate as their next-hop.

First, you will need to know the different subnets that you will use, in this example, I will use four subnets:

10.62.34.240/28, this one will be used for the WAN connection
10.62.34.160/27, this one will be used to connect machines to port1
10.62.34.128/27, this one for port2
10.62.34.192/28, and the last one for port3

Note that, depending on your interfaces need, you will have to choose a different appliance from the Microsoft pay-as-you-go catalogue. In order to create that appliance with 4 interfaces, I had to pick the Standard F4 (4 vcpus, 8 GiB memory) size. The virtual network and two subnets will be created during the creation process, call the first WAN and the second WEB or whatever makes sense for your port2.

So, you have created your subnets, then your appliance (FGT), and it’s now time to configure your network, before powering on the FGT. Unfortunately, it should have been turned on right after its creation, so shut it down for now.

Configuring the existing subnets

Edit the virtual network created by the wizard, at the time I’m writing this, it’s called FortigateProtectedVNet. You should see your two subnets:

We have to remove the Network Security Group (NSG) assigned by default, except for the WAN subnet, because we want to manage the security through the Fortigate itself. We could have choose to use them as an additional security layer, but in order to keep things simple here, we won’t discuss that part.

For each existing subnet, click on Network Security Group:

Click on the edit button, then on the NSG assigned:

Choose none in the left list:

On the next screen, clic save to confirm the change, repeat the operation for the second subnet.

Adding the other two subnets

We need to create the other two subnets. Go back to the FortigateProtectedVNet object, and within the left menu, click Subnets:

Click the + Subnet button to add a new one:

Add your new subnets, don’t add a NSG or a route table yet:

Adding and configuring routing tables

In order to tell your future VMs which routes using to access ressources, Azure is using routing table. These routing tables are linked to one or many subnets.

During the creation of your Fortigate, two subnets were created along two routing tables. We have to configure them (especially the second one) and add two new routing tables for the new subnets we have created above.

Go to your ressource group, then add a new route table object:

For the naming, I usually use the same name of the automated generated routes, updating only the subnet part, for instance:

Leave the virtual network route propagation enabled.

Once your new routes have been created, we need to configure them. The WAN routes doesn’t need to be updated. However, for all the other routes, do the following:

Click on the route table you want to configure, then Routes:

Remove all the current routes defined. Then create this following route to tell your VM to use the Fortigate interface as the next hop:

The next hop must match with the future IP of the Fortigate interface connected to that specific subnet. T

Once the route is added, it’s time to associate the subnet, click on Subnets on the left side:

Click the Associate button, and select the proper network/subnet:

Do the same thing for the other subnets (Except for WAN!)

Add new interfaces to the Fortigate

The current FGT has only two interfaces, turn it off then open its object in order to add two new interfaces.

Got to Networking:

Click Attach network interface:

Use the same pattern for the name, replacing the Nic part with the next number (Should be 2 and 3 for APP and DB). Check Static to set your Fortigate interface IP:

The interface will be created. But you still need to attach it manually on the next screen:

Repeat these operations for the last Nic3 interface.

IP forwarding and

We now need to configure those two new interfaces, for that, click on the interface tab, then on the interface name (#2 on the capture below) to access the configuration pane:

From here, click on IP configurations:

And enable the IP forwarding feature:

Save, and do the same thing with the last interface (Nic3).

In order to improve performances, we would like to enable accelerated networking, to do so, we need the FGT to be off (That should be the case at this stage) and then we need to run some commands.

Click on the Cloud Shell icon on the top right toolbar:

Choose Powershell when asked in the console area, down the window, create a storage (That’s mandatory). If you got a permission issue, click the Advanced Options and from there, create it:

Once connected to the Powershell console, run the following command to enable accelerated networking:

az network nic update --name Nic2 --resource-group YourRessourceGroupName --accelerated-networking true

Exit the console, refresh the whole portal, and you should now see the following:

Accessing your Fortigate

You can now start your FGT, and the use the public IP address created by the wizard to access the portal, using HTTPS.

If you have any issue accessing the interface of your Fortigate, ensure that the NSG created by default still associated to the Nic0 interface, you should have this: