ip

VMware hosts network speed tests with iperf

Ever needed to run speed-tests between your VMware hosts? There is an CLI command iperf3 for this.

This command runs as a server and client command. One host will be the server and the other the client. There is further the possibility that some storage vendors even support the iperf3 command.

Example scenario with two VMware ESX hosts:

IT-ESX-01P – will act as server
- IP: 10.0.0.1
IT-ESX-02P – will act as client
- IP: 10.0.0.2

Steps and commands to execute the network speed test:

Enable SSH on both hosts and connect with e.g. Putty to it, logon as well.
IT-ESX-01P will act as our server
1. disable the firewall
  1. esxcli network firewall set –enabled false
  2. The ESX firewall needs to be disabled temporarily to execute the tests – on client and server
2. List the kernel network IP addresses
  1. esxcli network ip interface ipv4 get
  2. choose the interface IP that is on the network you want to test, only kernel-IPs will work
3. go to the directory that holds the iperf3 command
  1. cd /usr/lib/vmware/vsan/bin
4. start the iperf server on this host on the kernel IP you need it on
  1. ./iperf3.copy -s -B 10.0.0.1
  2. this command starts the server respective listener on the host on the specified IP address
IT-ESX-02P will act as our client
1. disable the firewall
  1. esxcli network firewall set –enabled false
  2. The ESX firewall needs to be disabled temporarily to execute the tests – on client and server
2. go to the directory that holds the iperf3 command
  1. cd /usr/lib/vmware/vsan/bin
3. execute the speed test against the server IP address
  1. ./iperf3 -c 10.0.0.1 -t 10 -V
  2. this will start sending packets to the server – you will see the flow on both sides
  3. cancelling this command – cntrl + c – can take a minute, be patient, especially if you mistyped the IP or forgot to disable the firewall etc..
Review the results on the speed test
1. Below are result samples for a 1 GB kernel network, a 10 GB kernel network and a 25 GB kernel network.
2. Sample results – 1 GB
3. Sample results – 10 GB
4. Sample results – 25 GB
5. Be aware, those results will vary and depend on the network bandwidth available in the moment of the test, respective the current load on the network cards of client and server.
IT-ESX-01P exit server mode and enable firewall
1. cntrl + c will exit the server mode and go back to the CLI
2. enable the firewall
  1. esxcli network firewall set –enabled true
3. EXIT SSH
IT-ESX-02P enable firewall
1. enable the firewall
  1. esxcli network firewall set –enabled true
2. EXIT SSH
Done

Additional links to this topic:

March 9, 2021 by Florian Rossmark configuration server

Compare a TRACEROUTE against an expected route

Some times it is important to know if a route to a specific target changed. The best way to check this is for sure to engage a monitoring software. But there might not be a real standard solution in the monitoring software. Now you end up wondering how should you approach this and find a solution to monitor a traceroute result and the hops between you and the target and find out if something changed.

For this exact case I wrote a PowerShell script that expects the following two parameter:

a target IP address or DNS name
a list of IP addresses in order as they should come up in the traceroute (comma separated)
1. include the target IP address as a final HOP

The script then will execute a tracert or traceroute to the target and compare the results step by step against the IP address list you injected. If a HOP is different at a certain position from what was expected an error will be reported back which IP was found and which was expected.

If there are no differences as for the injected IP list, it still could be the case that the traceroute to the target was e.g. longer or shorter – if there then is a difference in the HOP count between the actual traceroute results and the injected IP address list you still will get an error raised and a report back on the amount of found hops and expected hops.

Since the whole output is optimized to be processed by Paessler’s PRTG you also get an error from the sensor / script when the target was not reachable.

Additionally some information on the round-trip-time (RTT) and total HOPs is reported back to the sensor, ending up in a total of four channels:

Target reached
Round Trip Time
Total HOPs
Route matches expected route

As of September 2020 there are two versions of this script. First see the original script, after this I detail out the second version, see further below please.

param(
  [string]$Target,
  [string]$ExpectedRoute
)

$ER = $ExpectedRoute.Split(",");

$Trace=TNC $Target -TraceRoute;
$Count=[string]$Trace.TraceRoute.Count;

$Message = "";
$asexpected = 1;
[int]$i = 0;
ForEach ($IP in $Trace.TraceRoute) {
    If ($i -lt $ER.Length){
        If ($IP.Trim() -ne $ER[$i].Trim()) {
            $Message = "Route change at HOP #" + ($i + 1).ToString() + " - IP expected: " + $ER[$i] + " - IP found: $IP";
            $asexpected = 0;
            break;
        }
    }
    $i += 1;
}

If ($Trace.TraceRoute.Count -ne $ER.Length) {
    If ($asexpected -eq 1) {
        $Message = "Route changed - expected HOP count: " + $ER.Length + " - detected HOP count: " + $Trace.TraceRoute.Count;
        $asexpected = 1;
    }
}

$reached = 0;
If ($Trace.PingSucceeded) {
    $reached = 1;
}


$XML = "<prtg>
    <result>
        <channel>Target reached</channel>
        <value>$reached</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
    </result>
    <result>
        <channel>Round Trip Time</channel>
        <value>" + $Trace.PingReplyDetails.RoundtripTime + "</value>
    </result>
    <result>
        <channel>Total HOPs</channel>
        <value>" + $Trace.TraceRoute.Count + "</value>
    </result>
    <result>
        <channel>Route matches expected route</channel>
        <value>$asexpected</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
        <LimitErrorMsg>$Message</LimitErrorMsg>
    </result>
</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...
{
    $StringWriter = New-Object System.IO.StringWriter;
    $XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;
    $XmlWriter.Formatting = "indented";
    $xml.WriteTo($XmlWriter);
    $XmlWriter.Flush();
    $StringWriter.Flush();
    Write-Output $StringWriter.ToString();
}

WriteXmlToScreen "$XML"

param(

[string]$Target,

[string]$ExpectedRoute

)

$ER = $ExpectedRoute.Split(",");

$Trace=TNC $Target -TraceRoute;

$Count=[string]$Trace.TraceRoute.Count;

$Message = "";

$asexpected = 1;

[int]$i = 0;

ForEach ($IP in $Trace.TraceRoute) {

If ($i -lt $ER.Length){

If ($IP.Trim() -ne $ER[$i].Trim()) {

$Message = "Route change at HOP #" + ($i + 1).ToString() + " - IP expected: " + $ER[$i] + " - IP found: $IP";

$asexpected = 0;

break;

}

$i += 1;

}

If ($Trace.TraceRoute.Count -ne $ER.Length) {

If ($asexpected -eq 1) {

$Message = "Route changed - expected HOP count: " + $ER.Length + " - detected HOP count: " + $Trace.TraceRoute.Count;

$asexpected = 1;

}

$reached = 0;

If ($Trace.PingSucceeded) {

$reached = 1;

}

$XML = "<prtg>

<channel>Target reached</channel>

<value>$reached</value>

</result>

<channel>Round Trip Time</channel>

<value>" + $Trace.PingReplyDetails.RoundtripTime + "</value>

</result>

<channel>Total HOPs</channel>

<value>" + $Trace.TraceRoute.Count + "</value>

</result>

<channel>Route matches expected route</channel>

<value>$asexpected</value>

<LimitErrorMsg>$Message</LimitErrorMsg>

</result>

</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...

{

$StringWriter = New-Object System.IO.StringWriter;

$XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;

$XmlWriter.Formatting = "indented";

$xml.WriteTo($XmlWriter);

$XmlWriter.Flush();

$StringWriter.Flush();

Write-Output $StringWriter.ToString();

}

WriteXmlToScreen "$XML"

As mentioned further above, after a request in September 2020 in the Paessler KB a version 2 of the script was created. This version includes a third needed parameter – Expected IP address. This also introduces a new channel, that will indicate if the expected IP was found or not (error status).

This can possibly help, as the requestor stated, to determine if route still goes over a primary gateway or a backup gateway. Sometimes you a route might change further below, but a certain router needs to be active. You could also easily change the expected IP behavior to not expected IP behavior, by changing the $ExpectedIPFound value in line 33 to 1 and in line 36 to 0. This will invert the script behavior for this function and therefor only alert (indicate) IF this IP suddenly appears on the route, besides the fact that the route will have changed anyways.

param(
  [string]$Target,
  [string]$ExpectedRoute,
  [string]$ExpectedIP
)

$ER = $ExpectedRoute.Split(",");

$Trace=TNC $Target -TraceRoute;
$Count=[string]$Trace.TraceRoute.Count;

$Message = "";
$asexpected = 1;
[int]$i = 0;
ForEach ($IP in $Trace.TraceRoute) {
    If ($i -lt $ER.Length){
        If ($IP.Trim() -ne $ER[$i].Trim()) {
            $Message = "Route change at HOP #" + ($i + 1).ToString() + " - IP expected: " + $ER[$i] + " - IP found: $IP";
            $asexpected = 0;
            break;
        }
    }
    $i += 1;
}

If ($Trace.TraceRoute.Count -ne $ER.Length) {
    If ($asexpected -eq 1) {
        $Message = "Route changed - expected HOP count: " + $ER.Length + " - detected HOP count: " + $Trace.TraceRoute.Count;
        $asexpected = 1;
    }
}

$ExpectedIPFound = 0
ForEach ($IP in $Trace.TraceRoute) {
	If ($IP.Trim() -eq $ExpectedIP.Trim()) {
		$ExpectedIPFound = 1;
		break;
	}
}

$reached = 0;
If ($Trace.PingSucceeded) {
    $reached = 1;
}

$XML = "<prtg>
    <result>
        <channel>Target reached</channel>
        <value>$reached</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
    </result>
    <result>
        <channel>Round Trip Time</channel>
        <value>" + $Trace.PingReplyDetails.RoundtripTime + "</value>
    </result>
    <result>
        <channel>Total HOPs</channel>
        <value>" + $Trace.TraceRoute.Count + "</value>
    </result>
    <result>
        <channel>Expected IP Found</channel>
        <value>$ExpectedIPFound</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>		
    </result>	
    <result>
        <channel>Route matches expected route</channel>
        <value>$asexpected</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
        <LimitErrorMsg>$Message</LimitErrorMsg>
    </result>
	<text>$Message</text>
</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...
{
    $StringWriter = New-Object System.IO.StringWriter;
    $XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;
    $XmlWriter.Formatting = "indented";
    $xml.WriteTo($XmlWriter);
    $XmlWriter.Flush();
    $StringWriter.Flush();
    Write-Output $StringWriter.ToString();
}

WriteXmlToScreen "$XML"

param(

[string]$Target,

[string]$ExpectedRoute,

[string]$ExpectedIP

)

$ER = $ExpectedRoute.Split(",");

$Trace=TNC $Target -TraceRoute;

$Count=[string]$Trace.TraceRoute.Count;

$Message = "";

$asexpected = 1;

[int]$i = 0;

ForEach ($IP in $Trace.TraceRoute) {

If ($i -lt $ER.Length){

If ($IP.Trim() -ne $ER[$i].Trim()) {

$Message = "Route change at HOP #" + ($i + 1).ToString() + " - IP expected: " + $ER[$i] + " - IP found: $IP";

$asexpected = 0;

break;

}

$i += 1;

}

If ($Trace.TraceRoute.Count -ne $ER.Length) {

If ($asexpected -eq 1) {

$Message = "Route changed - expected HOP count: " + $ER.Length + " - detected HOP count: " + $Trace.TraceRoute.Count;

$asexpected = 1;

}

$ExpectedIPFound = 0

ForEach ($IP in $Trace.TraceRoute) {

If ($IP.Trim() -eq $ExpectedIP.Trim()) {

$ExpectedIPFound = 1;

break;

}

$reached = 0;

If ($Trace.PingSucceeded) {

$reached = 1;

}

$XML = "<prtg>

<channel>Target reached</channel>

<value>$reached</value>

</result>

<channel>Round Trip Time</channel>

<value>" + $Trace.PingReplyDetails.RoundtripTime + "</value>

</result>

<channel>Total HOPs</channel>

<value>" + $Trace.TraceRoute.Count + "</value>

</result>

<channel>Expected IP Found</channel>

<value>$ExpectedIPFound</value>

</result>

<channel>Route matches expected route</channel>

<value>$asexpected</value>

<LimitErrorMsg>$Message</LimitErrorMsg>

</result>

<text>$Message</text>

</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...

{

$StringWriter = New-Object System.IO.StringWriter;

$XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;

$XmlWriter.Formatting = "indented";

$xml.WriteTo($XmlWriter);

$XmlWriter.Flush();

$StringWriter.Flush();

Write-Output $StringWriter.ToString();

}

WriteXmlToScreen "$XML"

Request a DNS based web site directly from an IP address

If you are using multiple web servers and some more advanced DNS configuration like e.g. Round-Robin (and other) to load balance and possibly even GEO divide traffic to those web servers while those are all hosting the same web site you might as well want to monitor if this is all working as expected.

The challenge in this case is – your monitoring system might get just a single IP that might even change every now and then back from DNS when it requests it from DNS. So you would end up needing to use the target IP addresses of the individual web servers instead. But those expect you that your request for the web site goes through the domain name, since the servers aren’t supposed to answer on direct host-name / IP address requests or another web-site might be hosted there.

In order to accomplish this – I wrote the script below. It allows you to request a website directly from an IP address while injecting the DNS host in the header section of the Invoke-WebRequest (IWR) in PowerShell.

If you need to target a specific target site you can inject of just the IP address additional information, for example:

Standard parameters
- -DomainName “google.com” -IP “8.8.8.8”
  - this will request the web-root / default page
Sub-Page parameters
- -DomainName “google.com” -IP “8.8.8.8/monitoring.html”
  - this will request the content of /monitoring.html on the IP 8.8.8.8 with the DNS name google.com

PS: Please be aware – the above will not work – 8.8.8.8 is a Google-DNS server and not one of their web-servers. The above is only an example to show what is possible.

The below script output was optimized to be used in PRTG.

param(
    [string] $DomainName = "",
    [string] $IP = "",
    [string] $KeyWord = " ",
    [int] $MinimumResponseSizeInByte = 1,
    [bool] $useHTTPS = $true
)
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

If ($useHTTPS -eq $true) { 
    $Prefix = "https://" 
} Else { 
    $Prefix = "http://" 
}

$IWRResponse = "";
$IWRError = 0;
$IWRErrorMessage = "";

$IWRResponse = Try {
    Invoke-WebRequest -Uri "$Prefix$IP" -Headers @{Host="$DomainName"} -ErrorAction Stop
} Catch {
    $IWRError = $_.Exception.GetHashCode()        
    If ($IWRError -eq 0) {
        $IWRError = 1 
    }
    $IWRErrorMessage = $_.Exception.Message
}
Try {
    $IWRResponse = $IWRResponse.ToString()
} Catch {}

$KeyWordFound = 0
Try {
    If ($IWRResponse.Contains($KeyWord)) { 
        $KeyWordFound = 1 
    } 
} Catch {}

$ResponseSufficient = 0 
Try {
    If ($IWRResponse -gt $MinimumResponseSizeInByte) { 
        $ResponseSufficient = 1 
    }  
} Catch {}

$XML = "<prtg>
    <result>
        <channel>Check result</channel>
        <value>$IWRError</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>0</LimitMinError>
        <LimitMaxError>0</LimitMaxError>
        <LimitErrorMsg>$IWRErrorMessage</LimitErrorMsg>
    </result>
    <result>
        <channel>Keyword found</channel>
        <value>$KeyWordFound</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
        <LimitErrorMsg>Keyword was not found in response</LimitErrorMsg>
    </result>
    <result>
        <channel>Response Length</channel>
        <value>" + $IWRResponse.Length + "</value>
        <VolumeSize>Byte</VolumeSize>
        <NotifyChanged>1</NotifyChanged>
    </result>
    <result>
        <channel>Reponse Length sufficient</channel>
        <value>$ResponseSufficient</value>
        <LimitMode>1</LimitMode>
        <LimitMinError>1</LimitMinError>
        <LimitErrorMsg>Response size not sufficient</LimitErrorMsg>
    </result>
</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...
{
    $StringWriter = New-Object System.IO.StringWriter;
    $XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;
    $XmlWriter.Formatting = "indented";
    $xml.WriteTo($XmlWriter);
    $XmlWriter.Flush();
    $StringWriter.Flush();
    Write-Output $StringWriter.ToString();
}

WriteXmlToScreen "$XML"

100

param(

[string] $DomainName = "",

[string] $IP = "",

[string] $KeyWord = " ",

[int] $MinimumResponseSizeInByte = 1,

[bool] $useHTTPS = $true

)

add-type @"

using System.Net;

using System.Security.Cryptography.X509Certificates;

public class TrustAllCertsPolicy : ICertificatePolicy {

public bool CheckValidationResult(

ServicePoint srvPoint, X509Certificate certificate,

WebRequest request, int certificateProblem) {

return true;

}

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

If ($useHTTPS -eq $true) {

$Prefix = "https://"

} Else {

$Prefix = "http://"

}

$IWRResponse = "";

$IWRError = 0;

$IWRErrorMessage = "";

$IWRResponse = Try {

Invoke-WebRequest -Uri "$Prefix$IP" -Headers @{Host="$DomainName"} -ErrorAction Stop

} Catch {

$IWRError = $_.Exception.GetHashCode()

If ($IWRError -eq 0) {

$IWRError = 1

}

$IWRErrorMessage = $_.Exception.Message

}

Try {

$IWRResponse = $IWRResponse.ToString()

} Catch {}

$KeyWordFound = 0

Try {

If ($IWRResponse.Contains($KeyWord)) {

$KeyWordFound = 1

}

} Catch {}

$ResponseSufficient = 0

Try {

If ($IWRResponse -gt $MinimumResponseSizeInByte) {

$ResponseSufficient = 1

}

} Catch {}

$XML = "<prtg>

<channel>Check result</channel>

<value>$IWRError</value>

<LimitErrorMsg>$IWRErrorMessage</LimitErrorMsg>

</result>

<channel>Keyword found</channel>

<value>$KeyWordFound</value>

<LimitErrorMsg>Keyword was not found in response</LimitErrorMsg>

</result>

<channel>Response Length</channel>

<value>" + $IWRResponse.Length + "</value>

</result>

<channel>Reponse Length sufficient</channel>

<value>$ResponseSufficient</value>

<LimitErrorMsg>Response size not sufficient</LimitErrorMsg>

</result>

</prtg>"

Function WriteXmlToScreen ([xml]$xml) #just to make it clean XML code...

{

$StringWriter = New-Object System.IO.StringWriter;

$XmlWriter = New-Object System.Xml.XmlTextWriter $StringWriter;

$XmlWriter.Formatting = "indented";

$xml.WriteTo($XmlWriter);

$XmlWriter.Flush();

$StringWriter.Flush();

Write-Output $StringWriter.ToString();

}

WriteXmlToScreen "$XML"

March 6, 2019 by Florian Rossmark monitoring prtg scripts web

ip

VMware hosts network speed tests with iperf

Example scenario with two VMware ESX hosts:

Steps and commands to execute the network speed test:

Additional links to this topic:

Compare a TRACEROUTE against an expected route

Request a DNS based web site directly from an IP address

Recent blog posts

Blog Archives

ip

Example scenario with two VMware ESX hosts:

Steps and commands to execute the network speed test:

Additional links to this topic:

Recent blog posts

Blog Archives

Tags