Checking if an email address exists using Powershell, Telnet, MX, and PONIES

(Ok, maybe not ponies.)

I’ve stolen and edited a fantastic telnet powershell function from here and am using it to directly telnet to an MX server and check for the existence of a user.

The edited powerful powershell generic telnet method is here:

[powershell]
Function Get-Telnet
{ Param (
[Parameter(ValueFromPipeline=$true)]
[String[]]$Commands = @("helo hi"),
[string]$RemoteHost = "HostnameOrIPAddress",
[string]$Port = "25",
[int]$WaitTime = 1000
)
#Attach to the remote device, setup streaming requirements
$Socket = New-Object System.Net.Sockets.TcpClient($RemoteHost, $Port)
If ($Socket)
{ $Stream = $Socket.GetStream()
$Writer = New-Object System.IO.StreamWriter($Stream)
$Buffer = New-Object System.Byte[] 1024
$Encoding = New-Object System.Text.AsciiEncoding

#Now start issuing the commands
ForEach ($Command in $Commands)
{
Write-Host $Command
$Writer.WriteLine($Command)
$Writer.Flush()
Start-Sleep -Milliseconds $WaitTime
}

$Result = ""
#Save all the results
While($Stream.DataAvailable)
{ $Read = $Stream.Read($Buffer, 0, 1024)
$Result += ($Encoding.GetString($Buffer, 0, $Read))
}
}
Else
{ $Result = "Unable to connect to host: $($RemoteHost):$Port"
}
$Result
}
[/powershell]

You call it by passing in a param array of commands, a host, and a delay time between each command being sent, e.g.:

[powershell]
Get-Telnet -RemoteHost "192.168.10.1" -Commands "admin","password","terminal pager 0","show run" -WaitTime 1500
[/powershell]

I’m only interested in hitting a mail server, so my commands are below.

  1. Start a session: “helo example.com”
  2. Create a new email from a dummy user: “mail from: <[email protected]>”
  3. Set the recipient to the user we’re trying to validate: “rcpt to: <$email>”

If the user exists then the third command returns status code “250”; if not, the status code is “550” – this is all I’ll check for.

The script below will split an email address and get the domain, find out the MX record using Resolve-DnsName, then log on via telnet to the mail server and attempt to create a mail to that user, returning the results of all commands and checking for an occurrence of “550”.

(syntax highlighter is throwing a wobbly if I use an email address inside the code block, so imagine “botATexample.com” is “[email protected]”)
[powershell]
try {
$notexists = true

$email = Read-Host ‘Enter an email address to test’
$columns = $email -split ‘@’

$mx = (Resolve-DnsName -Type MX –Name $columns[1] -ea 0)
if ($mx -ne $null){
$notexists = (Get-Telnet -RemoteHost $mx.NameExchange[0] -Commands "helo example.com","mail from: <botATexample.com>","rcpt to: <$email>" -WaitTime 500).Contains("550")
}

if ($notexists) {Write-Host "Doesn’t exist"}
else {Write-Host "Exists! "}
}
catch [system.exception] {
Write-Host $_.Exception.ToString()
}
[/powershell]

The results might look something like this:

check_email_exists_telnet_powershell

Notes

A helluva lot of mail servers will block this telnet connection! Therefore this is not a reliable method to get all existing users from a list of email addresses; it will result in a LOT of false negatives.

However, those that it returns as real users will certainly be real users.

Scripting a StatsD, MongoDB, ElasticSearch metrics server on Azure with Powershell

At Mailcloud I am constantly destroying and rebuilding environments (intentionally!), especially the performance test ones. I also need to gather oodles of metrics from these tests, and I have a simple script to create the VM and another to install all of the tools I need.

This article will cover using a simple Powershell script to create the Linux VM and then a slightly less simple bash script to install all the goodies. It’s not as complicated as it may look, and considering you can get something running in a matter of minutes from a couple of small scripts I think it’s pretty cool!

Stage 1, Creating the VM using Powershell

Set up the variables

[powershell]
# os configuration
# Get-AzureVMImage | Select ImageName
# I wanted an ubuntu 14.10 VM:
$ImageName = "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_10-amd64-server-20140625-alpha1-en-us-30GB"

# account configuration
$ServiceName = "your-service-here"
$SubscriptionName= "your azure subscription name here"
$StorageAccount = "your storage account name here"
$Location = "your location here"

# vm configuration – setting up ssh keys is better, username/pwd is easier.
$user = "username"
$pwd = "p@ssword"

# ports
## ssh
$SSHPort = 53401 #set something specific for ssh else powershell generates a random one

## statsd
$StatsDInputPort = 1234
$StatsDAdminPort = 5678

## elasticsearch
$ElasticSearchPort = 12345
[/powershell]

Get your Azure subscription info

[powershell]
Set-AzureSubscription -SubscriptionName $SubscriptionName `
-CurrentStorageAccountName $StorageAccount

Select-AzureSubscription -SubscriptionName $SubscriptionName
[/powershell]

Create the VM

Change “Small” to one of the other valid instance sizes if you need to.

[powershell]
New-AzureVMConfig -Name $ServiceName -InstanceSize Small -ImageName $ImageName `

| Add-AzureProvisioningConfig –Linux -LinuxUser $user –Password $pwd -NoSSHEndpoint `

| New-AzureVM –ServiceName $ServiceName -Location $Location
[/powershell]

Open the required ports and map them

[powershell]
Get-AzureVM -ServiceName $ServiceName -Name $ServiceName `
| Add-AzureEndpoint -Name "SSH" -LocalPort 22 -PublicPort $SSHPort -Protocol tcp `

| Add-AzureEndpoint -Name "StatsDInput" -LocalPort 8125 -PublicPort $StatsDInputPort -Protocol udp `

| Add-AzureEndpoint -Name "StatsDAdmin" -LocalPort 8126 -PublicPort $StatsDAdminPort -Protocol udp `

| Add-AzureEndpoint -Name "ElasticSearch" -LocalPort 9200 -PublicPort $ElasticSearchPort -Protocol tcp `

| Update-AzureVM

Write-Host "now run: ssh $serviceName.cloudapp.net -p $SSHPort -l $user"
[/powershell]

The whole script is in a gist here

Stage 1 complete

Now we have a shiny new VM running up in Azure, so let’s configure it for gathering metrics using a bash script.

Stage 2, Installing the metrics software

You could probably have the powershell script automatically upload and execute this, but it’s no big deal to SSH in, “sudo nano/vi” a new file, paste it in, chmod, and execute the below.

Set up the prerequisites

[bash]
# Prerequisites
echo "#### Starting"
echo "#### apt-get updating and installing prereqs"
sudo apt-get update
sudo apt-get install screen libexpat1-dev libicu-dev git build-essential curl -y
[/bash]

Install nodejs

[bash]
# Node
echo "#### Installing node"
. ~/.bashrc
export "PATH=$HOME/local/bin:$PATH"
mkdir $HOME/local
mkdir $HOME/node-latest-install

pushd $HOME/node-latest-install
curl http://nodejs.org/dist/node-latest.tar.gz | tar xz -strip-components=1
./configure -prefix=~/local
make install
popd

## the path isn’t always correct, so set up a symlink
sudo ln -s /usr/bin/nodejs /usr/bin/node

## nodemon
echo "#### npming nodemon"
sudo apt-get install npm -y
sudo npm install -g nodemon

[/bash]

Add StatsD

Here I’ve configured it to use mongo-statsd-backend as the only backend and not graphite. Configuring Graphite is a PAIN as you have to set up python and a web server and deal with all the permissions, etc. Gah.

[bash]
# StatsD
echo "#### installing statsd"
pushd /opt
sudo git clone https://github.com/etsy/statsd.git
cat >> /tmp/localConfig.js << EOF
{
port: 8125
, dumpMessages: true
, debug: true
, mongoHost: ‘localhost’
, mongoPort: 27017
, mongoMax: 2160
, mongoPrefix: true
, mongoName: ‘statsD’
, backends: [‘/opt/statsd/mongo-statsd-backend/lib/index.js’]
}
EOF

sudo cp /tmp/localConfig.js /opt/statsd/localConfig.js
popd
[/bash]

Mongo and a patched mongo-statd-backend

You could use npm to install mongo-statsd-backend, but that version has a few pending pull requests to patch a couple of issues that mean it doesn’t work out of the box. As such, I use my own patched version and install from source.

[bash]
# MongoDB
echo "#### installing mongodb"
sudo apt-key adv -keyserver hkp://keyserver.ubuntu.com:80 -recv 7F0CEB10
echo ‘deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen’ | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update && sudo apt-get install mongodb-org -y
sudo service mongod start
cd /opt/statsd

## Mongo Statsd backend – mongo-statsd-backend
## the version on npm has issues; use a patched version on github instead:
sudo git clone https://github.com/rposbo/mongo-statsd-backend.git
cd mongo-statsd-backend
sudo npm install
[/bash]

Ready to start?

Let’s kick off a screen

[bash]
# Start StatsD
screen nodemon /opt/statsd/stats.js /opt/statsd/localConfig.js
[/bash]

Fancy getting ElasticSearch in there too?

To pull down and install the java runtime, install ES and the es-head, kopf, and bigdesk plugins, add the below script just before you kick off the “screen” command.


# ElasticSearch
echo "#### installing elasticsearch"
sudo apt-get update && sudo apt-get install default-jre default-jdk -y
wget https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-1.1.1.deb && sudo dpkg -i elasticsearch-1.1.1.deb
sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

## Elasticsearch plugins
sudo /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head
sudo /usr/share/elasticsearch/bin/plugin -install lukas-vlcek/bigdesk

You can now browse to /_plugin/bigdesk (or the others) on the public $ElasticSearchPort port you configured in the powershell script to see your various ES web interfaces.

The whole script is in a gist here.

Stage 2 complete

I use StatsD to calculate a few bits of info around the processing of common tasks, in order to find those with max figures that are several standard deviations away from the average and highlight them as possible areas of concern.

I have an Azure Worker Role to pull azure diagnostics from table and blob storage and spew it into the Elasticsearch instance for easier searching; still figuring out how to get it looking pretty in a Grafana instance though – I’ll get there eventually.

Checking if a domain’s mail server is actually Gmail using Powershell

Firstly, run powershell as Administrator and execute
[powershell]Import-Module DnsClient[/powershell]

After that you can run the Resolve-DnsName cmdlet:
[powershell]
Resolve-DnsName reddit.com
[/powershell]

Which returns:
resolve-dnsname

You can specify which type of record you’re interested in:
[powershell]
Resolve-DnsName -type MX reddit.com
[/powershell]

resolve-dnsname-mx

So, take that logic and add a sprinkle of basic string functions to get:
[powershell]
(-join (Resolve-DnsName -Type MX –Name reddit.com -ea 0).NameExchange).ToLowerInvariant().Contains("google")
[/powershell]

Which returns True/False if any of the MX records contain “google”, which would mean the mail server is Gmail.

Formatted string from epoch date

Possibly the only really useful int extension method I’ve ever written.

Convert an epoch (unix date) to a formatted string:

[csharp]
public static class DateExtensions
{
public static string StringDateFromUnixTime(this int unixTime)
{
var epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var dt = epoch.AddSeconds(unixTime);
return dt.ToString("yyyy-MM-dd hh:mm:ss.fff");
}
}
[/csharp]

$’\r’: command not found

Working between Windows and Linux machines can be liberating, but a bit of a pain at times. One of these times is when creating scripts on a Windows PC and copying them over to a Linux box only to receive “$’\r’: command not found” when attempting to execute them.

This is due to the files being saved in DOS format where new lines are delimited with \r\n as opposed to the *nix \n format.

You can either ensure the files are saved in Unix format in your Win text editor, or even use this extremely easy method to process them on the Linux box:

[shell]
sudo apt-get install dos2unix
….
sudo dos2unix <your filename here>
[/shell]

Easy.

Automatic Versioning & “Cache Busting”

Implemented a CDN/caching layer but haven’t had time to get the versioning of assets worked out properly? Try some basic “cache-busting” using a querystring parameter which gets updated each time you build:

Update AssemblyInfo.cs to end with an asterisk (“*”):
[csharp]
[assembly: AssemblyVersion("1.0.0.*")]
[/csharp]

Create a little helper method:
[csharp]
public static class AppHelper
{
private static string _version;
public static string SiteVersion()
{
return _version ?? (_version =
Assembly.GetAssembly(typeof (HomeController))
.GetName().Version.ToString());
}
}
[/csharp]

And use this value in static file references:

[html]
<img src="/img/[email protected]()" alt="Logo" />
[/html]

Which will render something like:

[html]
<img src="/img/logo.png?v=1.0.0.20123" alt="Logo" />
[/html]

This number will change with every build, so should force retrieval of updated static – cached – content.

Upload to Azure Blob Storage using Powershell

I needed to automate the process of uploading images to Azure blob storage recently, and found that using something like the excellent Azure Storage Explorer would not set the Content Type correctly (defaulting to “application/octetstream”). As such, here’s a little script to loop through a directory and do a basic check on extensions to set the content type for PNG or JPEG:

The magic is in Set-AzureStorageBlobContent.

Don’t forget to do the usual dance of calling the following!

These select your publish settings file, and set which subscription is the currently active one:

  • Import-AzurePublishSettingsFile
  • Set-AzureSubscription
  • Select-AzureSubscription

Update

Actually, the Aug 2014 version of Azure Storage Explorer already sets the content type correctly upon upload. Oh well. Still a handy automation script though!

Getting past Powershell & SQL’s “Incorrect syntax near ‘GO’ ” message

Many a night have I bashed my head on the keyboard when seeing “Incorrect syntax near ‘GO'” come back from a powershell script trying to execute a batch SQL script.

After learning that “GO” is not actually SQL, and more of a SQL Server Management Studio “batch helper”, I quickly knocked together this powershell script to execute SQL batch scripts remotely. Fits nicely into an environment creation pipeline, so it does.

Param(
    [string]$Server,
    [string]$DB,
    [string]$user,
    [string]$Pwd,
    [string]$Script
)

$batches = $Script -split "GO\r\n"

$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Server=$Server;Database=$DB;User ID=$user;Password=$Pwd;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;"
$SqlConnection.Open()

foreach($batch in $batches)
{
    if ($batch.Trim() -ne ""){

        $SqlCmd = New-Object System.Data.SqlClient.SqlCommand
        $SqlCmd.CommandText = $batch
        $SqlCmd.Connection = $SqlConnection
        $SqlCmd.ExecuteNonQuery()
    }
}
$SqlConnection.Close()