Reducing latency through Proximity Placement groups in Azure

As a followup to my earlier blog post on Accelerated networking on Azure i am looking at other options available in Azure to reduce latency for Azure VM’s.

When you place your Azure VM’s in a single region, the physical distance between VM’s is reduced. Placing them within a single availability zone is another step you can take to deploy your virtual machines closer to each other. However, as the Azure footprint grows, a single availability zone may span multiple physical data centres resulting in network latency that can impact your overall application performance. If a region does not support availability zones or if your application does not use availability zones, the latency between the application tiers may increase as a result.

Enabling Accelerated networking reduces latency for Azure VM’s to a certain extent but Proximity placement group(PPG) provides Azure Virtual Machine logical grouping capability to further decrease inter-VM network latency.

As per Microsoft documentation, PPG can be used in the following cases

  • Low latency between stand-alone VMs.
  • Low Latency between VMs in a single availability set or a virtual machine scale set.
  • Low latency between stand-alone VMs, VMs in multiple Availability Sets, or multiple scale sets. You can have multiple compute resources in a single placement group to bring together a multi-tiered application.
  • Low latency between multiple application tiers using different hardware types. 

For the scope of this blog post we will be looking at the point #02(PPG in a single availability set). In the case of availability sets and virtual machine scale sets, you should set the proximity placement group at the resource level rather than the individual virtual machines.

Please see the github repository here for more detailed deployment steps

When proximity groups are enabled along with accelerated networking and availability sets the latency improves from ~72.9 ms to ~68.8 ms for an average of 10 test results on a set of 2 D3v2 VM’s within a single region.

Proximity groups enables moving VM’s closer to reduce latency but it compromises High availability if the VM’s are not placed in availability zones. To address this issue proximity groups can also be configured such that the VM’s are placed in separate availability zones to provide HA.

Some best practices to follow while creating PPG’s for new or existing virtual machines here

Securing Sitecore Topology deployed on Azure web apps(PAAS) using Application Gateway(WAF), Log analytics and Azure Monitor

Primary focus for the blog post would be to setup an Application gateway(WAF enabled) in front of a sitecore content delivery PAAS web app and test Azure WAF functiionality with SQL Injection attack using Log analytics and Azure Monitor(Log alerts) feature.

Update: WAF Support for Sitecore is officially available from SItecore 9.1 as mentioned in KB article https://kb.sitecore.net/articles/682999

Sitecore environments deployed using the standard Sitecore ARM templates on Azure PAAS are deployed on Azure App Service in the multi-tenant model(using Basic, Standard, Premium service tiers) . Azure App Service is a standard Web App offering is a multi-tenant environment configured for public access (with a publicly accessible endpoint).

Some organizations (e.g. Government, Financial) have requirements that all access to applications come through a Trusted Internet Connection (TIC), which means that the web applications should not be accessible directly from the Internet through their public endpoints and are only routed either through a

  • On-prem network integrated VPN or Express Route connection
  • User Traffic routed to Virtual network which controls the inbound/outbound communication using a Network Security Group.

App Service Environments(ASE) provide the Network isolation as well as private access through ExpressRoute integration by deploying the Azure web apps inside a virtual network using only the Isolated web app tier.

ASE being a very expensive azure service and comes up with some additional complexities that may not fit for all customer requirements, the other available options to secure web app would be

  • Restrict the access to web app using IP Restrictions – This approach will work for restricting access to Sitecore content management web app to content authors only with in the organization.
  • Front-ending the Web App with an Azure Application Gateway and restricting access to the Web App such that only connections from the Gateway are allowed – This approach can secure the Content delivery web app by making sure the user traffic only reaches the Gateway Public endpoint and can be secured with a Azure WAF with out directly reaching the content delivery web app.

sitecore-appgateway-waf-arch

For the purpose of this blog post we will focus on the second option using Application gateway(WAF) to secure the content delivery web app in Azure cloud.

Steps to create/configure Application Gateway(WAF) for Sitecore environment:

  1. Make sure the the App Service plan for web app to be fronted with WAF is scaled-up to Standard Tier.
  2. Create a Application gateway in the same resource group and location as the web app.

create-appgway

In the 2nd step during the creation process the Virtual network and Public IP will be configured and make sure the virtual network is created in the same resource group where the web app exists.

create-appgway-vnet

Point to note: Please set the Firewall mode to Detection for the scope of the testing Sitecore content delivery website.

3. Review the results and click on OK to create the gateway.

Frontend IP Configuration

frontend conf

Once the Application gateway is created, you see the Frontend IP configuration is created. Frontend IP configuration shows how the gateway is exposed. It can be either either public or private or both. The same configuration can be used by more than one HTTP listener, using different port.

Configure the Application Gateway

Backend pools feature provided by the Application gateway is used to configure the azure resources to which the user traffic needs to be directed. The resources supported as of today are NICs, virtual machine scale sets, public IPs, internal IPs, fully qualified domain names (FQDN), and multi-tenant back-ends like Azure App Service.

For the context of this blog post, we will be using App Service. Once the Application gateway is created there will be an appGatewayBackendPool already created and we can use the same backend pool to configure the content delivery web app.

edit-backend-pool

One the backend pool is assigned as shown below you can see the rule1 is created mapping to the HTTP url. Once he backend pool settings the incoming traffic that enters the application gateway would be routed to the backend address added here.

assign-backend-pool

Configuring SSL Termination/Offloading

Application gateway can be configured to terminate the Secure Sockets Layer (SSL) session at the gateway to avoid costly task of decrypting HTTPS traffic off your web servers. Application gateway decrypts the request and sends it to backend server and re-encrypts the response before sending it back to the client.

To configure SSL offload with an application gateway, a certificate (pfx format) is required. This certificate is loaded on the application gateway and used to encrypt and decrypt the traffic sent via SSL. For the scope of this blog post i will be using a self-signed certificate.

Use the below powershell script to generate a self-signed certificate in order to use that during HTTPS Listener.

$thumbprint = (New-SelfSignedCertificate `
-Subject "CN=$env:COMPUTERNAME @ Sitecore, Inc." `
-Type SSLServerAuthentication `
-FriendlyName "$env:USERNAME Certificate").Thumbprint

$certificateFilePath = "D:\Powershell\XPARM\$thumbprint.pfx"
Export-PfxCertificate `
-cert cert:\LocalMachine\MY\$thumbprint `
-FilePath "$certificateFilePath" `
-Password (Read-Host -Prompt "Enter password that would protect the certificate" -AsSecureString)

HTTP/HTTPS Listener:

HTTP Listener combines a frontend IP configuration and port it also include a protocol (HTTP or HTTPS) and optionally an SSL certificate. It will look for traffic based on its configuration and helps route the traffic to the backend pools

An HTTP listener is what the Application Gateway is listening to, for example:

  • Public IP xx.xx.xx.xx on port 443, HTTPS with a given SSL certificate
  • Private IP x.x.x.x on port 80, HTTP

basic-listener

Using the self signed certificate created earlier, please create a Basic HTTPS Listener as shown below by pressing OK button at the bottom. This process take about 15-20 mins for you to proceed with the next steps.

add-basic listener

Create Rule for HTTPS Listener:

Once listener is created, you need to create a rule to handle the traffic from the listener. Click the Rules of the application gateway, and then click Basic option. Type in the friendly name for the rule and choose the listener created in the previous step. Choose the appropriate backend pool and http setting as “appgatewayBackendHttpSetting” for now and click OK to save.

add-basic-https-rule

https-rule

Health Probe:

Once the HTTPS rule is saved, we can see that Health Probes menu option in the Application Gateway shows both the health probes created automatically. A  Health is described by a protocol, a URL, interval, timeout, etc.  .  Basically we can customize how a backend is determined to be healthy or not.

healthprobes

Once the HTTP/HTTPS probes are updated, you should see the HTTP Setting menu option with “appgatewayBackendHttpsSetting”setting set correctly or else you may have to manually upload the self-signed certificate(.CER file) and update the “appgatewayBackendHttpsSetting”

addprobe-https

Backend health

backend-health

Once the configuration is completed, please ensure WEBSITE_LOAD_CERTIFICATES app settings is added to the CD web app Application settings so that it accepts the self-signed cerificate

self-cert

Once the complete configuration is completed, the Gateway IP address found in the App gateway “overview” menu can used tested https://xxx.xx.xx.xx. If we use a CA authority certified SSL cert, we wont be seeing the invalid certificate issue and we should be able to route the traffic from Gateway IP to the Content delivery web app.

The Next step would be to restrict the content delivery web app to disallow web app public endpoint access. This can be achieved using web app IP restrictions by setting the Gateway IP address in the IP Address Block.

ip-restrict

For you to make it work in a local machine and test you need to add an Gateway IP address in Hosts file on your local machine and map it to a custom domain say “ga-sea-sitecore.azure.cd“. The path would be c:\Windows\System32\Drivers\etc\hosts.

ip-restrict

SQL Injection Security violation monitoring using Log analytics and Azure Monitor

  • Make sure a OMS Log analytics workspace is created and Azure Activity analytics solution is added to the workspace.
  • Make sure the content delivery web app diagnostic logs are exported to a storage account and storage account is connected to the log analytics workspace
  • Azure Diagnostics logs get collected under the Azure metrics solution

log-analytics

In order to inject SQL Injection we will use a simple sql script and invoke a website request

sql-inject

Now if you go back to Azure Diagnostics as Log search query you should see the SQl Injection attack has been logged.

AzureDiagnostics
| where Message contains “injection” and action_s contains “detected”

log query.PNG

Detecting and notifying customers on SQL Injection attack

Azure Monitor provides alerts on top of Log search queries which can be configured in Azure Monitor UI and selecting log analytics resource and the corresponding resource groups and select Log Search as the condition for alert

alert-log-src

Once the alert has been created and an action group configured in Azure to send emails/notifications for the alert configured, you will receive an email alert below.

email

Hope this helps in configuring and alerting customers on any security violations.

 

 

Monitoring Sitecore Topology on Azure – Azure Metrics, App Insights, Azure Monitor, Service Health Alerts

Sitecore topologies(9.0.2) officially provided at https://kb.sitecore.net/articles/267409 contains the below Azure resources

  • Azure web app
  • Azure SQL Database
  • Azure Search
  • Azure Redis cache
  • Azure Application Insights

Looking at an overall Sitecore topology the below would be an architectural representation of the monitoring story.  App Insights collects the telemetry from all the web apps and measures web app availability. Azure monitor helps in configuring alerts and metrics. Azure Health Alerts gives the flexibility to monitor the availability of azure specific services which can help to notify customers during an outage in a data center and in a scenario where only a specific azure service is out of service.

For the scope of this blog, i am not covering infrastructure monitoring using log analytics.

monitoring arch

Sitecore environments deployed a resource group can be monitoring using App Insights, Azure metrics and Azure Monitor and we use use the Azure Dashboards to give a unified view of the performance metrics and statistics of all the azure resources within a topology.

Some features that Sitecore developers have to enable to leverage the monitoring capabilities in Azure.

Application Insights

Enable the Application map feature in App Insights by uncommenting the below code snippet in the ApplicationInsights.Config in the sitecore web app instances

<TelemetryModules>
    <!-- Uncomment the following line to enable dependency tracking -->
    <!-- <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector"/> -->
    <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
    <! -- cut a lot if information -->
</TelemetryModules>
perfmap

2. Configure azure availability web tests(URL ping tests) in the azure portal to monitor the overall website availability

availabilitytest

3. Metrics for web apps, Azure SQL databases, Redis cache and Azure search can be configured in the azure portal. For all the above resources we can configure the azure metrics for each above mentioned azure resources using the Metrics menu option.

Add metric option allows the metric selection and ability to add multiple charts for the same resource.

webapp-metric-create

Once the required dashboards are added, “Pin to current dashboard” option adds the chart to the default azure dashboard.

webapp-pin-dashboard

Once you start configuring the required metrics for azure resources, you should be able to see a complete dashboard as shown below. For more clarity and simplicity i have updated the dashboard json for the below sitecore environment in Github at Sitecore-Azure-Monitoring . Few steps to make it work for you

  • Make sure the sitecore topology is deployed in a resource group in your azure subscription using Marketplace Sitecore Experience cloud.
  • Download the Sitecore-XPTopology902-Monitoring.json file
  • Replace the subscription id “xxxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxx” with relevant subscription id that you will be using for creating the dashboard
  • Replace the text “resourceGroups/ga-sea-exsmall-xp-rg” with the “resourceGroups/your-resourcegroup-name” in the json file.
  • Replace the other relevant azure resource name as required and upload the updated Sitecore-XPTopology902-Monitoring.json file in azure portal.

sitecore-dashboard

Azure Monitor

Azure Monitor gives the capability for alerting and notifications on metrics and logs as described in monitoring data sources. Azure monitor gives a cumulative monitoring on top of the below services

  • Metric values supported by Azure Metrics
  • Log search queries supported by Log analytics
  • Activity Log events supported by Log analytics
  • Azure Service Health alerts
  • web site availability test supported by App Insights

For each azure resource while adding the metrics for monitoring, a new alert rule can be created by specifying a condition for webapp-pin-createalert

Action Groups

An action group is a collection of notification preferences defined by the owner of an Azure subscription. Azure Monitor and Service Health alerts use action groups to notify users that an alert has been triggered.

webapp-pin-createalert

Once the alert is configured along with alert condition and notification requirements, you would see the metric alert rule created as below.

webapp-metricalert-added

Service Health Events

Service Health tracks three types of health events that may impact your resources:

  1. Service issues – Problems in the Azure services that affect you right now.
  2. Planned maintenance – Upcoming maintenance that can affect the availability of your services in the future.
  3. Health advisories – Changes in Azure services that require your attention. Examples include when Azure features are deprecated or if you exceed a usage quota.

By adding an Action group at a resource group level the action group can be used for any notifications related to the resource group.

action-group

Once the action group is created, the service alert can be created at a specific or multiple azure resources Ex: App Service , Web apps , SQL databases as shown below

service-alert

Hope this help!!.

 

 

Azure Autoscale & Custom Metric scaling feature updates

Improvements to Autoscale, faster auto scaling, simplified configuration, the ability to scale by a custom metric using Application Insights.

https://azure.microsoft.com/en-us/blog/manage-your-business-needs-with-new-enhancements-in-azure-autoscale/

Faster Autoscale 

  • Classic Cloud Services:  Autoscale setting can now be set to have as low as a five-minute time window to activate (previous recommended time window was no less than 30 minutes).
  • VMSS and App Services:The Autoscale engine for VMSS and App Services can also now trigger scale actions faster. The new engine is tuned to check for your metric based rules every minute, thereby enabling the ability to scale your instances as early as 1 minute after a metric value crosses the threshold set in an Autoscale setting.
  • Autoscale using custom metrics: Autoscale using custom metrics is available now and enables you to scale Classic Cloud Service, VMSS or App Service workloads by any Azure Monitor based metricor custom and application metrics collected by Application Insights, Azure’s application performance management service. Here is a sample of an Autoscale setting that allows you to scale your Web API app based on a custom metric ingested to Application Insights.
  • Improved Autoscale troubleshooting: The Autoscale engine logs an event in Activity Log every time it triggers a scale action, however, the target resource that is being scaled out or in can take the time to complete the scale action. It is important to know when the scale action completes or reports as failed so that you can take automated actions on the resource. To support this, the Autoscale engine now generates a scale action result event when the underlying target service completes the action or reports it as failed. This scale result event is also logged in the Activity Log and includes valuable information about why your Autoscale event failed. We’ve also introduced a new Autoscale Activity Log category so that you can easily filter to view only Autoscale-related events. You can leverage the new Activity Log Alertsto receive notifications or take automated actions via webhooks and Azure Automation, Logic Apps or Functions. This feature is now enabled for Cloud Services, VMSS and App Services.

Active FTP Connections using Jscape in c#

FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP is an unusual service in that it utilizes two ports, a ‘data’ port and a ‘command’ port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port.

Active FTP

In active mode FTP the client connects from a random unprivileged port (N > 1023) to the FTP server’s command port, port 21. Then, the client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then connect back to the client’s specified data port from its local data port, which is port 20.

Passive FTP

In order to resolve the issue of the server initiating the connection to the client a different method for FTP connections was developed. This was known as passive mode, or PASV, after the command used by the client to tell the server it is in passive mode.

In passive mode FTP the client initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. When opening an FTP connection, the client opens two random unprivileged ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but instead of then issuing a PORT command and allowing the server to connect back to its data port, the client will issue the PASV command. The result of this is that the server then opens a random unprivileged port (P > 1023) and sends P back to the client in response to the PASV command. The client then initiates the connection from port N+1 to port P on the server to transfer data.

Jscape is one of best 3rd party components used to develop ftp functionalities in your apps. The FTP class in Jscape provided the functionalities to create Active FTP connections.

The main properties in FTP class to use..

1. IsPassive

2. DataPortStart

3. DataPortEnd

4. PortAddress

The DataPortStart and DataPortEnd is the dataport range that are open in the server’s firewall exceptions. In case if the server exists in the Microsoft Azure VM, the data port range must be set in the Azure portal–> endpoint configurations.

PortAddress is the IP Address of the client from which the data connection is being initiated. In case we are trying to make an Active FTP connections from the Azure VM, by default it takes the static IP address of the VM until unless its specified explicitly.

Sample code looks like..

// Make connection…
Ftp ftp = null;
try
{
ftp = new Ftp(host, username, password);
ftp.LicenseKey = Constants.JsapeFTPLicenseKey;
ftp.Passive = !active;
if (active)
{
ftp.PortAddress = portAddress;
ftp.DataPortStart = Convert.ToInt32(dataPortStart);
ftp.DataPortEnd = Convert.ToInt32(dataPortEnd);
}
ftp.Connect();

if (overwrite)
ftp.Overwrite = true;
if (remotedir != null)
ftp.RemoteDir = remotedir;
if (useBinary)
ftp.TransferMode = TransferModes.Binary;
else
ftp.TransferMode = TransferModes.Ascii;
}
catch (Exception ex)
{
ftp.Disconnect();
ftp.Dispose();
throw new Exception(“FTP Connection Error: ” + ex.Message);
}

Blog at WordPress.com.

Up ↑