High Availability Settings

To configure High Availability settings, navigate to the following page in the Panzura Filer's WebUI:

Configuration > High Availability

Panzura Filer HA Solutions

The following HA solutions are supported:

  • HA Local: An active Filer is protected by a dedicated standby. When the active Filer fails, the passive standby assumes its identity and takes over operations. The takeover operation can be automatic or manual. HA Local is similar to the methods used by legacy enterprise storage product. In this configuration, an active Filer is protected by a dedicated, passive standby. When the active Filer fails, the standby takes over ownership of the file system and the Filer operations. The following HA Local options are supported:
  • Local: The active and standby Filers have different hostnames and IP addresses.
  • Local with shared address: The active Filer and passive standby have an additional shared hostname and IP address, which simplifies the takeover process. A shared address is required if you are using Auto Failover. (Maximum length of the shared hostname is 15 characters.)
  • HA Global: One or more Filers are protected by one or more shared standbys, which can be separated geographically from the Filers they protect.

Sample HA Deployment

The following figure shows a Panzura deployment with three working sites—Los Angeles, London, and Paris—and two sites provisioned for HA—Phoenix and Amsterdam. A Panzura Filer is physically deployed at each site. Users at the three working sites connect to their local Filer, have a complete view of the shared file system, and experience LAN access speeds to the data in the global file system.

The figure shows HA options deployed as follows:

  • HA-Global. The Filer in Amsterdam protects both subordinates in London and Paris, as well as the Master Filer in LA.
  • HA-Local. The Filer in Phoenix is dedicated to protecting the Master Filer in Los Angeles.

The following HA-Local options are supported:

  • Local: The active and standby Filers have different hostnames and IP addresses.
  • Local with shared address: The active Filer and passive standby have an additional shared hostname and IP address, which simplifies the takeover process. (Maximum length of the shared hostname is 15 characters.)

Auto Failover

HA Local can be configured for Auto Failover. Auto Failover enables High Availability of an active-standby pair of Freedom filers to automatically perform a failover.

In an Auto Failover configuration, the Active and Standby Filers require a shared virtual IP (VIP) address and regularly exchange health and status information. The Filers regularly exchange status information in two ways:

• Directly via peer-to-peer connection (SSH)

• Posting status information to the cloud in state files. Status information about both Active and Standby filers is available to both the Active and Standby filers.

Note: Please take note that most public clouds do not support the use of VIP. Because of this, Auto Failover is not supported for in-cloud deployments.

Auto Failover Scenarios

There are three scenarios whereby Auto Failover can occur:

1. Active Filer loses communication to both the cloud object store and to the Standby Filer. This is determined by the Active Filer when the following two conditions occur:

  • Communication to cloud object store has failed for 160 consecutive attempts (either download or upload) and for ten minutes or more AND
  • Communication to the Standby Filer has failed for 10 minutes (state information exchanged with Standby every 30s. Twenty (20) consecutive failures required to meet 10 minute threshold.)

If both conditions are met, the Active Filer will change its own state to “Standby” and stop accepting user connections. This is to avoid the possibility of a split brain scenario.

The Standby Filer will monitor communications from the Active Filer and initiate a takeover process to become the Active Filer when:

  • Communication to the current Active Filer has failed for 10 minutes (state information exchanged with Standby every 30s. Twenty (20) consecutive failures required to meet 10 minute threshold.) AND
  • The Standby Filer has determined that the Active Filer has not updated the state file in the cloud for at least 10 minutes.

In this case, Panzura Freedom’s architecture ensures that the Standby Filer assumes the role of the Active Filer.

2. A second scenario for failover requires that the Active Filer perform health checks on itself. The Active Filer must make the following determinations in order to change its own state to Standby:

  • Assess its own critical operational processes (including file system status) and find it is in an unhealthy state AND
  • The Standby Filer is in a healthy state AND
  • The Standby Filer is less than 50 system snapshots behind the Active Filer

If all conditions are met, the Active Filer will change its own state to “Standby” and communicate to the Standby Filer to become the Active Filer.

3. A third scenario for failover may occur during maintenance activity of the Active Filer. There may be instances where the Active Filer would require reboot. During this process, the Standby Filer is aware of the Active Filer reboot. If the Standby Filer does not receive communication from the Active Filer for 12 minutes after reboot has been initiated, the Standby Filer will take over as the Active Filer.

Maintenance note: If, during the course of a planned maintenance window for any 7.1.x release version and higher, the Active Filer needs to be powered down, Auto Failover may need to be disabled first. Please consider your requirements for the Active Filer and for the Standby Filer.

Status Information Exchanged by the Active and Standby Filers

The Active and Standby Filers (peers) exchange information in order to make a coordinated decision on triggering a takeover. The following information is exchanged between the two peers, over an SSH connection that is automatically established between the two filers when the HA pair is configured:

  • Cloud status: Determined by a configured number of upload/download failures over a period of time.
  • File system status: Status of the file system (based on number of metaslab errors encountered).
  • Import status: Based on whether the filer successfully imports the file system after a reboot (either forced or unscheduled).
  • Critical process failed: Triggers failover if all remaining retries for a process fail.
  • Snapshot sync status: Determines eligibility for a failover based on snapshot sync status. If the snapshot sync status is 50 or more snapshots behind, failover is not allowed.
  • Scheduled reboot: If the active filer is scheduled to reboot, HA takes this into account and does not perform a failover when the rebooting filer goes offline during its reboot.
  • State change: Used to figure out whether the takeover is impending.

Requirements for Auto Failover

  • Both the Active and Standby Filers must be in the same subnet.
  • The Active-Standby pair must use a shared IP address and hostname, which must be registered in DNS. Clients and other devices will reach whichever filer is Active using the shared address. This prevents the need to reconfigure DNS following a failover.

Auto Failover Default Setting

Auto Failover is enabled by default in new HA Local configurations created on filers running on the latest version release (7.1.x and later). For HA configurations created using earlier software versions, you can upgrade to the latest Panzura version release and enable auto failover using the WebUI.


  • Auto Failover is supported in deployments where the Standby Filer is configured as an HA-Local filer.
  • Auto Failover requires a virtual IP (VIP) address. Auto Failover is supported only for HA configurations that use a VIP.
  • Auto Failover is not supported on Panzura Freedom Virtual Hard Disk (VHD) for Microsoft. This is because VIPs are not supported in the Azure Cloud.
  • Prior to PZOS 8.1, Panzura does not enable takeover option when Auto Failover is enabled it follows different code paths.

Setting Up HA

HA can be configured either during initial setup using the setup wizard, or later using the management WebUI.

Setting Up Local HA with Auto Failover

To set up Local HA with Auto Failover, use the following steps.

During Initial Setup

When configuring the secondary filer (the one that initially will be the standby), use the following settings in the Role section of the wizard:

  • Configuration Mode: HA Local
  • Auto Failover: Enable
  • Shared DNS Hostname: DNS hostname shared by the active-standby pair.
  • Shared IP Address: IP address that is mapped to the shared hostname on the DNS server.
  • Peer-to-Peer Authentication Key: Click Upload to load the Master Filer’s authentication key onto the secondary (HA Local) filer.

No specific settings are required on the primary filer (the one that initially will be the active filer).

Setting Up Auto Failover After Software Upgrade

If the filers you plan to configure for Auto Failover are already deployed, use the following steps:

  1. Navigate to the Master Filer and log in to the WebUI.
  2. Navigate to Management > High Availability.
  3. Set the Virtual IP option to enable, if not already enabled.
  4. Enter the shared hostname and IP address for the active-standby pair of filers.
  5. Set the Auto Failover option to enable.
  6. Click Done.
  7. Click Save to write the changes to the configuration.

Additional Information

If the Active Filer in this HA local is pairing with the Master filer, please use these steps

  1. Update all the filer's master-cc-host settings to direct to the shared hostname of the master.
  2. The master-cc-host field can be changed via CLI or webui (webui->Configuration->Configuration Mode->Subodrinate->Master: shared hostname and not the active hostname of the master filer)
  3. Access the filer with \\sharedhostname\ instead of \\active-filer-hostname\
  4. Update DFS-N to targets to use \\sharehostname\ instead of \\active-filer-hostname

All of these points make sense because now that we configured a share hostname and a share virtual IP (which points to that shared hostname in the DNS), we might as well use it.

Auto Configuration Options

You can configure the following Auto Failover options.

Option Description Default
Primary Filer Filer that initially will be the active filer in the Auto Failover pair. This filer remains the active filer until there is a failover. (none)
Secondary Filer

Filer that will begin as the standby filer.

(none)
Virtual IP Enables the active-standby pair to use a shared IP address and hostname. The Virtual IP (VIP) option allows clients and other devices to reach whichever filer is active, even if a failover has occurred. (none)
Shared Hostname DNS name shared by the active-standby pair. (none)
Shared IP IP address shared by the active-standby pair. This is the address that is mapped to the active pair’s shared hostname in DNS. (none)
Time To Wait for Maintenance Reboot If the active filer has a scheduled reboot (typically for maintenance), this is the number of minutes the standby filer allows for the reboot to occur. This timer prevents unnecessary failovers that occur because the standby filer assumes the active filer is unavailable. 12
Number of Allowed Dirty Snapshots Maximum number of un-synced snapshots the active filer can have, and still remain eligible for failover to the standby. The unsynced snapshots reside in the active filer’s dirty cache, in the lost+found folder on the failed filer. If the file can be rebooted, the un-synced snapshots can be recovered from this folder . 50
Peer Update Threshold

Maximum number of seconds the active and standby filers wait for updates from one another. These updates are exchanged directly between the filers over SSH.

If the standby filer does not receive an update from the active filer before this threshold expires, failover may occur. (The other failover criteria also must be met.)

200
Cloud Update Threshold Maximum number of seconds the active and standby filers are allowed to take to send status updates to the cloud. These updates are not exchanged directly between the filers but instead are read by each filer from the cloud. 10
Cloud Failure Count Maximum number of acceptable cloud failures. 20

Setting Up HA Local (no Auto Failover)

To set up HA Local (with no Auto Failover), use the following steps.

During Initial Setup

When configuring the secondary filer, use the following settings in the Role section of the wizard:

  • Configuration Mode: HA Local
  • Auto Failover: Disable
  • (optional) Shared DNS Hostname: DNS hostname shared by the active-standby pair.
  • (optional) Shared IP Address: IP address that is mapped to the shared hostname on the DNS server.
  • Peer-to-Peer Authentication Key: Click Upload to load the Master Filer’s authentication key onto the secondary (HA Local) filer.

The shared hostname and IP address are optional. If you do not configure them, you will need to update DNS to point to this filer following a failover.

Using the WebUI

After the HA Local active-standby filers are deployed, you can change HA settings from the WebUI of the Master Filer.

  1. Navigate to the Master Filer and log in to the WebUI.
  2. Navigate to Management > High Availability.
  3. Set the Virtual IP option to enable, if not already enabled.
  4. Enter the shared hostname and IP address for the active-standby pair of filers.
  5. Set the Auto Failover option to enable.
  6. Click Done.
  7. Click Save to write the changes to the configuration.

Additional Information

If the Active Filer in this HA local is pairing with the Master filer, please use these steps

  1. Update all the filer's master-cc-host settings to direct to the shared hostname of the master.
  2. The master-cc-host field can be changed via CLI or webui (webui->Configuration->Configuration Mode->Subodrinate->Master: shared hostname and not the active hostname of the master filer)
  3. Access the filer with \\sharedhostname\ instead of \\active-filer-hostname\
  4. Update DFS-N to targets to use \\sharehostname\ instead of \\active-filer-hostname\

All of these points make sense because now that we configured a share hostname and a share virtual IP (which points to that shared hostname in the DNS), we might as well use it.

Setting Up HA Global (no Auto Failover)

To set up HA Global (with no Auto Failover), use the following steps listed in the link here.

During Initial Setup

When configuring the filer that will be the global standby, use the following settings in the Role section of the wizard:

  • Configuration Mode: HA Global
  • Peer-to-Peer Authentication Key: Click Upload to load the Master Filer’s authentication key onto the secondary (HA Global) filer.

Using the WebUI

Use the setup wizard to configure the global standby.