“Learn 3 things A Day – Learning 3”

Sentinel: Automatic failover system in Redis

High Availability options in Redis include “Clustering” and “Master Slave” topologies. Clustering is an extension of multiple “Master Slave” sets, where data is partitioned into mutually exclusive data sets and stored in dedicated “Master Slave(s)” sets.

In Redis all these three tasks are performed by a separate process “Sentinel”. Sentinels can be run on same server as Redis or on an independent servers.

Master – Slave Topology: In Master Slave topology, applications connect to master Redis instance to perform read – write operations. Data thus generated will be asynchronously replicated to Slave Redis instance. To persist data to disk, Redis provides features like Snapshot and AOF (Append Only File). But such persistence of data to disk is for disaster recovery purposes though could be used for High Availability.

As shown below data generated by multiple clients is written to Master, that ultimately (due to asynchronous nature) gets replicated to Slave. Sentinel process(es) (could be more than 1) can be configured to monitor multiple master sets.

  • M1…Mn = Masters
  • S1…Sn = Slaves
  • C1..Cn = Sentinels

image 

Redis Clustering: Redis clustering is extending multiple Master – Slave sets to store portion of data. Simple clustering topology is depicted below. Notice in below topology data is split to multiple disjoint sets and each set is stored in a different Master Slave sets.

image

As per Wikipedia, http://en.wikipedia.org/wiki/High_availability, three principles of high availability engineering are

  • Elimination of Single point of failure
    • This means adding redundancy to the system so that failure of a component does not mean failure of the entire system.
  • Reliable crossover
    • In multithreaded systems, the crossover point itself tends to become a single point of failure. High availability engineering must provide for reliable crossover.
  • Detection of failures as they occur
    • If the two principles above are observed, then a user may never see a failure. But the maintenance activity must.

A more practical requirements for system to highly available:

  • Monitoring is to be enabled
  • Failure Detection. To detect failures or perceived failed scenarios
  • Notification, Action for automatic correction of situation (generally failover)

Example of High Availability in SQL Server on Windows Clustering:

  • Monitoring: SQL Resource executable that is installed as part of cluster installation monitors health of SQL Server instance.
    • In earlier versions of SQL it performs IsAlive and LooksAlive checks.
  • Failure Detection: If multiple IsAlive and LooksAlive checks fail it is considered as SQL Server instance failure
  • Decisive Action: Failover SQL Server from Active to Passive windows node

Nature of failures that systems have to withstand:

  • Executable crash
    • Redis may fail (improper exception handling??)
    • Sentinel may crash
  • System crash due to H/W or OS related issues.
    • In such a scenario Redis instances and any monitoring Sentinels present will fail.
  • Network disruptions
    • Set of systems of same network may become unavailable (even though Redis and Sentinel instances may be working)
    • Network disruptions (also terms as network partitions) can be across multiple networks.

Sentinels should detect failures of Redis and take appropriate actions.

How does Sentinel monitor and failover Redis instance?

  • Sentinel instances ping Redis instance and queries port of master Redis instance. Failure to contact / query master Redis instance is considered failure.
  • To avoid fault positives of Redis failures, multiple sentinels monitor same master. All sentinels cooperate (called Quorum) to declare master has failed and trigger failover.
  • But prior to failover, majority number of sentinels need to be present.

This is just a tip of iceberg on failover of Redis.. Need to learn more.

–Abhyast

Advertisements

Redis Cluster Setup (Linux on Windows)

There are so many things to share but so little time.. Though started writing about installation and setup of Hadoop on Windows using HyperV, deviating and writing about Redis. Have grand plans to write about Hadoop, Cassandra, Redis and SQL 14 along with my favorite Machine Learning. If only I could be better organized :(…

Coming to Redis, Redis recently got Cluster Support. Below from Redis release notes, Cluster feature was released on April 2015.

https://raw.githubusercontent.com/antirez/redis/3.0/00-RELEASENOTES

image

So, tried setting up Redis Cluster (different from Master / Slave concept) on Linux (Centos 7.0). Below is captured notes / lessons learnt. Follow previous posts to install / configure Linux on Windows machine using HyperV

https://abhyast.wordpress.com/2015/05/01/centos-installation-steps-on-windows-using-hypervisor-part-1/

https://abhyast.wordpress.com/2015/05/02/centos-installation-steps-part-2/

 

Few points, 

  • Installation process below is comprehensive for setting up redis on Linux that is setup from base (thus lengthy)
  • Below process installs Redis as Service and not an simple executable. If Service is not needed skip running install_server.sh  step
  • If you want to avoid following below process Redis provides script to setup cluster.
    • Under “utils” folder there is “create-cluster” script. Running below commands would setup cluster with 3 Masters and 3 Slaves
      • create-cluster start
      • create-cluster create
  • If Ruby is already installed, skips steps for installation of Ruby.

Requirements to install Redis Cluster:

  • Connection to Internet to download setup files (Else download manually and copy for Offline installation)
  • Permissions to install

#####################################################################

From here installation process starts.

#####################################################################

  • Update YUM installer
    • yum update
    • yum install wget

clip_image002

 

  • Goto a folder where redis needs to be installed
    • cd /opt/
  • Download Redis Stable version
  • Unzip File
    • tar xvzf redis-stable.tar.gz
  • Goto unzipped directory
    • cd redis-stable
  • Compile code
    • make
    • make Install
  • To run Redis (Standalone or as a Service) below setup files are needed along with their locations.
  • To run & test redis below files are needed
    • redis.conf (/opt/redis-stable/redis.conf)
    • redis-benchmark (/opt/redis-stable/src/redis-benchmark)
    • redis-server (/opt/redis-stable/src/redis-server)
    • redis-check-aof (/opt/redis-stable/src/redis-check-aof)
    • redis-check-dump(/opt/redis-stable/src/redis-check-dump)
    • redis-cli (/opt/redis-stable/src/redis-cli)
  • To setup cluster multiple instances of Redis need to be running either on same or different machines.
  • On same machine: Each instance should have dedicated configuration file along with data and log directories. Create folder structure as needed. Sample below
  • Create a directory name redis-base
    • mkdir redis-base
  • Copy files from above folders to redis-base folder
    • cp /opt/redis-stable/redis.conf /opt/redis-base
    • cp /opt/redis-stable/src/redis-benchmark /opt/redis-base
    • cp /opt/redis-stable/src/redis-server /opt/redis-base
    • cp /opt/redis-stable/src/redis-check-aof /opt/redis-base
    • cp /opt/redis-stable/src/redis-check-dump /opt/redis-base
    • cp /opt/redis-stable/src/redis-cli /opt/redis-base
  • Based on number of instances of redis to be run, create as many folders.
    • mkdir redis_6379
    • mkdir redis_6380
    • mkdir redis_6381
    • mkdir redis_6382
    • mkdir redis_6383
  • Copy files from Redis-base to each of these folders
    • cp /opt/redis-base/* /opt/redis_6379
    • cp /opt/redis-base/* /opt/redis_6380
    • cp /opt/redis-base/* /opt/redis_6381
    • cp /opt/redis-base/* /opt/redis_6382
    • cp /opt/redis-base/* /opt/redis_6383
  • Starting each instance of redis. Below steps should be followed for each instance and port numbers (command port and connect port should be different for each instance of redis, if redis servers are running on same linux server).
    • cd /opt/redis_6379
    • cp redis.conf redis.conf.default
    • vi redis.conf
    • port <each instance to have its own port number>
      • Redis_6379 port: 6379
      • Redis_6380 port: 6380
      • Redis_6381 port: 6381
      • Redis_6382 port: 6382
      • Redis_6383 port: 6383
  • Scroll down to end where it has cluster configuration

clip_image004

 

  • Change below configurations:
    • Remove “#” before cluster-enabled yes. Should look like
    • cluster-enabled yes
    • cluster-config-file nodes-6379.conf
    • cluster-node-timeout 15000
    • cluster-slave-validity-factor 10
    • Press “ESC” button followed by “:” , “w” and Enter to quit vi editor
  • Install by running below command
    • ./opt/redis-stable/utils/install_server.sh
  • Running install_server.sh prompts for
    • Port Number 
    • Config file
    • Log Directory
    • Data Directory
    • Executable Directory
  • Run install_server.sh command as many times as Redis installation required.
  • Start Redis Service:
    • service redis_6379 start
  • If required to stop Redis Service:
    • service redis_6379 stop
  • To check running services, run chkconfig command
    • chkconfig –list
  • Connect to each instance of Redis and ensure it is running and cluster-enabled is set to 1
    • redis-cli –h 127.0.0.1 –p 6379 <Port Number>
  • To get help from client, redis-cli –help
  • After connected it show IP and Port Number. At the prompt type command “info”
    • 127.0.0.1:6379>info

clip_image005

  • Cluster setup tool is written in RUBY. Ruby needs to be installed. Follow below steps to install Ruby and then configure cluster.
  • install all required packages for ruby installation
    • yum install gcc-c++ patch readline readline-devel zlib zlib-devel
    • yum install libyaml-devel libffi-devel openssl-devel make
    • yum install bzip2 autoconf automake libtool bison iconv-devel
  • Install Ruby Version Manager (RVM)
    • gpg –keyserver hkp://keys.gnupg.net –recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
    • curl -sSL https://get.rvm.io | bash -s stable
      • Optionally add any of these –rails or –ruby or –ruby = 1.9.3
  • Rvm provides shell script to setup RVM environment before installing Ruby
    • source /etc/profile.d/rvm.sh
  • After RVM is setup, install RUBY
    • rvm install 2.1.2
  • Use RVM command to setup default Ruby version
    • rvm use 2.1.2 –default
  • Check current ruby version
    • ruby –version
  • Ruby Script to configure Redis Cluster requires redis. Ruby script references redis. Prior to running Cluster setup script run
    • gem install redis
  • After ruby installation is complete run below command to setup Redis Cluster
    • /opt/redis-stable/src/redis-trib.rb create –replicas 0 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381

This completes “Redis Cluster Installation with default options”.

References:

-Abhyast