LAMP scalability and high availability

Today was a very busy day at the office, I felt like problems were coming at me from all directions. It’s often hard to keep focused on things that really matter with people and situations coming at you from all directions. When all the hubbub dies down and all the fires are put out you are still left with projects, planning, and relationship building that has to be done, but there is just not enough hours in the day. I am sure many of you can relate to my day at the office, but there is a better way, a more effective way, to manage your time that will bring you success in all areas of your life. Ask yourself these two questions and write down the answers:

1. What is one thing you could do on a regular basis (you are not doing now) that would make a huge positive difference in your personal life?
2. What in your professional life would bring similar results?

I have found that the key to effective time management is to organize and execute around priorities. Some priorities can vary from day to day, but others stem from the person’s character and are long term commitments to yourself and people around you. Some examples of these are relationship building, mentoring, problem prevention, recognizing new opportunities, P/PC (production and production capability) balance, delegation, or any activity that brings long term results in business or personal life. Stephen Covey talks about this in his book Seven Habits of Highly Effective People and provides an illustration called the “The Time Management Matrix” that can help you manage yourself more effectively. I would like to give credit to Stephen Covey for creating this diagram.

This diagram contains 4 quadrants with different types of activities. Using this diagram you can identify where you are spending most of your time and make corrections based on the suggestions below.

II. Not Urgent Important

Quadrant two contains issues that often don’t require our immediate attention and can be put off for later and unfortunately that’s often what happens. Important tasks such as relationship building, documentation, business development, employee mentoring, and training get left behind, because they don’t produce immediate results or recognition. The other enemies of tasks in quadrant two are interruptions, crises, and busy work. Effective people spend as much time as possible in quadrant two working on not urgent, but important issues. It’s rare to see senior managers running around with their heads cut off putting out fires and that’s mostly because they spend their time on low urgency high importance activities. Exercising prevention and anticipating problems can reduce interruptions and let you spend more time in this quadrant building your future success. Writing a blog like this is also a 2^nd quadrant activity?!

I. Urgent Important

Quadrant one holds tasks we can’t say no to. These include such matters as deadlines, system outages, very angry customers, and things of that sort. Some of these issues are unavoidable and it’s normal to spend some portion of your time in this quadrant; however, focusing your efforts on quadrant two will shrink the amount of time you spend in quadrant one and make you more effective.

III. Urgent Not Important

Quadrant three includes issues that other people think are important, but are really not. Some of which are meetings about meetings, spam by coworkers, phone calls, or any problems that if ignored will not bring the company to a screeching halt and will be forgotten about in 2 hours. Don’t be acted upon, stay on track and out of this quadrant.

IV. Not Important Not Urgent

I know that none of us spend any time here, but for the sake of completeness…! News, YouTube, and other time wasters should be avoided at all costs and don’t contribute to your effectiveness. One of the reasons that people go to this quadrant is burnout from spending too much time in quadrant one.

The philosophy behind concentrating on not urgent important issues in quadrant two is to be opportunity minded by starving problems and feeding opportunities. Once you identify the shortcomings in your current self management style you will need to find a planning tool that will take you and keep you in quadrant two where you can be most effective. I recommend making a plan for the whole week by scheduling quiet time to write it. The weekly plan of course can be adapted to each day during the week as needed.

This weekly planning tool needs to have six qualities

• Coherence

Helps you keep focus on your long term goals, priorities and plans and makes sure that all these synergize with values and principles important to you.

• Balance

Lets you balance different areas of your life such as work, family, health and personal development

• Quadrant II

Keeps you focused on quadrant two activities and away from crises.

• People

Nothing is set in stone. Allow for spending time with people even if it throws you off schedule. Don’t forget the human element.

• Flexible

The tool needs to be customizable and made to serve your needs.

• Portable

The tool needs to be portable so you can take it anywhere and review it easily.

A planning tool with these qualities forces you to look at your week not in terms of crises and action items that need to be addressed, but more in terms of principle centered long term goals. This is an effective way to manage yourself, because it’s easier to commit and carry out tasks that are principle driven and that have personal value and meaning to you. You will be effective and feel good about what you do!

Let’s backup a little, and as an experiment look at the answers you have given to the questions in the beginning of this blog. What quadrant do you think they fall into?

I bet its quadrant two! So what are you waiting for? Stop procrastinating and get to it.

by Pavel Pragin / pavel@clearscale.net

ppragin Management stephen covey time management

The Problem:
A customer called us in a state of panic telling us that the MySQL server hosting the database for their application had a hardware failure and the data was not recoverable. We were asked to get them back online as soon as possible plus re-architect their environment to prevent a disaster from happening again.
The Goal:
Engineer a cost effective solution that will provide a highly available web server and database server architecture without using a load balancer.
The Solution:
A Highly Available web and database architecture utilizing open source software available for free on the Internet. This solution will consist of DRBD (Distributed Replicated Block Device) software used for replicating block devices over the network, Heartbeat software which provides death-of-node detection and MySQL Master-Slave configuration used to replicate databases to another host.

Overview of the Solution:
This solution allows for a failure of one web and/or database server without interruption of service using DRBD/Heartbeat and MySQL Master-Slave configuration. DRBD and Heartbeat allow you to configure one of the web servers to be in standby mode. No HTTP requests go the Standby server. The purpose of the Standby server is to be available in case the Active server fails, and take its place. The Standby server has identical system configurations, Apache configurations, and the “/var/www” directory holding the application files. Keeping the application files identical is accomplished using DRBD which runs on both servers and replicates “/var/www” to the Standby server. Heartbeat is a piece of software that runs on both servers as well and actively monitors that both the Active and the Standby server is online. Whenever the Active server becomes unreachable via Heartbeat the failover process will be initiated and the Standby server will be promoted to Active. During the failover process the Standby server assumes the primary IP address of the Active server, mounts “/var/www/” and starts Apache. Although DRBD and Heartbeat are different pieces of software, their configuration files are intertwined and both of them work together to make the failover process possible.
Up to this point we have been talking about highly available web servers, but what about database servers and making them highly available. This is where Master-Slave replication can be used. The currently active web server connects to the Master database server to execute queries by default. Whenever any changes are made to the databases on the Master all data is immediately synced to the Slave. In an event that the Master fails the applications logic will automatically route all the queries to the Slave making the failure transparent.
You may have noticed after looking at the diagram that the Slave database server and the Standby web server are on the same physical machine. This was done to eliminate cost of adding a dedicated Slave database server, and to utilize the hardware for two purposes. If you are able to allocate an additional server for this solution I would recommend a dedicated database Slave server.
How DRDB and Heartbeat works under the hood:
DRDB software runs on both web servers and replicates the underlying file system that is mounted on “/var/www” block by block over the network. This replication happens over a dedicated crossover cable hooked up to “eth1” interfaces on both servers. This dedicated link provides a high performance connection between the systems bypassing all network hardware like switches and routers that can fail. During normal operation the “/var/www” is only mounted on the Active server. This means that data is only written on the Active web server and then replicated to the Standby.
Heartbeat software runs on both web servers to monitor whether both nodes are online. The Heartbeat signal is sent over the crossover cable hooked up to “eth1” interfaces on both servers. The crossover cable connection insures that there is no latency or network timeouts that could trigger a false fail over when the signal does not reach one of the nodes.
Each Active and Standby server has one primary interface “eth0” which is bound to IP address of “192.168.1.10” or 192.168.1.20” and is used for connecting to the servers using “SSH” and can be called the system’s administration interface. The Active server also has a virtual interface “eth0:1” which is bound to an IP address of “192.168.1.50” and is used for HTTP traffic. In most situations this IP address is translated to a public IP on the Internet using NAT and is associated with a domain name. The magic of heartbeat takes place when the Active node becomes unreachable and the Standby node brings up “eth0:1” interface and binds it to “192.168.1.50”, mounts “/var/www/”, and starts Apache. After this sequence of events is completed the Standby server takes the role of Active. This process takes seconds and is usually invisible to the users. After the failover event Heartbeat monitors the original Active server and when it comes back online it can be setup to automatically fail over to the original state.

How MySQL Master-Slave works under the hood:
Replication enables data from a Master database server to be replicated to a Slave database server. The Master server records all queries that are executed in a binary log file and stores it on the file system. The Slave then connects to the Master, retrieves the binary log and replays the “Write” queries keeping itself in sync. This process can be very fast depending on the type of queries, and provides a working copy of the database on another server at all times and can be used in case the Master server fails. In this particular solution the application has logic that will automatically re-route queries to the working server in case one fails providing high availability on the database layer.

by Pavel Pragin / pavel@clearscale.net

ppragin HA and Scale drbd mysql linux apache heartbeat