Politically Activating Pakistan Online

Like any country, Pakistan has had a tumultuous ride through history to come as far as it has. A quick shuffle through the history books reveals only a small, perhaps more glamorous side of politics, culture and socialism that makes it seem inconsequential. But the recent advances in technology platforms, development tools and infrastructure have made it possible to capture every action and move taking place on the ground, as it happens. And it is also very realistic to archive and create the library of history as it is made.

To better understand how technology is being used to do exactly this, we got in touch with the administrator of a political portal to follow its growth trend and how it links to the advancing technology.

PKPolitics.com was started as a project to archive the political history of Pakistan in video format for the upcoming generations. What started off to be a site to keep the administrator in touch with ongoings in Pakistan, has turned into the largest privately run site on current affairs in the country. Here’s a technical log of updates, patches, upgrades and modifications, as they happened…

The original idea for site originated in June 2007, when I was traveling to the US and could not find the coverage of Pakistani News channels in any of the hotels where I was staying. So for months at a time, I would be out of touch with the situation in the country. In a quest to stay in touch, and being able to do so at my own convenient time, I began exploring the options on internet and found that almost 50% of the popular television programs were present on YouTube. However because these uploads were being done by volunteers, the archiving was quite haphazard.

I had some previous technical experience on setting up blogs and I simply decided to record and archive my favorite talk shows online so that I could watch them from anywhere and whenever I wanted.

It took less than 24 hours from the origination of idea till the implementation and I had a site up and running with the embedded videos before the next day. The way I had my setup was that I could use my computer back home, which was connected to my TV, remotely from anywhere I was. I signed up for a cheap and shared web hosting in the US to run the WordPress-based front-end.

Once I had couple of videos on the site, I discovered many people commenting on YouTube and found that, like me, they were looking for several missing episodes of programs on YouTube. In an attempt to help them, I began uploading videos on YouTube, along with links back to PKPolitics.com, so that the people looking for these videos can watch them on them on my blog.

Let the Games Begin
I was quite surprised to find so many people who came to watch the videos. Within a short span of just a few days, I had lots of people commenting under the videos on the performance on anchors and guests in the programs, which I was not anticipating. The arguments in some of the comments used to outclass the points made by the host or guests in the program and these comments attracted more community who started to visit the site just to read and share their opinions, most of who had already watched the programs on their TVs.

The video uploads and site maintenance required lots of time and I wrote several scripts in attempt to automate most of the daily tasks that I did.

Video Hosting – Phase 2:
I started the site initially by hosting videos on Google video but soon had to try several other video providers due to the unpredictable delays in encoding times. I tired several of other services including Blip.tv and Viddler and went okay until end of 2007, when these companies refused to host the videos because of the amount of traffic PKP visitors were generating.

At that time, I weighed several other options and came to the conclusion that free video hosting account is just not the way to go. It can be closed anytime which would have hampered my idea of creating an archive for future reference and access. I dis cussed the options publicly with my visitors and privately with the some of the most senior members of the site and we agreed to experiment with the idea of premium membership, where people would contribute to share the bandwidth bills for dedicated servers.

For the videos, I was looking for a powerful solution that would cope with the unexpected loads during any critical political situation in Pakistan. You have to realize that unlike many networks, there was a different challenge with PKPolitics. The traffic on the site would experience a surge when there was a hot topic being discussed or breaking news was taking place. And though over time, we’ve been able to learn to intelligently predict what will be a catalyst for the surge, it’s been fairly spontaneous.

It was around this time that Amazon publicly launched their Elastic Compute Cloud (EC2), which was flexible enough to dynamically handle more load and also charged for the extra load on an hourly basis.

The video move to Amazon EC2 went smooth and I initially started the first week by hosting opening a few videos to the general public for a stress test. The server handled the load very well and streamed more than half a gigabits per second smoothly, but also landed me a cool bill of $100 per video per day. On Amazon EC2, I had used a small sized Linux instance, for which I paid $0.10 per hour for the CPU, in addition to the $0.17 per GB transfer of data. I used lighttpd web server, a very light weight server (wikipedia uses it) and supports flash streaming, allowing you to watch the movie instantly from anywhere without buffering.

The highest load used to be and still remains during the first 24 hours of the publishing of the video, therefore I decided, after consultation with senior members of PKPolitics, to restrict the videos in the first 24 hours only to the contributing members in order to avoid the high bandwidth bills.

Front End – Phase 2:
By end of year 2007, when political leaders were making a comeback into Pakistan, the traffic surged on PKP making it impossible for shared hosting to handle that amount of traffic. At that time, I moved the WordPress blog to mediatemple.net; the providers of gird hosting that could handle the extra load dynamically. It was more expensive but the content was being served without any intervention.

Somewhere between Phase 2 and Phase 3:
PKPolitics soon become the only site where people starting plugging in and reading minute-by-minute live commentary for any major political event taking place in Pakistan. It currently has the most detailed text logs of all major political events that have taken place in the last 2 years. Discussions would carry varying views and news items or articles would land hundreds of comments within hours of its posting.

While I was improving the infrastructure for PKPolitics, the main difficulty I faced was capacity planning. The roller-coaster political situation of Pakistan, where the visitors would jump from 5,000 per day to 50,000 during any major political development, needed more even more attention.

The bandwidth consumption was also very unpredictable. It would jump 10 folds on some days and reach up to 10 Terabytes a day, depending on nature of the events unfolding. In order to plan a more proactive approach to serving our traffic, I started to baseline the user CPU, memory and bandwidth consumption of all the servers during the last one year and analyzed the resource usage during all political situations in Pakistan. I also took an advanced course on web and database server capacity planning and performance tuning lasting several weeks long, from a prominent university.

The course provided me to have a better understanding of how to handle everything from scratch myself and I began planning my next phase for the site.

Phase 3:
Earlier this year, I moved all the services to my own dedicated servers, where I distributed the services between 4 different servers based on their specific load requirements of CPU, memory and bandwidth. The configuration was designed by analyzing the load usage during all major political events during last year and enough resources were put so that they could handle the maximum ever load that was benchmarked in the previous year.

In addition to creating a cluster of dedicated servers, I wrote some configuration scripts so that I could launch and install all the required components across multiple instances of Linux machines on Amazon EC2 cloud in a matter of minutes, in case some extremely unpredictable event happens in Pakistan.
In Phase 3, I had setup multiple servers from scratch with Linux and minimum libraries to bring up the system. The video hosting was setup on servers with lower CPU and memory specs, but with higher speed RAID storage and cheaper bandwidth. The front-end server for WordPress was chosen with highest CPU and memory but fixed bandwidth.

The Core of PKPolitics:
The main drive for creating PKPolitics remains the same – to create an accessible archive of the political scenario in Pakistan, in real time. Technology has been a key enabler in making this happen. Like most activism, online political activism that takes place through the portal, is on a voluntary basis where people who have strong opinions and inside perspectives on situations, come and express them on the public forum. To keep up with the traffic and the changing technology constraints has been a learning experience – something that technology and business managers must invest in doing. Learning from users and creating a collaborative ecosystem where the users have enough faith in the system to voice their opinions to bring about a change for the better, is an ideal environment to have.

Despite the traffic that is generated, because people continue to ‘stick’ to the site, it is our responsibility to continue reinvesting and re-engineering the backend so that the vision for converging politics with technology, both my passions, can remain alive.

Unlike a formal structured organization, PKPolitics does not have additional avenues to generate revenue and create a sustainable environment. The project barely covers its cost and breaks even with its expenses in salaries and bills, but because of the passion and interest that the contributors and PKPolitics team has in technology and politics, the site runs.