Techniques for data migration across hosting servers

Just these few days I had the opportunity to handle the migration of data for a few web sites. Some were running on the fat cow server. Others were running on the host hi server. All of them were to be ported over to the Vodien server.

The fatcow server has over the past few months proved to be a disappointment, while the hosthi server has all along been a disappointment. Often do I get complains from customers that there web servers are either not loading or their sub domains are showing errors. Come next year I will allow both the fatcow and hosthi accounts to expire. I guess the moral of the story is that if some service providers promise unlimited something, there must seriously be a limited something else somewhere.

While migration of files across servers are easy, the migration of databases and email accounts are somewhat more complicated.

Large MySQL Database migration process

There is an instance where I had to migrated a database that is 100mb in size. To do so via the standard database admin panel called the phpmyadmin would have took forever. Hence I wrote a php script that to fetch a file via a handle

$fp = fopen(“databasedata.sql”, “r+”);

instead of reading in the whole file at one shot.

file_get_contents(“databasedata.sql”);

The latter method would have used up all the allocated resources in a normal php installation unless of course I went ahead and changed the configuration in my php.ini file. However this latter method is not at all a very elegant method to handle the task.

To further expound  on the prior method, I coded the script to parse the $fp  stream line by line and consolidate them into one big line everytime the latest line does not end with the string chunk  ”;\r\n”. In the event this chunk was detected, I will that this consolidated line and execute them to the database.

while (!feof($fp) && $counter<100 ) {
$query .=”\r\n”.trim(fgets($fp));
if(substr($query, -1) ==”;”){
//echo “query = “.$query;
connect_query($query);
$query = “”;
//$name = fgets(STDIN);
}
}

while (!feof($fp) && $counter<100 ) {

$query .=”\r\n”.trim(fgets($fp));

if(substr($query, -1) ==”;”){

execute_query($query);

$query = “”;

}

}

Lastly to free up unnecessary resources, I closed the stream $fp which I did not need to use anymore.

fclose($fp);

var_dump($myvar);

Thus the problem of database migration is solved easily with minimal throughput required on the web server. Of course, if I had configured my php installation, it is possible to run it from the command line.
Email Account Migration from webmail to google mail server.
This is another tricky situation I had to resolve. Apparently one of my clients used her webmail account rather extensively. In her webmail account she created various folders in which to store her emails. She wants all her mails migrated over to the new Google Mail server and at the same time have her email folder structure preserved.
What I did to fulfill this requirement was to create an imap stream to the server the original webmail account lies on to retrieve all the folders. Since I have not configurated her webmail to use any security ticket I added the line novalidate-cert. This line is very important and I noticed quite a lot of developers faced this problem when try to access their webmail via the use of PHP scripting.
set_time_limit (600000);
$mbox = imap_open(“{imap.yourwebhosting.com:143/novalidate-cert}”, “userid@somedomainname.com”, “password”);
$folders = imap_listmailbox($mbox, “{yourwebhosting.com:143}”, “*”);
For each respective folders that exist, I created an imap stream call to retrieve all her emails. For each respective email, I retrieved all the header information, subject and email content and reconstructed an email packet.
Next due to the fact that the google mail server has been configured to block all emails from sources with a domain name,  ip address mismatch, I had to create another email account (Account B) somewhere else so that I could send my mails to from my local web server. There after I configured the new google email account to retrieve those mails from Account B.
This in a nut shell is how I migrated both the database and the email accounts.

My constant failures in my pursuit of philosophy

Each morning I wake up with gratitude knowing that I have been once again presented with the gift of life for another day. Each day I strive along my path. Each night I sit in regret knowing I have once again failed.

Until the day comes when I could live each and every second of my life unstained by personal egotism, all I have done along the pursuit of my path will amount to nothing. Until that day comes, each and every of my action will translate immediately into failure at the very moment of their manifestation. Until that day comes, I can only live in gratitude that I have been once again presented with the gift of life and regret that I am unworthy of that gift.

When that day comes, my works and efforts will no longer be a manifestation of personal egotism and vanity, something I have personally come to despised and look upon with regret. When that day comes, my works will be a tribute in its purest sense unstained by my failings due to the animal within. When that day comes, I will truly see myself as a man and not a abomination of nature that is half man and half animal.

Testimonials from a dysfunctional

I saw a text message notification a few days ago. It was from Rodney, a Christian missionary from America currently in his 60s. I got to know him 4 years ago in Melaka. I was on my sabbatical then. My sabbatical was a necessity. I had delved too deeply into the night scene of Singapore prior to that so much so that I felt spiritually and emotionally drained by the superficially, decandence and hopelessness I witness around me. It was ironical how my devotion to bring about success in the tasks set before me brings me recognition from my peers while at the same time proportionally amplifying my sense of self loathe quietly hidden away from the sight of all around. This self loathing which over time grows into a deep hatred for self, a hatred that would most likely have resulted in suicide had I not drop everything material gained thus far and left. In retrospection doing a pyschoanalysis of self, it is tragic to see this cycle repeated over and over again in this life of mine, to have success grow proportinately with self-loathing.

I was in the late stage of this self-destructive cycle of mine when we met in Melaka. He was leading a band of Christian musicians then. It was strange that he noticed and approached me of all the people he saw walking in the streets of quiet town Melaka. We talked late into the night then and I travelled with them for another 3 days before leaving the Christian band to proceed further up north on my own journey which eventually lead me into the hearts of the Golden Triangle where the militias would hold in their hands a sub machine guns while collecting their bribes from you. I guess one must really have a death wish to venture into such places. Maybe I did, maybe I didn’t. Knows to who? What I did know was the urge to just keep pursing the direction of North. It was only when I finally found peace in that abandoned war torn church in Mong La did I decided it was time to head home.

It has been quite a while since me and Rodney last time met. If I did not remember wrongly it was more than 2 years ago. This time we met on Friday morning at Plaza Singapore for breakfast. Rodney hasn’t changed much, he was as jolly looking as ever. I was told over breakfast he have had for himself 2 operations since our last meeting. He proclaimed that while he was still young at heart, he was physically no longer that young anymore.

Unknowingly Rodney stirred our conversation that morning towards the clash of values. He related that one of his protege a student from Malaysia who recently went over to US for studies got involved with a Swedish girl who was also a student. He talked about how strange it was to the Swedish girl his boyfriend had to send money home every month ever since he graduated to his parents, when parents and off springs live somewhat financially independent lives in the west. This topic on value clashes did struck a chord. However in my opinion, what he described was a relatively simple two polarity scenario where all one had to do was to find a position between the stated two extremes as his own. What I have in mind is a multi polarity issue which to date I have yet found an answer to. Things would have been simple if things could be sorted so easily into right and wrong, good and evil. However the current situation is such that these values which I hold to be right and good are contradicting each other. Right and wrong does not exist, neither is there good and evil. I do not foresee I will resolve this issue that soon.

Milestones and more milestones

From the statistics shown in Google Analytics, the changes made to the web application thingstodosingapore.com over the past few weeks has thus far had a positive impact on on the length and depth of incoming visits.

  • Average length of stay has increase a 100%, from a pathetic 53 sec beginning of last month to 2 mins and 30 secs
  • bounced rate has  dropped to an average of 6% from the previous 12% beginning of last month

The results could be attributed to the following changes:

  • Migration of web hosting server to Singapore where up to 70% of the visits are coming from, a choice resultant from analysis of data available in Google Analytics, Alexa.com as well as valuable inputs from end users.
  • Ajax auto count of items in each category when visitor scrolls to a new section of the map resulting in preempt of information users could browse through. (A feature adapted from YouTube.com)
  • In the event whereby a visitor arrives in a page other than the main page, the starting category will be set to the one this content belongs to. The pheriperal markers will also be from the currently selected category. This results in increased relevancy in contents presented (A feature adapted from Google.com).
  • The migration from in map content presentation to the use of the Thickbox library. Users can now browse through the current content with more ease and less irritation. This is a choice resultant from the series of usability analysis I conducted.
  • Auto population of Event contents from Facebook.co. This is a feature inspired by Facebook’s newly launched Graph API and the existing Google Map API. The process is made more comprehensive by an ever growing internal dataset of locations specific to Singapore. After months of experience dealing with the Google Map API, I have concluded that while the Google Map Geocoding feature is pretty extensive, it has problems detected locations in the event whereby the location is referred in the local jargon.

Over the past few days, I have had the great opportunity to correspond with a few corporate event organizers and in the process receive some feedbacks. One of the features, resultant of my correspondences, will be the visitor analytics which event organizers will find useful.

Another set of features related to the first set will be the viral feature which I will hope to implement over the next two weeks, it will be somewhat inspired by one of the prototypes we have had the opportunity to build in Silicon Valley during the Techcrunch Hackathon. This second set of feature will be designed to increase foot track to events by organizers happening in Singapore, increase visibility of such events to the general public as well as aid in the process of customer retention.

Considering the fact that Singapore is in the midst of positioning itself as business hub where multiple international conventions will take place, the infrastructure for the online social media support for events is thus far minimal, if not non-existant. This might in fact be a niche that is uncompetited for and in demand.

Not only does ThingsToDoSingapore.com concentrate on providing valuable information to people interested in attending events happening in Singapore, it also complements these events with information on related facilities at nearby premises.

Another thread of work to be done is the android application. I have had the opportunity to bring the HTC desire I recently acquired out for a test run and have identified some issues of concern which I will have to that into consideration when building the Android Application that will allows users to gain access to relevant information on ThingsToDoSingapore.com while on the road. Thus far these are the issues identified:

  1. Content Synchronization vs Bandwidth Lag
  2. Content Synchronization vs Android SqlLite memory size
  3. Content Synchronization vs Battery Life
  4. Content Presentation vs Screen Size
  5. Content Presentation vs Relevance to User Experience

The options available for the Android Application are as follows:

  1. Map
  2. Listing View
  3. Grid View

While the above issues are important, there is a more fundamental underlying area which begs for my attention. I will need to identify the use cases that will be relevant towards the Android Application User experience when exploring Singapore on foot. This presents a difficulty for me at the moment as I have yet gathered a comprehensive set of data, enough at least to determine which features are critical to a Singapore explorer and which are not. The option available for my pursuit at the moment is that I do trial runs using other peoples application to determine the painful spots not covered as yet. However even, the resultant set might be biaised as it due to my personal experiences as Ricky once said. Hopefully this biased could be resolved via the feedbacks gathered from my test users (guinea pigs) after the initial prototype is developed.

The path ahead is still long. There is much to be accomplished.

It runs in the blood

“Sure, no problem, I will do it for free…” thanks to my Chinese upbringing  is one of the usual responses I would give with a smile when asked to do something for free. However to blame is the very same  Chinese upbringing for my omission of the second part of my response “…when I am free and when I feel like it. However if you absolutely must have it, please pay the price then.”

This phrase Alena said a long time ago still etches deeply in my mind ” to be Chinese is be somewhat hypocrite.” We never really do say what we mean or mean what we say but we really do expect the opposite party to understand what we mean and misjudge not the meaning by what we say. Conversation is thus very contextual. To a person from a culture where conversation happens on a more contextually independent level, the Chinese thus comes across as as a race that is highly deceitful and untrustworthy one. A person who truly understands the rules of the game in this part of the world would in fact come to appreciate this “face saving” gesture given in response to a seemingly unruly request, thereafter depending on the quality of his upbringing resolve this is conflict in disguise with a more appropriate offer.

Of course there always exist the odd one out whom despite, being born and raised in this highly contextual culture, choose to ignore the social norms and push ahead mindlessly/greedily his own personal agenda. Such fools are usually the ones who end up with a bad reputation, smiled at by most but ostracized by all.

Much as I hate it, and usually function out of this mode when I am in the other parts of the word, this practice has became so deeply etched in me, it is hard to discard when I get back to this parts of the world. A typical example could be seen from how my dad treats the helpers for his stall.

There is helper A whom when free would drop by in the stall to help out. My dad would insist of treating him to breakfast, as well as passing him some money and left over goods from the stall after the stall closes for the day, despite helper A’s insistence that he does not want anything in return. Harmony is maintained and bonds are strengthened.

There is helper B who draws a salary, is alway late for work, slow in action and consumes more goods than he should  from the stall during working hours than he should. He finally crossed the line when a few hundred dollars went missing from the cashier counter during one Chinese New Year. Dad finally let Helper B go thereafter saying business was not too good and he could not afford the continued hiring of a full time help. Grace is saved, however relationship has broken down beyond repair. It need not be mentioned by due to his natural disposition, he has been avoided much by people from his social circle.

Application of the framework from the Tipping Point

David K.K. a friend on facebook got my attention when contacted me two days ago stating his interest to have his events published on ThingsToDoSingapore.com. I instructed him that by simply signing in as a member on ThingsToDoSingapore.com, the system will automatically poll his account and his friends’ accounts for newly created events  every 24 hours. Just for curiosities sake I went and did a manual polling for his account after getting his account id. To my surprise his account itself contributed close to 300 new upcoming events. To gain more understanding of this phenomena, I went and did a study on David K.K.’s facebook profile. Results showed he is one of the top ten members who have the most number of friends on ThingsToDoSingapore.com. This fact brought immediately to my mind a key point which I ever came across from the book “The Tipping Point”.

All individuals are not equal by nature, the differences could be attributed to their character, personality and natural disposition, as such when analysed in the context of the social network, some people are spokes (connected with just a few main people) while other people are hubs (connected to many people). According to the theory preached in the tipping point, to gain wide coverage for a message, it is important to gather the support of people that are hubs in a particular section of a social network as ambassadors of a sticky message.

After gaining this new insight, I modified a section of my codes to prioritize the polling of members accounts for events based on their number of friends.

Collaboration across time zones

It was Machiavellic of me.  However being unfortunate enough to be on the receiving end of absolutely contradictory information from two unfamiliar parties, not knowing who to trust and faced with the need to move progress forward, I saw no other alternatives than to force a show down between the two of them. (aka do a back burn and then throw a bomb into the main fire so that it runs out of fuel immediately). The drama concluded today and the truth was revealed. I managed to find out both sides of the story and put them together in one comprehensive piece.

If only all such scenarios were so easy to settle. Right at home is one such case, which I have been working on that till today has yet been resolved. Damn family politics! Guess that explains why I devote most of my time and effort to work of some kind. It is simple and utterly rewarding just like what Pasha said one night during supper. A good day’s work gives you a  ”YEAH! I kicked ass!” kinda feeling.

Start of next week, I will need to put what I have been doing thus far aside and really start focusing on working up the business plan with Ben and his team, if we are to even to expect any funding prior to the start of our project. The app challenge will be starting officially in the month of January and I still haven’t done much shit on our mobile apps. I seriously wanna nail that competition and get our application showcased in quite a few technological and trade conventions as they claimed would be the case for winners.

However before that we will need to figure out the logistics, we are dealing with two time zones over here thankfully US is 16 hours behind which means 8 hours ahead in a typical day, so we could conference during mornings in Singapore and evenings in San Francisco. This is gonna be big, if not, then its gonna be a big flop Texan style.

Contrary to conventional practises, while most love to brag about their victories, I rather love more the idea of bragging about my failures. There more to be gained from in failures than there are in victories. Lessons learned from victories will soon be forgotten while lessons learned from failures will linger on for years.

Moving on

The campaign with Mr Gwee, the Feng Shui Master, will finally draw to a temporary close at the end of this month. It had been a long one, full of fun, intrigue and excitement which has last for 1 years and 6 months. However the general trend in the market has changed and what I can offer in the arena of his business operations is no longer of use. It is convenient it finally came to an end of its own natural accord, as I have been deliberating on this issue of abrogating my services to him  for the past 6 months to free up resource capacity of my own for other things.

Along the way he had handed me quite a few valuable words of advice, which on many occasions have proved useful in pointing the way for my path ahead, though not often construed in the directions he of his intentions. These words I will keep to heart as I move along on my journey into new territories.

During our final session together at his office, our theme of conversation revolved around surprisingly the decisive difference between the rich and the poor.

“The poor”, he said,  ”sits around in coffee shops all day thinking and talking, but never turn their thoughts into action.”

“The rich”, he said ” on the contrary constantly operates in the cycle of thinking accompanied by immediate action.”

“Wealth is a mentality and not something physical like big cars, big houses…. aka  fool’s gold ” he concluded ”  to grow big it is important to utilize the tool called incorporation. ”

Along the way too, he did express some disappointment that his son fell amongst the second group of people and not the first, earning a meager $7K per month. That however is a story for another day.

7 pm came soon and it was finally time to leave, I bid my farewells to Peter and his staff and step out into the streets. With this I have released the last of the hawsers, mooring me to this harbour which I happened to chance upon during the storm of 2009.

The sun was in its zenith. The breeze was blowing gently. The sails has been set. All hands on the deck then, it is time to cast off into the unchartered waters. The journey down the path continues and the next adventure awaits just over the horizon.

Superficial contemplation versus abstract contemplation

Last night while having supper at Holland Village with Garis, his girlfriend and Pasha, Garis mentioned that I have not stopped update my blog of late.  Or rather most of my writings has recently be restricted mainly to that of more superficial matters, aka the ones I posted on ThingsToDoSingapore.com.

In essence, contemplating on issues pertaining to that which is material is of lesser difficulty as compared to issues that are abstract in nature. An example belonging to the former category of contemplation would be “the sky is blue”.  It is trivial and requires not any deep analytical consideration past the point of making the observation that the sky is indeed blue.

An example belonging to the latter category of contemplation would be “can the sky ever be made to look green?”.  One comes not easily to a conclusion in this instance. The points for consideration are inexhaustive and requires extensive periods of contemplation. Furthermore, there could be multiple conclusions drawn from the same train of thought of varying validity. Contemplation of such matters comes also hand in hand with the consideration of the repercussions and opportunities should such a possibility come into existance.

To engage in abstract thinking, one requires a certain amount of personal space, in laymen terms time alone.

Of course, why the sky is blue and how it can ever look green is not of my interest.  The matter of interest for me has remained one and the same for the past 2 months ever since I left Silicon Valley. “What is the defining difference between a web site and a web application?” I believe this one statement will remain a central theme in my thoughts over the next year at least.

Borrowing the concept from “the world is flat”, technologist who consider himself a web site designer or programmer and who choose to remain so will soon be relegated to the back seat. With the proliferation of tools ,applications and increased availability of Chinese or Indian human resources, their professions has been slowly downgraded from the traditional chocolate topping to the vanilla cream. They will experience increasing competition and face lower and lower profit margins in the long run.

The new prevailing trend will require a new class of talents known as web application designers, people who are able to envision a somewhat abstract notion and turn it into a concrete working full featured web application, aka product designers of the web. A few days ago I heard over the news, Singapore will over the next few months start a Masters program for design. Quoted from one of the spokesperson “this program will attempt to boost the level of innovative thinking amongst the human resource belonging to the Singapore workforce”. This short piece of news vindicated my matter of interest to be current and relevant.

Back to the notion of the web application versus the web site. At a glance the mere observer might not notice any difference between these two groups. However further analysis will reveal that a web application is in fact a sub set of the web sites group, with a restricted set of functionalities that aim to fulfill a small related group of use cases. It may or may not be social in nature however it will deploy very extensively the use of Ajax, taking into deep consideration the usability experience as well as programmability (exposed APIs).

Good examples are

  • google document for word processing
  • google mail for business communications
  • twitter for short messaging and rss feeding
  • facebook for social network
  • foursquare for mobile location exploration
  • tweet deck for real time trending
  • google map for geo location
  • youtube for video crowd sourcing
  • flickr for image crowd sourcing

During this period of consolidation, I have been engaged in the building of two prototypes. The first as all will have known by now is ThingsToDoSingapore.com. To date, it remains still a conceptual experimentation of what the new web has to offer.

The other which will remain underwraps until it is ready for launch is a mobile based application that runs on the Andriod platform, the choice of which is well explained in this article Woz: Apple Almost Launched A Phone In 2004, Android Will “Win The Race” on TechCrunch.

The new trend also brings into this new consideration of what is truly worth building. In the event whereby a tool already exists, the coming into being of a second tool of similar nature without the potential of fulfilling at least one critical use case that has not bee thus far fulfilled by the first will be somewhat worthless and waste of time to build. With this in mind and taking into further consideration that time is a priceless resource that money cannot buy, I have become hesitant in taking on any new projects that is in my opinion not worth coming into existance, despite the enthusiasm of the potential client. Hence in conclusion my time and effort has become seemingly decoupled from money.

One thing leads to another

With age and experience comes understanding,
With understanding comes the knowledge that somethings must never be pursued,
With the knowledge that somethings are not meant to be comes deep regrets,
With deep regrets comes profound sadness,
With profound sadness comes great emotional depth,
With great emotional depth comes great inspiration,
With great inspiration comes superb motivation to acquire more understanding and experience.

These set of relationships is non-sense. I think it is time to go to asleep and stop brooding on what Ern expressed to me.

“Since the age of twenty, I have not been happy” that was what she said to me the other day. She choose to embrace this understanding in contrast to Orlando who choose to avoid it.

“I choose to surround myself with people, so that I don’t have to think. Too much thoughts, I cannot stand it” Orlando’s way of life.

Good night my friends.