jay's old blog

this blog will be deleted soon - please visit my new blog - https://thesanguinetechtrainer.com

Project TD - Day 55 update – A Word About Microsoft Azure

[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

I have been working on Project TD for about 55 days now. It has been an interesting experience because of what came before Project TD, and the incredible potential TD holds, if it should be fully realized as per its original vision.

Day 55 is also significant in many ways because, till now, most of the blog posts have been about explaining the concept, the design and the work process. All this is good, and obviously essential. If one were to compare this to a vacation, then, till now, I have been mostly planning the vacation. Deciding on the places to hit, the modes of travel, the clothes to pack, accessories being purchased and so on. It’s all happening in the drawing room. It’s like the first half of a heist movie.

Now, things are getting real. The vacation is planned. The places chosen. It’s time to step out of the comfort of the home and go outside. Time to carry out the heist, and I guess, I am stealing knowledge, to complete the metaphor.

For the last couple of months, I have spent a small fortune (to be honest, it’s a lot of money for a small time businessman such as myself) in setting up all the necessary hardware. I have got the computers, power supply, mobile devices, tablets, office space and everything else. In each case, I also had to invest in a backup system, which simply multiplied the cost by 2. All this is the ‘planning part’.

Now, all these ideas must become real, and the services must run on the cloud. The cloud is the backbone. The cloud is where Project TD will live. I have been using Microsoft Azure for a little over 5 years now. However, I have only been using it in bits and pieces. To host a website here, run a mobile data service there, setup a notification center here, host a virtual machine to test an application and so on. None of them were related to each other, so it wasn’t a complicated system. Project TD though, by any measure, a complicated system. It will push my ability to harness the power of the cloud to levels I have never done before. Almost every part of this system can be automated, post deployment. I will taking advantage of that too and see how far ahead automation has come.

And the monetary cost! Man, I have going to be paying huge bills for at least the next 2 years!

While the cost – time and money – is huge, I am convinced that the cloud thing is pretty much in place now. As the amount of data people consume is only going to increase, the requirement for cloud services will only increase. Project TD will help me figure out how to make multiple cloud components work with each other in a rather complicated system. By extension,  this should provide me insight into what is best fit for the cloud and what are not.

I will be the first to admit that Microsoft Azure is expensive when compared to Amazon and Google. However, I find that Azure has a lot better support to Visual Studio, which is a deal maker for me. Microsoft is one of the big dogs (if not the biggest dog) in the cloud business, and they are constantly adding new stuff to their cloud services.

So yeah, Dear Mr. Azure, don’t let me down man.

Follow me on twitterfacebook and instagram for more updates. Thanks    

The Machines Are Coming – Automation Is Nigh

I have previously touched upon automation here, and here. I am forced to revisit the topic again because now things are becoming a little closer to comfort. Two weeks ago, a friend of mine who works as the lead in the database team were discussing about tech in general and he told me that they have received official emails from HR elaborating upon the effects of Automation. Then, the same thing with my own office, and other places too. Till only a few weeks, this was all just rumors and news clips, but now it seems like this thing is really happening.

In a way, it does not take a super genius to realize that this is happening. For five years now (since I became an independent consultant) I have been cajoling, requesting and begging any and all students and professionals alike that run into me to think of this day arriving. Some (may be 1 out of a 100) took my advice, but even in those 1’s who took my advice, very few actually followed through on the advice. In some ways, it is a bitter sweet moment for me because my predictions and forecasting is coming true (sweet) but so many of these folks who had 5 years to prepare will soon be unemployed, or continue to remain unemployed (the bitter part), at least in their chosen field of career i.e. software. Sure, many of them will have to migrate horizontally to unrelated industries (because, let’s face it, jobs are always available for skilled folks) which is sort of a bummer.

Repetitive tasks have always been under threat by machines since the invention of machines. I remember watching this old Hindi movie Sholay, where there was one guy who’s sole job was to put coal into the coal fire room. Of course, later trains replaced this manual fueling with automatic fueling, so that guy’s job was doomed. Horses were replaced by cars, so all the horses lost their jobs. Traffic signal systems took over the jobs of many traffic signal folks. Closer at home, (and brag shamelessly) I am now working on an assignment which only 2 years ago would be done by at least 15 developers.

This is not to say that I am so awesome. It only goes to say that code automation has improved so much, that more can be done by less. I mean, entire trains can be manned and operated by no more than two train folks, I am sure. Testing is become automated like crazy. There are websites that allow interested business folks to build entire solutions (as long as they don’t need super serious customization and are happy with the available default options). On an individual level, I manage my own domain, servers, databases, backup systems, email service, all thanks to Microsoft Azure, with surprising ease. Tomorrow, if I should end up building a company, I doubt if I will have a dedicated IT team. I will simply manage it myself (on my smartphone, no less!), and spend zero money on either hardware or software. Again, the magic of automation!

10 years ago, I would have to spend money on everything from hardware to software to air conditioning and some guy to manage all this. Of course, there have always been managed IT solutions but they come with their own baggage like annual commitments, fixed cost operations and of course, limited flexibility. I mean, let’s say I sign a contract for 3 mail boxes, I would have to do it for a year. Now, if I layoff 2 people the next day of signing the contract, I am stuck with the bill for a year. With the cloud email service, I use the admin panel and simply cancel the extra email box. At most, I may have to pay for the current month and be done with it. Later, if 2 more employees should be hired, it takes a few minutes at most to create them new email addresses.

The more I think about it, the more I can see the appeal of automation, from a business perspective. Isn’t this what is already happening with online services? If one where to look at non-online stuff, I would rather shop at a super market (where I can pick items at my own leisure) than at a mom and pop store where I provide a list of items and the shopkeeper picks up the items for me. One reason I prefer booking tickets online is because I don’t want to deal with the ticket guy or girl who slow down the whole process, what with the time it takes for them to print the ticket out, look for the change, chit chat with their co-workers and being forced to smile and all that. Of course, getting them to give me exact seats I want (even the movie hall is empty) can be a hassle sometimes.

Overall, the essential idea (while detaching myself from related ethical concepts) is to remove the human element or reduce the human element entirely. For instance, many movie halls have now begun to allow patrons to simply flash the online ticket directly from their smartphones so that even that small step of ‘collecting tickets’ is eliminated. Does it mean that we should not interact with humans at all? I am not going to ask the ethical questions related to that, so I won’t answer the question here, nor is this the place to do the same.  Of course, better people, and more influential than me have already asked such questions. The banking industry asked and answered that questions when they replaced bank tellers with ATMs. The government asked and answered the question when they replaced traffic signal folks with automated traffic signals. Large IT organizations will now have to ask and answer similar questions as ever more software related tasks are taken over by automated engines.

As with the earlier blog posts, the solution to any challenge is to see what is being removed and what could be replaced. The march of technology has always been about allowing others to up their game. When cars started replacing horse carriages, the same folks who were driving the horse upped their game by learning how to drive the car. Horses had limited range which reduced the distance that people travelled. Once cars came, the distances increased. Costs became lower, so more people could afford it. Same thing with ATMs and Traffic Signals. People were given an opportunity to do more in their lives. They no longer had to do a menial task of checking the books and collecting cash/giving cash. They could do more with their super awesome brains. Let’s never forget. Our brains are awesome! When the brain starts doing stuff, more things happen, and the economy becomes bigger, creating even more jobs. Awesomeness leads to even more awesomeness.

Obviously, I have been preparing for it for the last 5 years. Unless my preparation is fatal, I would like to think I am already equipped for an automated world. I have believed that what I do (the work of a developer, and I am only halfway decent) and what we developers do is integrate systems. No matter what project I have working, most of the time is spent integrating different systems.

Take my beloved Project TD for instance. Although I am developing it, the project itself depends on all sorts of external, open source projects for everything. The IT part (servers and related stuff) is also automated. I don’t have an IT team managing anything. I am managing it with no input from any third party. Project TD is already doing extensive automation of so many things. Visual Studio already does so many things automatically for me. As have mentioned many times in the posts related to Project TD, I am harnessing every possible iota of automation so that I myself can build the entire thing with little or zero outside assistance.

In other words, although I am technically a developer, I am spending all my work hours, integrating one system with other. In other words, I am an integrator who also programs and codes and develops when the occasion calls for it. To some extent, integration is perhaps that one thing, automation is yet to do. Automation depends on the assumption that a given task in repetitive. Integration is rarely repetitive. For instance, the process of buying a movie ticket is repetitive (once the movie is chosen, of course) but the selection of the food items during the interval is not. I might choose to combine samosas with extra sauce or go with chutney and coke. Alternatively, I could pick samosas, no sauce or chutney, and opt for tea to go with it.

Similarly, anytime there is a matter of choice, a human must be involved. Anytime there is repetition, automation will take over. Developers who are aware this will equip themselves to make sure that their skills are aligned on the ‘choice’ part of the software development. Those who refuse to see this (either by design i.e. denial or accident i.e. they are unaware, not being educated) … I don’t know.

I just don’t know. 

Follow me on twitterfacebook and instagram for more updates. Thanks!    

Error and Logging

[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

A system, no matter how good it, should be capable of giving the right communication. This applies to people too. If there are two individuals, one of them an excellent work person with poor communication skills, and the other with average skills but excellent communication. On my best day or the worst day, I will always go with the person who has excellent communication skills (and medium work skills). That’s because a person with good communication is the one who will keep me informed about the good and the bad stuff, no matter how comfortable/uncomfortable the truth is.

On the outset, this all seems like common sense. However, in all the years I have been working, common sense is perhaps the rarest of commodities. People do stupid things, and then hide behind a veil of ego and false claims and losing touch with reality. That is why, as I continue to work on my app ecosystem, I realized that it is essential that it has a proper error and logging system.

From a strictly programming perspective, error and logs, both do the exact same thing. When something happens as expected, it is ‘logged’. When something unexpected happens, it is also logged, but shows up as a ‘error’. For instance, when a user signs in successfully, that is a ‘log’. A user signs in but it does happen for some reason, that would be an ‘error’.

A collection of logs is useful for making decisions related to performance optimization. For instance, if every user in the system is taking 10 seconds to sign in, and I discover that there is a new library that allows signs in to happen in 5 seconds, I know I need to implement that. There by saving 5 seconds for every user on the system. Logs are more of a proactive measure at making a system better. We use the collection of, the collection of (two repeated collections, not typo here), logs to find out what about the system can be improved.

A collection of errors is useful for finding out what is going wrong. This, unlike the logs, are about reacting to something that is going wrong. Let’s use the sign in example. If 10% of users are having difficulty signing in, then that is a problem. A problem that needs to be fixed as soon as possible, and the system should automatically trigger when the error percentage rises. Inform the operation folks and established documentation should also be automatically triggered allowing emergency response to start working.

In fact, if possible, the system should not only diagnose the issue on its own but also, if possible, fix it on its own. I could be throwing darts in the dark here but I think this is what machine learning is all about. The machine learns from its own mistakes and then starts doing things. Like say, fix problems with no intervention from anybody. I think this is practical and even possible. By my own experience, mistakes are like wheels. Some wheels need to be invented, while others have already been invented, and hence need not be reinvented.

Let’s say, the system logs an error system for the first time. This is the first time, so a human is involved in fixing it. Once the issue has been fixed, programming is done so that next time, when the same error is triggered, the system will attempt this fix. That means, no human need to be involved. The human can now focus on fixing new issues, instead of reinventing a wheel that already has been wheeled out last time. Machines are good at repetitive tasks, and that means, if something has been repeated, I don’t quite understand why humans should be involved? The time spent by the human re-fixing what has already been fixed could be used for something else. That time could be spent fixing new issues. That time could be spent improving the system, like making it faster or better. For all I know, the human can use the same time to watch a movie or fall asleep on the couch or go for a walk. The idea is to avoid reinventing the wheel, let the machine do what is already done and the human do something new and exciting.

So yes, my app ecosystem will have a machine learning enabled error and logging system.

Follow me on twitter, facebook and instagram for more updates. Thanks!

Automation and Self Driving Cars - Part I - Where will the drivers go?

CES is currently live and there is a whole load of information coming in about upcoming products. Some of it, either directly or indirectly, has to do with self driving technology. Technology is almost always about making things better, faster and simpler. More popular technologies usually end up replacing something that currently exists, by doing it in less time and less effort. Last year I had a 2 GB RAM phone, and this year, I will have 4 GB. Last year I had a wireless keyboard with a dongle. This year, I own a wireless keyboard that foregoes the wireless dongle, making my life that much better. This is how stuff works and that in turn makes other stuff better, and life evolves.

A similar evolution is taking place (at least in the big cities) in everyday life with app based taxi services such as Uber and Ola. As a everyday user of these services, I know a thing or two about. For one thing, the invisible hand of a larger company ensures that I am guaranteed a minimum standard of experience. Thanks to this, Of the 1000s of times I had to commute in the last 2 years, I may have had to travel by auto or regular taxi only a handful times. Today, the auto and regular taxi stuff still exists but I wont use them, instead opting to go with app based taxi services. As a result, a few years down the line, I imagine that the regular auto and taxis will completely dissapear and all we will have are app based services. This is even more evident when one sees that many existing taxi chains (who worked on a phone based or road hailing based) are either closing shop, trying (and mostly failing) to upgrade to a app based model or staying float only thanks to government based regulations and protectionist measures. One way or other, the app based model will probably win out, simply becuase it is convenient and frankly, more innovation and hence a higher chance of success.

Irrespective of who wins (app based or the old fashioned taxis), one thing is for certain. Commercial driving is poised to grow and it will continue to provide employment to people who have, perhaps, no other option. I use Uber like about 20 to 40 times a week, and I also tend to be pretty chatty. The drivers are by default chatty, and most of them or men in the age group 20 to 40. Many of them have limited education, and have migrated from the allure of city life and better earning opportunities. Commercial driving makes sense to them becuase of its low educational as well as skill requirements. Getting a drivers licence only requires basic education, so almost anybody and everybody has it. Further, obtaining a car is also easily done as the car itself acts as a collateral, making it easy for financing institutions to work with it.

The main appeal of app based services such as Uber (and by extension others such as Ola) is that they dont want people owning their own cars. Uber's propsition varies depending on the country it serves. In India, it tries to position itself as the company that provides 'a driver' for the middle class Indian who likes to be the boss who travels in the back. In Middle East, it positions itself as that company that will safely transport the women becuase women arent allowed to drive. In the developed nations, it talks about stuff like reduced traffic, pollution, parking space, debt (of owning a car) and whatever other things that bother the high income life of the wesetern economies. All in all, right now, Uber wants to have a lot of drivers under its payroll, who will ferry millions (and some day, billions) of passengers from point A to point B. Of course, let us not forget that Uber by itself does not own a single car. At the end of the day, it is a platform like Amazon. Each driver owns his own car, which shifts a substantial risk from the cab company (which is Uber) to the individual driver. Despite this shift of risk, perhaps the biggest expense for Uber are it's large collection of drivers, the same folks who bring in the money.

Folks who own shares in Uber (and Uber is, and continues to be, one of the most valuable companies in the world) are constantly thinking. Thinking about what? Increasing profits of course! They will have a keen eye on this autonomous driving technology. For all we know, many of them probably have direct stakes in these companies that are developing the driving tech. Ealier, rather than later, we will have self driving cars on that road. That would (or already has) give the obvious idea that their biggest cost on the balance sheet is the 'human' i.e the driver. If the driver is eliminated, imagine the cost savings!!!

Come to think of it, stuff like this has already happened. I remember when I was a kid, ATMs were going up everywhere in and around India. Before the advent of ATMs, a standard Indian bank branch was filled 'bank tellers' like crazy. Despite the low penetration of bank accounts in our country, those who did were still a large number. Folks would visit, and stand in serpentine queues in the bank to withdraw cash alone. The large is the bank, the more tellers would be sitting and doling out cash. Private banks had smaller queues and government banks had longer queues. Some would say, 'going to the bank to withdraw' was almost like a social event. You go there, spend half a day, and then return by evening with the cash you wanted. Then, internet finally came to India, and banks realized that they could eliminate most of the tellers and replace them with machines. That's precisely what they did. A scheme called Voluntary Retirement Scheme was introduced (which was a polite way of the banks saying to its long serving employees that they were no longer needed) which was pretty generous by all accounts. A lot of bankers took the offer because they thought it was one simple way to cash out. Further, those were the days when Indian IT industry was booming like crazy. Many who took the VRS had young sons and daughters who had ready jobs waiting for them. Back then, getting a job in software was easy. I myself saw many of my classmates (100s of them) who did not how to write a proper for loop getting plum jobs simply because the IT industry wanted people. It did not matter if they dont know the difference between a constant and a variable. This is an important note because the loss of monthly income for those who took VRS was offset by their sons and daughters who themselves had confirmed employment.

That brings us back to self-driving cars. Once companies like Uber realize that they have an opportunity to cut their 'human' costs just like the banks did, they will do everything they can to make it happen. Just like the banks, Uber will not outright tell its drivers to go home. It will probably 'phase them out' just like the banks did. It will tell them that they are no longer needed in a polite way. Of course, if the polite way is not possible, alternatives will be created. This may happen in 5 years or 10 years. Perhaps much earlier than that. By extension, individuals who wish to own their own car will also take into self driving cars. Eventually, we will have entire populations who suddenly find themselves travelling in a world which is less accident prone that gives the ultimate gift of all. The gift of time!

Continued in the next blog post, 'The gift of Time'.

Follow me on twitter, facebook and instagram for more updates. Thanks!