jay's old blog

this blog will be deleted soon - please visit my new blog - https://thesanguinetechtrainer.com

Project TD - Day 41 update – Core Tech For The First Batch of APIs for the API Engine


[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

Earlier, I have written extensively about the design technique that I plan to follow to build the API Engine. The API Engine itself is a collection of APIs. For the last few days, I have been analysing the APIs that will make up the first batch of contribution towards the engine. I noticed that there are some essential tech components that they need to use to make it work.

Going by the design that I put in place, this is the second bubble from the API design titled as ‘Identify all Tech’.

So, here is the full list of tech’s that are identified to make this happen. The blog post image above lists it.

  • Build and Deploy an API
  • POST and GET to the API
  • PUSH and PULL from the API to the Data Service
  • Google Maps API on Android, Web and iOS
  • Facebook Login API on Android, Web and iOS 

I will of course, follow the design plan, and repo the above stuff, tutorial it. After that (these steps will take a few weeks to complete), I will finally have some serious unit testing to work upon.

As they say in the good books (mostly, there are no good books that I refer to. It simply me using them as proxy :P), exciting times lie ahead!

Follow me on twitterfacebook and instagram for more updates. Thanks

Project TD - Day 28 update – Design for the Modules of API Engine



[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

I have already blogged that the entire app ecosystem depends on what the heart pushes and collects. The heart being the API Engine. I have written multiple times that the whole system must be modular so that parts can be replaced and re-engineered if something goes wrong. Or, if there is an opportunity to make things better tomorrow, then, that option should be readily available.

To that effect, I have come up with a design plan (pictured above) that should drive all engineering efforts towards the API modules. It’s a simple and plan design, but it should be enough to help me. Obviously, it is subject to further revisions if the occasion should call for it.

Pick an API

The API Engine is but a collection of API calls running on a single API server. On one side, it provides access to facilities to end users. On the other side, it has the necessary access to the underlying database. As of now, 12 API calls that would form the first batch of API services that will be part of the API Engine.

The idea is to pick up, one API service (each of them will have their own designs) and then start working on it. First step would be figuring out the design which means, illustrations, finite discussions about its usability with the stakeholders. Once the design is confirmed to work (to the best of data available), then we move on the technical part of it.

For instance, for the patron app, there would be one service that pushes the restaurant data to the server. This would be one API call. So, I would start off by designing the work flow. Show the workflow to as many people as I can (my, mom friends, my tech mentor, the many students who are under charge and of course, random strangers) and collect its feasibility. Use this feedback data, and design the final workflow.

Identify all tech

At this stage, the design and the flow of activities of each identified API has been done. The API design itself is completely independent of the underlying tech that will actually power it. For instance, the design is like I wake up in the morning and decide to have breakfast. The actual activity of eating breakfast might involve many things. I could have a pack of biscuits. Two packs of milk. A luxurious breakfast at my folks place. Or go out and pay a decent amount and generous tip at my favorite restaurant. As you can see, the decision to have breakfast and actually having it, though related, are quite independent of each other.

Similarly, the design is always independent of its implementation. It’s part of the modularity and replacability thing I keep going on and on about. Here, I will be thinking only of implementation and not question the design choices.

Earlier, I mentioned an example API service that pushes data from the patron app to the cloud. So, I need to figure out all the technologies that will make this happen. It would mean, I need to build a web app, android app and a iOS app that has a simple button. This button would collect data (that mean, I need to figure out how to collect data on all these platforms) and trigger a cloud push on button press. Like this, I break down the design into individual components and see how the code would actually work for these things.

Repo it

Once the tech part has been figured, the next part is pretty straight forward. I have already promised that a lot of stuff related to the project would be open sourced. Anybody can use it non-commercial uses. So, I will push the code of the web stuff, the cloud things, the app solution, everything on a public repo.

Tutorial it

The repo will have the code. As is my standard practice, I write pages and pages worth of documentation within the code. However, it helps me (and my career) to have some tutorials that point to the approach I took while converting the design into runnable code.

Unit Testing

Once the repo and tutorial is done, I need to check the individual fitness of the resulting module. This will ensure that the unit itself is independent and will involve writing dummy external connections.

Integration Testing

Once the unit is confirmed to be fit to work, next comes the part about integration. I have to check if this module will work with the rest of the modules. I also need to check to make sure that the system won’t come crashing down when the module dies or goes corrupts or simply stops working for some insane reason.

Once the integration testing is done (if it fails, I will probably have to restart at the design stage and work down from there), I put the module aside, and start over by picking up the next API service to work on.

Rinse and Repeat

This part is useful when, at a later point, if some new tech is introduced which will make things faster. Or perhaps some technology on which the app depends gets rebooted and revised. For instance, Microsoft might replace their existing mobile data service with a new one. Or, Facebook might implement a new login system. There are so many gears moving here man, anything can change.

Anticipating change and being ready adapt is the only way I can keep this engine relavant and allow it to evolve.

Note about Sketchbook from AutoDesk

I have upgraded from hand written illustrations to using the Sketchbook app by AutoDesk on the iPad. For the purpose of drawing, I am using an Amazon Basics stylus. I must confess, there are some serious advantages to using this over the hand written thing. Obviously, the free app has certain limitations. For instance, I can only have 3 layers (which is alright but annoying) and I cannot select components (which is the biggest handicap of the free version). Eventually, I will have to get the pro membership, but I have to evaluate this system first. Once I am satisfied, I will kick things up a notch with the pro membership which provides tools for desktop and android as well. It is fairly inexpensive and should be useful.

The Amazon stylus is extremely basic, and there is a learning curve involved here as well. I feel that it is good enough for now. If this art design thing goes to the next level, I will probably spring for a Wacom stylus for a better control over the designs. For now, I am still trying to figure out the stylus.

Follow me on twitter, facebook and instagram for more updates. Thanks!

Project TD - App Ecosystem – Day 16 – Designing the App Engine and API Development Status Dashboard and User Account


[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

I have learnt (perhaps the hard way) that a project without a deadline, is not a project at all! Sure, the deadline may get extended, and the project may die away. That’s the negativity in me speaking. Of course, the deadline could be met, and the project may actually reach something that resembles completion. 

Project TD needs to hit beta phase in less than 10 months. The official launch is less than 6 months after that. I am racing towards a deadline, and that means, I need to pick up a starting point. The requirements planning is all good and fine, but at the end of day, some programming needs to be done. The more I think about it, the more I feel that the API Engine is where it all starts. 

That brings up the question -  what is the API Engine? Everything and anything that works with some service is an API. For instance, the app ecosystem will use the Facebook provided login system on the end user side. The admin folks will use a Microsoft Identity library. I do not think I will bother with providing alternate login systems for end users. This does limit access to some extent. For instance, users who don’t have a facebook account might choose not to be part of Project TD. 

That should be okay. As I have already mentioned, this entire project will be open source non-commercial usage. So, if someone wants to include some other login system, the design is modular enough to allow another developer to latch on Google or Twitter or email registration based system. I must understand my limitations here. For the foreseeable future, this entire ecosystem will be built by me. If there is an opportunity that can be exploited and work load can be reduced. The login system, at least for the end user, I am going to limit myself to Facebook. I like Facebook. The Graph API is simple to use, works on almost any platform and it has the color blue. 

That’s the story with the login system. And the above Facebook login system is made possible because of the extensive login API’s that the Graph API (Facebook branding for their API stuff) provides.

Anyway coming back, the API Engine will be the collection of all the API calls, the entire ecosystem will make. That includes the many apps that will be used by the restaurant patrons and the owners, the admins, the ops folks and other roles that might be included as the design evolves. The API Engine is the only way the users can interact with the database, which stores the data. Now, the API engine (which is essentially server that listens and responds to calls from the users) is the store front for all the apps. On the other side, the API Engine interacts with the database, processing all the stuff that the calls are asking for. 

This allows for a breakdown of responsibilities, which is essentially, the whole point of modularity. Thanks to this, it is possible for me to design redundancy around the database. Build backups, replace when data goes corrupt (or gets hacked) and replace, hopefully on the fly. Since everything is broken up into pieces and connected with bridges. The source and destination don’t really care where the bridges actually go on the other side. All they care is that, on their side, the bridge is fine. 

This API Engine, is like an island at the very center. A huge island that has everything. Below the island (the API Engine) is the underwater volcano. Think of this as the database. Now, this huge island is surrounded by small water things like tiny islands, ships, ports and boats. All of these folks need something but they cannot talk to each other directly (that is whole point because boats die over time. We don't want the bridges to die with them, bringing the entire system down with them). They are all connected to the island, and everything goes through it. The island, on its own is nothing without the underwater volcano (the database). 

Thinking backwards, the center of this whole ecosystem would be the island and the volcano under it. Once these two are in place, the rest – boats, ships, sea gulls, ports, sharks and dolphins – will start settling around it. Everything in the system can be replaced of course. The island, volcano, the hundreds of boats, ships and sharks. Redundancy is the name of the game man!

As a consequence, I am now spending all my time designing the list of APIs that will from the first set of API calls, which together will become the API engine. As with the island-volcano thing from above, the API engine is useless without a database. So, along with the API engine, I will also have to design the data storage system. For both the database and the server that will respond to API calls, I am going with Microsoft Azure services. I have enough experience dealing with Microsoft Azure, and it has every possible thing imaginable to build and deploy pretty much every aspect of the ecosystem. There are alternatives (Amazon comes to my mind) but I simply have had better exposure to Microsoft services, and despite their higher charges, I will go with that. 

As I design (and then build the API Engine and the database), I figure that I will need an API dashboard that will keep track of the API calls currently being designed, developed, tested, deployed and available. This should become extremely useful to allow the stakeholders to keep track of what is happening, and obviously, what is not happening. 

The dashboard will be the choice of tool for the operations team whose job is to make sure that the system is working as designed. This also means, they need to be provided with the necessary information to deal with problems when they occur. Then, when the issues are fixed, the data should reflect the resolution. This also, will be routed through the API calls, and will be part of the API engine. The smart thing to do would be to separate the API Engine to multiple parts. However, if the entire system is backed up (and it is simple enough) then it makes sense to maintain just one engine, and then have backup engines ready to go when things go bad. 

The interesting thing here is that, the island and volcanos are separate. Each of them have copies of them, waiting in the pipeline if disaster should strike. If I split the engine into multiple parts, the redundancy maintenance factor will go up by that much. For now, it seems wiser to simply have one system to maintain, instead of multiple specialised systems. 

Next up, I must pick up the design tools which is where my new iPad and the pre-owned printer comes into the picture. I will talk about that in the next blog post. 

Follow me on twitter, facebook and instagram for more updates. Thanks!

Choice of Technology for API



[Ongoing series of blog posts to inform potential developers, users and (hopefully investors) about this new app ecosystem I am architecting, designing, developing and deploying. More details at this page]

At the core of any ecosystem is the API system. The API is consumed by the apps, that are in turn used by the users. They could be the apps used by the operations team. Apps used by the administration team that oversees user content that is pushed. Apps that are used by the restaurant owners and of course restaurant patrons. All of them will interact with the cloud storage that actually serves and takes data over a collection of API calls.

That is why, the choice of technology used to build the API is important. As much as possible, the whole eco system will take a modular approach to development. If some technologies should die (which happens more often than you might think), we should be able to remove things and replace them without affecting the rest of the components in the ecosystem.

Although the API system is also modular (at the end of the day API calls are but simple web routes) and follow the remove and replace principle. Still though, rewriting the API itself will be a huge task. That is why, if I am going to choose an API technology it must be something that I will keep for a long time and only replace it if I have no other choice.

There are quite a few technologies out there that can be used build RESTful API’s – Node.js, PHP, Python, Rails, Dot Net and Java.

First up, I am not a big fan of PHP. Because, with all due respect, PHP HELL.

That leaves other technologies of which I am not going for Java because, I have already invested a lot on c sharp and I am not going to spend an equal amount of time on Java, unless it is a matter of survival.

Python, Node.js and Rails are things that I am familiar with but it makes sense for me to leverage on my existing dot net skills, which is where DOT NET based API building comes into the picture. As it is, the project uses a lot of Microsoft technologies (except for the android mobile, almost everything else depends heavily on Microsoft related technologies) and using the same thing for API reduces total effort time.

However, as far as planning goes, if it comes down to using an alternative, I would side with node.js. It seems to be the one that is much more modern and well document among other alternatives. So, that would be the alternative. Yeah.

[Last Updated February 10th 2017]

Follow me on twitter, facebook and instagram for more updates. Thanks!