How College Plus backend got automated?

How College Plus backend got automated?

- 3 mins

Everyone of us remember our firsts, the first success, first failure, first medal, first trek. I know where are you going now, hold on! Today I’m going to talk about my first project in my development journey. College Plus. 😍 I learned Android development while working on it.

Watch the demo on Youtube: https://www.youtube.com/watch?v=SlszrEz5KJg

College Plus Android app provides information about mess menu, lecture schedule, bus time table etc to college students.

Here is the link if you want to explore or use it:

Following were the challenges I faced while developing College Plus:

Challenge 1

I knew that mess menu, lecture schedule will change from time to time. And it will become too hard to maintain an app with updated on play store. So keeping this in mind, a dedicated server was deployed which is called by app n order to fetch the latest data. And ya it worked. I was very happy to see that the app which I developed in my first year got adopted by the majority of the college 💚.

Challenge 2

After a few days, I realized that it is taking a long time to manually update data from PDF (which are shared by college authorities) to an externally hosted database. Mess menu usually get changed every two months, the lecture schedule gets changed every semester i.e 4 months. Sometimes it was taking ~12 hrs to manually update hundreds of entries in the database. Data entry is one of the most boring work and nobody was volunteering to do it.

I talked to a few people to solve this, all suggested to create a portal for admin and ask college authorities to directly update in the database. But this was again a lot of coding work and most importantly coordinate with appropriate authorities.

One fine day, a thought came to my mind why shouldn’t we automate this process. I mean let’s write a script which converts PDF to JSON and then pushes it to the database.

And the next day of this, College Plus was again back to work 😅.

Here is how the whole process got executed

The main challenge for me was to convert PDF to JSON which is a memory consuming task on a free server resource available to me 😜.

So

And now the real work:

System Design

Files are uploaded via a web portal. The system verifies that files are uploaded by authenticated users. Then preprocess file by splitting the file into multiple files for fast parallel processing and add the task into the queue. Now the different thread which is running in background founds that there is a new task to execute, it picks up and gets JSON from PDF. Now the system removes the noises from the JSON data and pushes to the database.

Now to scale this system, background worker instances are increased to handle multiple requests.

Watch the demo on Youtube: https://www.youtube.com/watch?v=SlszrEz5KJg

Learnings

Conclusion

“Never think that it is of no use, everything has its own importance, just wait for right time.”

Vishwesh Jainkuniya

Vishwesh Jainkuniya

Learner

comments powered by Disqus
rss facebook twitter github youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora