One of the first projects I got in the company, that I worked at that time, was to observe and automate a process done for a VIP client (automotive industry giant). This process in essence was done by one technician and two key accountants in a period of 2-3 weeks, dependent on the amount of updates and corrections needed. Still it's a big data company, so huge amount of data is in game.
First the technician creates an input file for the legacy app and feeds the process with it. Of course he gets the instructions from the key accountant (better to say from the client). After that the app do the calculation for a day and spit out some 35 xls files with 5-8 sheets per file, full of data. That is the point where the fun begins and I jump in.
The end result was to be organised in 5 xlsx files wit 4 sheets per file. They should contain sums, groupings and sample data presentation all extracted, recalculated and organised by dynamic rules set by the client. And when I say dynamic I mean the client can suddenly express a wish for different sample data. So the manual processing of data was painful enough without any changes and with them often they would enter the third week of work. So in essence when they finish the monthly report they can start the new one.
It was the first time I tackled this type of task and I choose Python to solve it. I knew he had the tools needed for this. Although I was not certain which where they. Still the task was new, intriguing and challenging. Was it a test, or what ever, but the task that was consuming time for years in the company got the highest level priority all the sudden. Like it was not enough headache that I have to read all the documentation, organize the data and create a precise schema what and how I want to do.
I'm code purist, I really like clean and organised code (of course to the extent it is possible to do). One of the reasons I love Python is that it forces other developers to write it in clean manner. And I saw really bad code in my career. Some so bad like - unreadable bad.
One more bad thing I try to escape is writing classes just because. If something is done by other people, learn it and use it. If you see place for improvement ,do it, this is an opensource community. Do it and commit it, contribute to the community. But something even worse that writing things, already done, is taking them and overriding half of it. Now this is a personal opinion and you don"t have to agree, but from my perspective this means that the code you took is useless. Else why overwriting half or more of it.
But back to the topic. The good thing was no GUI it should be a micro service, so no wasting time on moving the button just a little bit left, then right... But as any project when you get deeper into it and you have almost 2/3 of the code done, you tackle a problem. And mine was that the input xls (proprietary) format written by the library used in the API does the compression bad. The file can be opened with Excel but not with the app before you save it with excel. So you have to use different libraries to read the data, from the ones used to manipulate and save it.
The code was growing and so was the custom class I created for it. While I was writing it, I saw the places for code optimization. But there was just no time. It was an everyday multiple times per day event - push push, push, are we deploying?! But what about testing?! Finally the day of deployment came. I barely got a period of one time parallel creation, just to do a proper testing and a bug fix. Suddenly it was over. I never got a time to properly revise and optimize my code. It went into the production and was used by my colleagues.
When I look at it a get scared, it is my personal monster code. It's not really that bad, bud it needs tightening up. The thing bothering me and making me proud at the same time is the performance. Since than I wrote a lot of code. I got better at the topic. I got better in Python. I know more libraries, etc. But this peace of code works unbelievable fast, it reduced 2-3 weeks of work literary to five minutes. And I still wonder how this organised chaos (mess code) works so well. Maybe it is my enthusiasm build into it that pushes it forward. Who knows?
Comments
Post a Comment