Porting Aware To Django
Published Nov. 11, 2017
A few years back I started a PC profiling service known as AwareLabs. It was originally written in C++ with a Windows MFC downloadable client. Six months ago I re-wrote the static website front end in Django so I could get a better handle on news and static updates. Today I’m re-writing the entire back end data management piece and customer manager.
To be clear thats roughly 25,657 lines of C++ code, a dozen SQL tables, and all of it has to remain backwards compatible with the existing Windows client (MFC code not a part of this re-write). I will cheat a bit and just wholesale steal entire chunks of code from my other projects. Also I’m shaving off a feature or two because it has become clear that no one cares about it.
A quick history and product description. Aware Labs is my company, Aware the service profiles Windows PC using a downloadable client and displays neat graphical and tabular reports of drive space, memory usage, application usage, and a whole lot more. Its a very cool app, completely maintains privacy and gives you a very comprehensive snapshot of how all your computers are used. So in other words its a huge complex site, not just Bobby’s home page.
Its a 12 hour dash to complete in one day what once took weeks of development. Here is that day described live with every up and down…
Just woke up, showered, and ate breakfast. Very optimistic at this point, started writing this post.
An outline of the tasks is now complete, I’ve created a subversion repository, and ported the existing code to my local machine. Using the already created Django project from my previous re-write. I’m a bit worried I’ve bitten off more than I can chew.
MySQL is set up and I’ve created the models file for Django. This required reverse engineering the Django model file from the original SQL. Since I’m replacing the original user tables with Django’s there was a bit of fiddling around with foreign keys. My best friends are now "db_table" and "db_column" for renaming columns and tables to match the old schema for backwards compatibility with the Windows client. I’m expecting no change to that code.
I’m sitting here with two laptops doing a column by column comparison of the schemas. Fixing minor things like column lengths and unique parameters. Django ads some primary keys to tables by default so I hope they are not an issue. The client relies on "auto_increment" for primary keys so those are critical. I need the Django "syncdb" command to generate the tables right the first time, manually adjusting them will be a pain.
I just ripped the login, signup, password recovery, and contact page from my Arkayne site and my EVENTALLY site. Took 5 min to copy code, integrate with new templates, and create URL handlers. No time for testing must move forward.
Ran into Captcha issue with signup and contact, again copied code from Arkayne. Templates and style sheets meshed nicely with new code so I’m less stressed for time.
Twenty minutes of time spent coordinating another project over Skype with startup partners. Hopefully time well spent, customers are ready to go we just need signatures on dotted lines… back to my saturday distraction.
Missing template for login page after signup. Slight adjustment to CSS style sheets. Also had to change template blocks from quoted to plain. Login a success!
Notes on scratch pad:
add login forms to top of page
create tabular displays for each table
spoof up some fake data
client download page
menus for each report
Arkayne linking server stopped. Logged into server, debugged problem with filters and restarted processes. Wonder why Wired is submitting links? Quick check showed no widget on their pages, must be an API, will investigate later.
My brother Arthur called, needed directions to cafe for short notice meeting. Discussed our plans to test and finalize content for the work I’m doing today.
Downloaded client information page from old site. Adjusted HTML and content. Able to display client list for currently logged in user and download page is in place!
Friend called for lunch, will be over shortly for a Costco run, monthly BBQ get together with friends. Want to get at least one report generated before then… maybe extensions.
Back from Costco, nice break from coding, not sure if it qualifies as getting out but at least I got to interact with other people.
Ok all tabular reports for: drives, extensions, applications, activity, and logins are filled in. Plus I had a chance to add the logged in menus at the to of each page. There will be plenty of clean up later but for now on to some test data…
The all important commit just wrapped up, code is safe. Feeling good right about now. I’m crazy for not testing anything yet, still concerned how well the Windows client will play with the new interface but I need to push ahead to get the Google Charts in place.
The Google Charts API looks very simple, does not even require an API token. They limit hits to 250,000 a day, which in my case could be 50 charts per page, with 50 clicks per user per day, gives me about a 1,000 user capacity. So the plan is to cut back to only a few eye candy charts for logins and activity. The rest will be tabular or cached, does Google allow that?
I just dumped live data from the master DB, am going to FTP it down and see if it plays nice with the Django generated schema…
Found out that "unique_for_date" does not work on my version of Django. Decided to use "unique_together" instead. Sigh…
Turns out the extra columns are going to be a pain when porting the data. MySQL inserts need the column counts to match, see if I can export using update instead, or maybe just add the missing columns.
Re downloaded fresh dump of the live database with the "mysqldump -c -t" options. That creates a column list and will hopefully clear up the column mismatch issue. Now to prune the extra tables and try another import.
The import is a success. Surprisingly only forgot one column, named one incorrectly (need to fix my fix now that I think about it), and three or four issues with uniqueness. Now I need to massage the new built in Django account system to fit the data and everything will work. Right!
Ok patched the fix above in the models not the MySQL dump so the Windows client doesn’t barf. Also I just dropped, created, and imported the database 10 times in 3 minutes after fiddling with the schema, never underestimate the power of database abstraction. Props to Django.
Solar City just called, my install is ready to go Monday morning. Only hang up is my HOA has not approved the plans yet. We’ll see if we can get it all taken care of Monday. Although considering my HOA’s past I’m guessing not so much.
So this is incredible, I dumped the data in and created a user id, logged in and all except one of the tabular views I created earlier work. Minor CSS formatting required. The user I created happened to line up with a dangling foreign key in the imported data. Great that proves integration works!
Strange I’m getting a "year out of range" error from Django on one of the date columns. Shouldn’t MySQL be throwing an error too? Why was the record even stored on import?
Looks like some of the dates imported as "0000-00-00 00:00:00". I deleted the 6 offensive records, everything now works, and I will keep that in the back of my mind for future woes.
Well the formatting looks good except I have to add more columns to fill in the page. Then fix up the activity report to display data in human readable format versus the raw database strings for activity. As far as charts I’m thinking:
Drives – none, a Google meter would be nice but would waste hits
Extensions – pie, groupings of extensions by category, text, images etc
Systems – none, again a Google meter for memory and cpu would be good
Applications – none, this is just tabular data
Activity – something that represents usage well, bar graph, sparkle chart, ???
Logins – pie, groupings of logins by user
Come to think of it would be great to do breakdown by day, week, hour, month, quarter, and even year for the activity data. Should be all point and click, is this in scope for today?
Referenced an old post to figure out how to add multiple columns using "modulo". See: Another IfModulo Template Tag Or If Else End Tutorial. Turns out I can use filters in an if statement with Django’s templates. Who knew the site does not make it clear although it is implied I guess.
Eye candy is in! The Google API for charts and graphs is stellar. In a short hour I was able to read the documentation, copy and paste an example, and then mutate it into the graph I want. I found myself playing around with minor features for the last 10 minutes, the 80/20 rule strikes again so I quit while I was ahead. Still the results are very nice, pie charts for drives and users and bar charts for activity. Tired need food…
OK had dinner of left over tacos. You’re thinking junk food but it was fresh diced peppers, onions, carrots, and ground beef with taco seasoning. Threw in some low fat cheese and salsa on top. Delicious and it only took 10 minutes to make. Russ Klettke’s "A Guy’s Gotta Eat" has paid for itself ten times over. Should be called “A Developer’s Gotta Eat”, awesome book for how guys with a time crunch can eat healthy. Back to work.
I’ve decided not to salvage any of the existing live DB. There aren’t that many active accounts to be worth salvaging from the previous version. So now the mad dash. Im getting tired so well wrap this up with brief updates…
Create SVN and check in code.
Drop database and create new via Django.
Test site account creation and data upload.
Have Arthur (my business partner) test site.
Call Boys and Girls Club Of Arizona to set up demo.
SVN created and updated, had a few minor issues with moving the old site out of the way. Apache was a snap, pretty standard setup to get a Django app runing using mod python. Really I just copied an existing record and changed it.
The signup and login system works on the first try. No not a miracle, Django does an awesome job of isolating concerns, remember we copied the entire system from a previous project.
The client upload piece was not working because it was pointing at wrong IP. I wont have time to tackle that today, sigh. I was hoping this would work out of the box but the C++ code on the server that handles the client connection is not cooperating. Something about missing libraries for MySQL. I dug around net but no one has a simple solution. For years now the C++ MySQL libraries have been crap [insert primal scream here]. They will have to be replaced. I’m tired and running out of steam.
Even though I feel a bit down about the client I think the overall day was a tremendous success. I’ve re-created the entire database and front end GUI of a 25,657 line C++ program in 4,189 lines of Django on top of python. So after a 6X reduction in code I think it will be much easier to integrate and release new features going forward. On top of that this exercise only took an 13 hours which less than the total time I’ve put into trying to get C++ to play nice with MySQL. Granted things run a bit slower on some of the more intense activity reports, but its well worth it.