Search
Blogging from Sydney | Australia
Some past thoughts
Thursday
Feb042016

Python 3 in Production at Mathspace

And Python 3 only in development.

A huge thank you to everyone on the Mathspace Team who supported, reviewed and helped out with this endeavour.

One day the Mathspace Team will put up a dedicated engineering blog, but until then this can live here.

For those who haven't heard me talk about bringing step-by-step working out, handwriting recognition, teachers never needing to mark homework again, adaptive mastery-based learning, geometry, interactive graphing, calculus, The Martian content (currently under development) and so on to High School and College education:

https://mathspace.co

Love it? Want to help make it happen faster for parents, students and teachers? We're hiring.

 

Mathspace Overview

8 or so developers, similar number of content team members, lots of sales and support staff, moving to a bigger office soon.

 

86 Python Dependencies 

pip freeze | wc -l

 

147 Django models across two databases

len(get_concrete_models(models.Model))

 

~160k Python LOC

find . -name "*.py" | xargs wc -l

 

Driving change

Above all do it incrementally, following lean startup ideas such as reduce batch size, thanks Don Reinertsen, and thanks for coming to YOW! Sydney.

Based on:

for i in $(git log --grep="py3k" --format=format:"%H" --all); 
do git diff $i $i~1 --stat | grep "changed"; 
done;

Some quick git stats:

  • 42+ specific merges to master between 30 August 2014 and 11 January 2016. It just wasn't quite ready for Christmas/New Year 2014 so as things go we built out more math features.
  • 1629 files changed
  • 18027 insertions(+)
  • 16885 deletions(-)
  • So around 11% of our code base changed, including the final 2to3 conversion, but excluding final few bug fixes for legacy code without unit tests.

Most of the development battle felt like it was dependencies because frankly Github is awesome but there's always higher latency than tapping a colleague on the shoulder or @colleague in Slack or Trello.

Upgraded dependencies

Switched out dependencies

  • Fabric => Fabric3 (at least until @bitprophet decides to give us Fabric2's alpha)
  • python-cloudfiles => apache-libcloud
  • suds (SOAP/WSDL) => Stripe billing gateway (might also use PySimpleSoap which claims Python 3 support, but it failed our unit tests before the Stripe switch and its master was broken with a print statement for months which is unfortunately not very confidence inspiring)

Removed dependencies

One pleasant point was upgrading Django because we have pretty good unit test coverage:

# -Wonce works too, just
# -Wall is funner to read out loud
python -Wall manage.py test

Tells you what's RemovedInDjango18, RemovedInDjango19, RemovedInDjango110, which you'll need to fix sooner or later.

 

Making the switch

Basically the things that broke were things with no unit tests. We fixed all of them over a weekend with a few quick deploys, thanks to David Cramer and the Sentry team.

How did we build the final branch?

0. Move anything that works under Python 2+3 off into a separate branch, reviewed and merged regularly.

1. pyenv (or homebrew) to install Python 3

brew update
# Mavericks and El Capitan come with versions of SQLite3 that
# segfault when running "./manage.py test", so update it
brew install sqlite3
brew link --force sqlite3
brew install pyenv
# Tell pyenv to use updated SQLite3
# https://github.com/yyuu/pyenv/issues/333
# Takes around 2 minutes
time PYTHON_CONFIGURE_OPTS="LD_RUN_PATH=/usr/local/opt/sqlite/lib LDFLAGS=-L/usr/local/opt/sqlite/lib CPPFLAGS=-I/usr/local/include" pyenv install 3.4.4
# Test it
$ python
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.10.1'
# Remember to use it when making virtualenvs:
mkvirtualenv mathspace_py3 --python=$HOME/.pyenv/versions/3.4.4/bin/python

2. Look for outdated dependencies, update them, split off upgrades of dependencies with 2+3 versions when possible.

3. Update / migrate our code base internally. As an end project, 2to3 was a better choice than six.

  1. 2to3 . --output-dir=. -n -w
  2. Fix imports 2to3 did not get right so unit tests run. DiscoverRoadRunner was invaluable saving me 4 minutes per test-driven-development cycle, I know I should spend more time on it though Django 1.9's builtin test runner does "--parallel" 🍻
  3. Fix other things 2to3 tool did not get right, e.g. Django's smart_unicode => smart_text
  4. Update or fix unit tests under Python 3 accordingly, things like tests that failed with different PYTHONHASHSEED values that were bugs under Python 2 but only detected under Python 3.
  5. Check things like the following don't throw errors, add unit tests if they did:
    ./manage.py runserver
    ./manage.py celery worker

4. Add new Python 3 servers - thanks Rackspace devops for getting poise-python working so supervisor can stay under Python 2.

5. Test on staging environments for any issues.

6. Deploy to production.

7. Fixed a few things fast. Thanks to Sentry we could pounce quickly. Maybe 20-30 users affected by Sentry events total, so 99.94% or more of our >50k users in that particular low-traffic week would never have noticed, anecdotally better than our last Django upgrade.

Was it overengineered? Perhaps, but to channel Django, Mathspacers can be perfectionists with deadlines, and we do think it is very important to provide the best experience we possibly can for parents, students and teachers.

 

Profit

  • Better unit testing
    with self.subTest()
  • Cleaner superclass calls:
    super()
    
    # Instead of monstrosities like
    super(MyClassName, self)
    super(MyClassName, cls)
    
  • Cleaner types and unicode support:
    long->int, str => bytes, unicode => str
  • Can't easily go the wrong way when using:
    .decode() or .encode()
  • No re.UNICODE flag required in regular expressions.
  • Don't need to remember
    # -*- coding: utf-8 -*-
    from __future__ import unicode_literals
  • No cryptic backtick syntax:
    repr(a) == `a`
  • Don't lose original error if another error triggered in an except block
  • Hundreds of other bugs fixes and feature improvements: https://docs.python.org/3/whatsnew/changelog.html

Nick Coghlan can tell the full story far better than I:

http://python-notes.curiousefficiency.org/en/latest/python3/

Thursday
May152014

Why I used my house deposit to pay off my HECS-HELP loan

“Compound interest is the eighth wonder of the world. He who understands it, earns it ... he who doesn't ... pays it.” - Albert Einstein

So I tweeted this, and followed through today with the largest B-Pay I've ever made. Should be confirmed debt-free by the end of the week, at the cost of my house deposit.

 

The short version

A little more supporting reasoning

  • $40,000 is a relative drop in the bucket on a >$720,000 Sydney median house price. (And I'd be hard-pressed to save $30,000 a year just to pay 5% interest-only on $600,000 - a doubling of rates to 10% would wipe me out entirely and 17% from the Hawke/Keating era... I can only dream of earning that on a savings account if I could hold a job in such likely recessionary times).
  • CBA Goalsaver today = 3.81% total interest requiring $200 put in every month, with rates probably continuing to fall while CBA makes more record profits (built on a bubble, not touching our big four's shares with a 10 foot pole except in my super fund so I can argue that itch is scratched).
  • I'd pay a marginal tax rate of at least 38.5% (inc Medicare Levy) for FY2013-14 (39% in 2014-15, thankfully dodged the $80,000 additional deficit levy but such a shame they didn't try reforming superannuation which would have hit me, CGT discount which would have tripped me into speculating more, or negative gearing which would have been the best thing I think they could have done, I stand by my previous thoughts that it's the defining ridiculous misallocation of resources... paying people to lose money... utterly insane), leaving me with 2.34% effective interest on the CBA Goalsaver at most, about 0.5% under inflation.
  • Interest rates on all savings accounts I've looked at over the past year or so have continued to meet expectations and fall from well over 4% to the current levels.
  • Commonwealth Government decided to raise rates to the 10 year T-Bond rate, today funnily enough is about 3.81% (an instant rise of 0.9% relative to last year's indexation of 2.9%).
  • In sum, I would be paying between 1.5% to 3.5% for the optionality of holding my house deposit that is going to struggle to reach a reasonable 20% or $150,000 over the next 6 years anyway (with no price growth or rather a price crash), well up from 0.5% (though that may be long enough for some windfall rebalancing gains to come through if I get lucky with my other investments). But since it's never going to be enough anyway, I might as well seek the Australian Dream, er Farce in a country that has less red tape stopping developers building houses.
  • Retain some ethical / moral high ground, not one of these.
  • USA is looking more interesting all the time... (it's also nice when a highly-productive web/Python/Django/Machine Learning first class honours CS degree holder gets offers to be poached by certain technology companies) why should I support a broken system? (though they are at 90% debt to GDP so it looks like the boomers have raided everyone's cupboards...) 2015's budget will be very important to actually ending the age of entitlement, but I'm not holding my breath, just continuing to move bits and pieces offshore (with full reporting to the ATO of course).
Saturday
Apr052014

Speculation: Negative gearing is on the chopping block

I'll believe it when I see it, but if they have the guts to do the right thing and remove this artificial negative gearing distortion saving many billions of $$$ (of course it's a scam, why should I and the members of the millenials in effect personally be paying thousands of dollars in taxes or interest on ballooning government debt to prop up existing house prices), they'll get my vote at the next election even though they've completely stalled what little progress was being made on the NBN (Turnbull, we brought you in to get things moving, not stall it completely). Thanks MB and SBS for the speculation.

http://www.macrobusiness.com.au/2014/04/report-negative-gearing-is-on-the-chopping-block/

Of course, I can only personally say such a thing because if I choose to stay in Australia, which is looking like a very BIG IF over the next 5 years, this is a necessary (but sadly unlikely to be sufficient) condition to deflating house prices into a somewhat affordable range.

Take a reasonable long-term historical perspective, such as average loan size just 10 years ago, c.2003 was A$120,000. Then flip it and just consider today, for instance that Boston median house prices are around US$400,000 (vs more like A$700,000 in Sydney). Or that I could buy (today with no debt, like WOW that was an eye-opener!) a dogbox studio apartment in Boston for just US$65,000 (thanks Flipping Boston S2, Ep8 via Foxtel, the worldwide perspective alone has just paid for itself many times over) whereas the bottom in Sydney starts around A$200,000. This is a lot of ranting, but it's supposed to be theraputic to get the problems off one's chest.

It's worth it if some of our politicians do get the message that outside of the 24 hour news cycle, people still need to make ends meet, still need to figure out how to plan to have a family, and want a reasonable (I'd say fair, but it became unfair when the pension age didn't start rising 20 years ago) deal from the society they are a part of. Ross Gittins has some excellent articles on economic rent-seeking, or basically taking a larger piece of the eco-cake than you should reasonably be entitled to. People will stand some oppression (for instance I fully support any efforts to reduce income inequality though I believe in bottom up economics where the poor will generally spend the money they get, boosting the economy, and even though that will disproportionately hurt me as a relatively high income earner), but too much will break the proverbial camel's back. I'm also aware that the grass always appears greener on the other side which is why I'm OK with getting a somewhat unfair appearing deal. Mostly because I think that I'm a reasonable enough person that once I've found a way to pay for one of these houses or 3+ bedroom apartments, I'll be pretty close to having "enough", then at that point I don't care if the rest goes to charity or porkbarrelling or whatever else, it's all cherries on top =)

It would be nice if the government also reduced demand in other ways, e.g. by not boosting immigration to the moon (407,000 new people or 1.8% per capita, yes it's not Libya's 20% per capita but it's still a lot, though I believe asylum seekers should be classed as immigrants), but I'm a realist so I don't support reducing that intake. An argument could be made that we actually should take these people so we have a greater societal capacity in the longer term to handle the inevitable climate refugees without having huge societal shocks, since little real progress seems to be being made on climate change.

The solutions remain as they have always been - primarily about supply, e.g.

  • Cutting red tape stopping developers getting out there and building houses and apartments (some positive signs on apartments at least).
  • Considering a broad-based land tax replacing stamp duty, to reduce the effectiveness of land banking speculation.
  • Releasing more government land for purchase by private citizens, for instance the NSW goverment appears to be planning to release land for about 800,000 people over the next 17 years (8 regions identified, around 100k per region). If one third of the 400k p.a. immigrants come to Sydney and surrounding regions, that's easily 2.2 million people (at a constant immigration quota) or a shortfall in housing for up to 1.4 million people (yes there are regional centers outside Sydney, don't know how many people can move there without an NBN) which will need to be made up by building upwards or increasing dwelling sizes just as the baby boomers kids are leaving and probably needing dwellings of their own. <Insert generation rent joke>...

To our good politicians (from all sides, primarily state and federal), I sincerely hope the electorate rewards you if you make the tough choices and pop the bubble.

I guess I understand from the expected mining capex cliff why the Reserve Bank of Australia decided it was safer to go all in on the bubble by dropping rates over the past year or so. Hope they finally get rewarded with some serious inflation that makes us all poorer. In a perverse way, the higher they rise the further they fall... so if the serious (like 30% or so as happened in Ireland) crash comes somewhere between 2017-2024 as the baby boomers look to fund their retirement and get whacked by falling "investment property" prices, I would be quite pleased. But this shouldn't be about intergenerational warfare, it should be about building (pun intended) a sustainable society, not a speculative society.

Wednesday
Dec112013

Goodbye Holden: Let the dominoes fall so we can focus on innovating

From General Motors, as reported by MacroBusiness and many others, Holden is finally leaving in 2017. I'd love to rant more about subsidising an unprofitable industry to the tune of $1.9 billion dollars annually, but we've been given a gift. The gift of certainty.

Jonathan Pincus, a former senior advisor to the Productivity Commission reckons in the long run it will be a significant net benefit. I'd agree wholeheartedly - up to 250,000 employees are now armed with the knowledge their jobs are at best under threat (Toyota) or basically gone (Ford, Holden, Mitsubishi, etc). 

They are now truly freed to think for themselves and start looking for productive employment elsewhere (perhaps other countries if they're really committed to the auto industry), retraining, going back to uni, etc - they are armed with up to 4 years to decide when they want to make that transition - remember the average punter gets two weeks notice when being fired by their boss. Further these auto workers will likely get decent redundancy payments so that gives them an additional margin of safety in thinking about what their dream job is and how to make it happen.

Free. Free and empowered to start innovating, creating, experimenting, learning, coming up with new and better ways of doing things, building things; actually generating more value for all humankind.

We might just get a Liberal government that will be up for some serious structural economic reform after all (hopefully this is not just a one-off). Good work Joe Hockey and everyone else who participated in calling the bluff and getting GM's decision out there for all to know. Or they could with the removed debt ceiling start building a lot of new infrastructure our cities so badly need - that wouldn't be too bad of an outcome either. Shame it takes 10 years to build anything these days with all the red tape that needs cutting.

Oh and when combined with the mining capex cliff there's probably that little recession thing ... I've never experienced what it means as Australia has had 22 years of continuous growth. Of course it's all a dartboard and no one really knows the future. For instance you can get concerned by listening to the gold bears or the demographic headwinds as the boomers retire (e.g. Japan). Or you could try thinking for yourself if the rest of the world starts growing after their GFC blues, they'll probably demand more resources. Or they could demand more of our world-class education system, financial services, tourism and other successful industries. Dartboard. Except for the lucky auto workers who know they have four years to prepare for their now far more certain future. Thank you GM and Holden for being honest with the Australian people.

Tuesday
Sep242013

Awesome Six Dogs Comment & Sydney Housing = Insane, Crisis, Ponzi

Disclaimer: As always, you seek professional financial advice (not listen to my speculative rant) before doing something like taking out a 30+ year mortgage, investing in any Australian or international stock market, or any other thing that could lose (or if you're lucky make) you a lot of money. No one knows the future of things such as real estate prices, and anyone who suggests otherwise is almost certainly lying to you.

I'm just a moderately well-off punter looking at the numbers and the hockey-stick graph of real house prices and thinking there is no realistic way the average Sydneysider can make the numbers work without fudging them. Hence my thesis is what goes up must eventually come down. Why?

It is my hope that a not unreasonable 50% down part happens in the next 5-10 years as the baby boomers retire (needing to sell assets such as real estate to try to finance their retirement, there's support to my thesis if you believe people like Harry Dent on TDI #330 who claims to have been a bull for 20 years turned bear in the last 3 years due to the demographic cycle changes primarily from the baby boomers, applicable to the US and probably by reasonable extension to Australia). I can barely believe my parents only had a $130K mortgage on a double-block house on the Northern Beaches just 25 years ago, but such is life sometimes.

To balance that, it is my fear that governments do nothing about the six dogs (see copied comment below) and Ponzi fever infects the market driving prices even higher - 17th century tulip mania or house prices in Sweden where I think I read it's common to have a perpetual loan you just never pay off, proves it is certainly not impossible. It's easy to add other evidence like Chinese speculation driving up prices. Perhaps I should just give in and change city or even country, it would certainly be a lot easier. Scrap that, it would be hard to give up the fact Australia is definitely a very good place to live. e.g. our biggest long-term gun threat is from 3D printed guns unlike the regular high school and other massacres occurring in the US.

This is in the context of hopefully getting to the stage of starting a family and wanting a stable family home somewhere that isn't at risk of up to a 75% decline in price if governments really get their act together (i.e. chase off 4 or more dogs), or run out of magical stimulus bullets, 2.5% nominal RBA interest rates is terrible, but not that far from 0% from which they can't go any lower without money-printing like QE3+. Though with the kind of money we're talking about I could literally add another child or two to the proposed family, buy a half-decent new Ferrari (almost making a really cool status-symbol Tesla look cheap), probably make a small fortune in the stock market, see a huge chunk of the world on overseas trips, go into space, the list of wants is practically infinite ...

 

The article was good (though I feel MacroBusiness is pushing this housing = Ponzi theme so take it with what grains of salt you will), but this particular comment was just brilliant. Kryptonite for me (since I know while it might go a bit higher in the short term, I believe it's very unlikely to outperform (from such a high base) in the long term, and if a mortgage is typically 30 years then all I'd rationally care about is the long term. All said while the statistics are against you, if you really have done the hard yards researching and found a genuine bargain in real estate then good luck and more power to you!

 

http://www.macrobusiness.com.au/2013/09/the-history-of-australian-property-values-redux/#comment-281321

Stoking Demand and Choking Supply

Government has done something very bad to the supply and demand of starter homes which has led to outrageous prices of starter homes, and supported much higher prices of better homes. In short, government has stoked the demand and choked the supply of starter/marginal/extra homes.

Stoking Demand:

* Government brings in many immigrants

Choking Supply:

* Government rations permission to build extra housing on the fringe or extra units in the city, and new cities
* Government allows landbankers to monopolise the permission and dribble it onto the market to inflate prices
* Government adds taxes, charges and levies to extra housing
* Government requires onerous compliance with regulations
* Government creates delays in approving dwellings.
* Government neglects transport and other infrastructure which reduces the area in which well-located and well-serviced homes can be built

There is much debate on which of the six chokers (refusal, cartel, taxes, compliance, delays, neglect) is the biggest and baddest. Interestingly, if refusal is the big one, then lowering taxes will give a windfall to developers, whereas if refusal is a small one, then reducing taxes will cause a drop in prices. This debate is fascinating from an academic point of view, but rather pointless if the aim is to solve the housing crisis.

It is like watching a man being attacked by six dogs and debating which dog has the bigger bite. Far better to chase off ALL the dogs and save the man.