Search
Blogging from Sydney | Australia
Some past thoughts
Saturday
Mar052011

Google and social, go together?

So here's one experience I've just had - and an insight into why it's sometimes said Google doesn't get social.

Social is about conversation and interaction - here's something I can do for you, would you mind doing this for me?

So here's why - the first comment on this blog, 3 months ago. Discovered by me just because I decided to go back and see what I was writing on this humble weblog, see what I was thinking.

Now Blogger has the option to turn on comment notification email. But if I'm not Engadget, Gizmodo, Dvorak, i.e. someone or something that can afford a real hosting / blogging / content management solution, this is the kind of thing I'd like to know about!

Basically this "Comment Notification Email" box, buried at the bottom of Settings > Comments, should have had the email address I signed up with. It's not spam if its a real person behind the scenes (and the link that's not a link and contains the Session ID and a blog behind it is a clear example of that =). And it's easy enough to link back to these settings in such an email so if the user decides it is spam - they have the option to turn it off.

Google - time to listen to dear leader turned advisor and think a little less engineering and a little more social. We'll like you more for it, promise!

Saturday
Mar052011

First house move, memories and moving on to future opportunities

Moving house for the first time in 22 years takes a phenomenal amount of time.

Time to clean, to prepare, to pack, to decide what to take, to decide what to box and put in storage, to decide what is important and what is junk. Sorting, sharing, helping the rest of the family, cleaning, washing, bubble-wrapping, padding, taping, vacuuming, clearing out.
And then more on the other side unpacking...

Unwiring the sprawling Cat5/Cat6/TV/phone/power mess with its tentacles reaching into nearly every room of the sprawling multi-story six bedroom mansion.

Finding old school projects and taking photos because there's no way you can keep them. Clearing the areas where life, water, and wildlife (possums were adventurous - one once poked its head through a hole in my wall and stared at me, at least before I taped over it).

Yes 'twas a grand old house, but one which really is reaching the end of its life. The brick fireplace roof flew off in strong winds last year so it's just a column up to open sky. The roof is still mostly covered in aging wooden shingles where the possums haven't knocked them off, and leaks in many places. Mosses and fungi are starting to invade sections of walls. Lichen grows on the fireplace bricks, especially in the cracks over the old cement. Some of the glass has been broken and taped over as a quick fix to keep the elements more outside than in.

Still much is habitable and the house still feels solid, even when it's hailing and pouring with rain. The downstairs is generally far more pleasant, especially in the hot summers when hot air escapes leaving the downstairs areas cool. It has a charm, an air about it, though perhaps that's just 22 years of memories staring back at me.

In any case, on to the future, apartment living and continuing study for this year at least. Let the good times roll, being closer to a new university in a new suburb, being close to good buses and trains, meeting new people and just keeping up with the continuous learning, the wave often likened to a tsunami, or the tide itself rising up - the ever-growing ocean of  the sum of human data, information, knowledge and wisdom on the internet.

Sunday
Jan232011

Water prices should be trending downwards, not upwards

http://twit.tv/gtt18 - Green Tech Today 18: Oasys Water.
Aaron Mandell, CEO's aim is to get the price down to as low as $0.25 per cubic meter. He says current processes (where energy costs are often subsidised) range from $0.65-$1 per cubic meter for current desalination.

How? Using some clever thermodynamics and chemistry at scale.
Why? The Oasys process in practice uses 1/3 to 1/2 of the energy of currently in use methods (with theoretical potential for 1/10 of the energy), primarily because it can use normal osmosis rather than reverse osmosis.

http://hypertextbook.com/facts/2004/JosephSabatelle.shtml - New York Water Prices
A little math - 1kL = 1 cubic meter. So $0.000537 per liter = $0.537 per kL (cubic meter)

http://www.sydneywater.com.au/Publications/FactSheets/UsageCharges.pdf#Page=1 - Sydney Water Usage Charges. $2.012 per kL, a big piece due to the cost of the Kurnell Reverse Osmosis Desalination Plant.
http://en.wikipedia.org/wiki/Kurnell_Desalination_Plant

I wonder if in 5-10 years this process could be retrofitted into existing plants for big potential savings as well?
Rob McGinnis, CTO and co-founder says the "plants are modular building blocks" so they should be scalable down as well as up - perhaps for use by farmers near saline aquifers? These are in addition to the other applications - industrial waste reuse, and even a new form of hydroelectric power.

In any case I think this is pretty amazing that some clever salts and thermodynamics could become a huge piece of the solution to the world's water issues.

Explore more:
http://www.oasyswater.com/press.php

Friday
Dec032010

Carbonite - Workarounds to back up executables, videos and other files

Carbonite is the online backup service I've ultimately decided to use to protect my data off-site against those proverbial theft/fire/flood events. Unfortunately a Windows Home Server box, as awesome as it is, just isn't off-site (so it gets destroyed or stolen with everything else).

For me, Carbonite does a great job out of the box - documents, photos, music, program/web application source files - ASCX/ASPX/CS/CONFIG/C/CC/CPP/H/HTML/CSS/JS/PHP/Python (ironically including the .pyc files which are actually almost useless to me) /Ruby/Scala/ASM/eqn/JED, program data files - SQL/YAML/XML/XSD/in/out, text files, PDFs, Mercurial repositories (.d and .i for example), compressed and encrypted files - Zip/7z/TrueCrypt/Axcrypt and the vast majority of the other important stuff.

Of course, it's detailed by Carbonite that they do not automatically back up executable files, video files, or many other kinds of files.

In effect, you manually have to go into the relevant folder, right click and select backup (thankfully Ctrl+A to select all and then backing up does work for all files, just not folders).

Why it's a problem
The core is simply - your complete backup is no longer automatic. That means human error begins to creep in.

I'm a computer science student. I have dozens of different projects and assignments completed over the last 4 years, and I know if I come back to them in 5 years, it will be a lot easier to have their executable forms lying around so I can remember the application context more easily.

These are primarily .exe, .o, .jar, .dev and other miscellaneous file formats. Now I will be clear - Carbonite backs up the source forms as detailed above. I'm just saying I want the whole package because some things like compilers and IDEs can become misplaced, hard to find, etc over time. Stuff disappears from the internet all the time, for example if you missed it one of the major original content portals on the internet - Geocities - closed recently.

Workarounds
So, I said Carbonite does by default back up all zip files. This suggests a relatively simple workaround, just put each important file inside a .zip file. Easier said than done?

We'll need some programmatic way of manipulating these files (otherwise it's literally back to the Windows GUI). 7-zip provides such a useful method.

http://dotnetperls.com/7-zip-examples

7za.exe is the command-line version, this one worked for me (though YMMV)
http://downloads.sourceforge.net/sevenzip/7za452.zip?use_mirror=puzzle

Like many programmers, I see myself as pragmatic (e.g. Rasmus Lerdorf, father of PHP - http://itc.conversationsnetwork.org/shows/detail3298.html so I'm going to just do something simple and easy for me.

My backup strategy for these files is basically to call something like the following on the Windows command line, which will create one big .zip file:

C:\Users\Peter>7za.exe a -r -tzip myfiles.zip *.exe

Since this is fundamentally a 3rd tier backup for me, I'm satisfied even though it will take me a little longer to recover the data, and it's possible I've missed something (or Carbonite changes their program's rules, but I reckon they aren't looking to start a war because they'll only lose customers).

Now again being lazy (though I should say this is in the spirit of automation, removing human error), I'd rather not type that into the command line every time (and if I forget something?), so let's turn this into a file called backup_via_carbonite.bat :

GOTO EndComment
This BAT-file zips up executable files,
web site favicons, development files, DLLS,
installers, compilers and other miscellaneous
files so they are backed up by Carbonite.

Please run it through YOUR OWN TESTING if
you plan to use it as part of your backup
strategy!

Notes: - * is a wildcard meaning match all
- "a" means create archive
- the -r recurses through the entire
folder structure
- myfiles.zip is the name of the
resulting .zip archive
- the -x!Downloads\* excludes files in
the Downloads folder, same for AppData

Written by Peter Schmidt
03/DEC/2010
:EndComment
7za.exe a -r -tzip myfiles.zip *.a *.bak *.cab *.com *.dev *.dll *.exe *.ico *.ini *.jar *.lib *.msi *.o *.win -x!Downloads\* -x!AppData\*

Now I can just double click backup_via_carbonite.bat from Windows, put it in my Startup directory, or in the Event Scheduler so it happens as close to automatically as I'd like it to.

My understanding of .zip is it is not a solid compression format and so should be more resilient to small amounts of data corruption, i.e. one flipped bit will not corrupt on average half the files, just one of them.

Now I did say video files. That's another set of extensions - easy to add but hard to discover. Here's a start:

GOTO EndComment
This bat-file zips up some video files
so they are backed up by Carbonite

Written by Peter Schmidt
03/DEC/2010
:EndComment
7za.exe a -r -tzip myvideos.zip *.flv *.mpeg *.mpg *.mp4 *.m4v *.qt *.wmv

Of course anyone with a sizable video library will know this won't scale, will consume a ridiculous amount of space and well...just be unwieldy and bad. It's possible on many connections that this file would never even be completely uploaded (or may change too frequently) - resulting in no backup at all!

I don't have the solution, except to say Windows 7 does provide the lovely feature of libraries. Try compositing all your videos into one library so you can use the Ctrl+A backup above. If you store your videos in separate folders or across multiple drives - you'll need to add each drive/folder to the library unless you've got a more creative workaround - good luck thinking =)

Final note: Ironically because Carbonite stores older versions of files (including the .zip file that will now be being regenerated automatically on schedule or when I restart Windows), this solution will end up costing them significantly more storage space, bandwidth and time than if they just gave me, the informed paying customer, the option to back up what I wanted conveniently.

Eventually I might get around to splitting this up into separate archives for separate folders, but I'm probably too pragmatic with too much other stuff to get on with...330MB is not too bad a zip file to upload dozens of times over.

Saturday
Oct092010

Electronic Voting's Issues

Just another part of my extended memory, someone said we should be going to all electronic voting machines in Australia.

For one very good reason, I believe the majority of electronic voting machines are fundamentally flawed - it is hard to give a reasonable guarantee that:

electronic records are written once and from that point read-only.

This is not an impossible problem, for example, modern database management systems can enforce such permissions (assuming there are no bugs affecting this functionality in the DBMS). Ideally this kind of functionality should be implemented in hardware, but that is generally more expensive.

In short, making such systems secure and reliable is a fundamentally hard problem.

Perhaps the best example of a voting machine done (almost) right, the Sequoia AVC Advantage, is well explained on Security Now! Episode 211 - Hacking Electronic Voting Machines:
Main Page - http://twit.tv/sn211
Transcripts - http://www.grc.com/securitynow.htm#211

The best documentary I've found explaining the problems with Diebold and many other machines is the HBO Hacking Democracy Special:
Google Video - http://video.google.com/videoplay?docid=7926958774822130737#
Main Site - http://www.hackingdemocracy.com/

The Princeton University findings are also good:
http://www.youtube.com/watch?v=aZws98jw67g

And the reason I remembered to write this post, a great post linked by the Electronic Frontier Foundation on Twitter:
http://www.freedom-to-tinker.com/blog/jhalderm/hacking-dc-internet-voting-pilot

Fundamentally in electronic mediums, things like trust are still being developed and are often at best fragile and relatively easily broken and overridden. That's the reason the heart of democracy still in my opinion requires a paper trail, literally a reliable, stable audit trail of every single vote - because paper is still a known medium that is much easier to protect and much harder to tamper with.