Search
Blogging from Sydney | Australia
Some past thoughts
Friday
Dec032010

Carbonite - Workarounds to back up executables, videos and other files

Carbonite is the online backup service I've ultimately decided to use to protect my data off-site against those proverbial theft/fire/flood events. Unfortunately a Windows Home Server box, as awesome as it is, just isn't off-site (so it gets destroyed or stolen with everything else).

For me, Carbonite does a great job out of the box - documents, photos, music, program/web application source files - ASCX/ASPX/CS/CONFIG/C/CC/CPP/H/HTML/CSS/JS/PHP/Python (ironically including the .pyc files which are actually almost useless to me) /Ruby/Scala/ASM/eqn/JED, program data files - SQL/YAML/XML/XSD/in/out, text files, PDFs, Mercurial repositories (.d and .i for example), compressed and encrypted files - Zip/7z/TrueCrypt/Axcrypt and the vast majority of the other important stuff.

Of course, it's detailed by Carbonite that they do not automatically back up executable files, video files, or many other kinds of files.

In effect, you manually have to go into the relevant folder, right click and select backup (thankfully Ctrl+A to select all and then backing up does work for all files, just not folders).

Why it's a problem
The core is simply - your complete backup is no longer automatic. That means human error begins to creep in.

I'm a computer science student. I have dozens of different projects and assignments completed over the last 4 years, and I know if I come back to them in 5 years, it will be a lot easier to have their executable forms lying around so I can remember the application context more easily.

These are primarily .exe, .o, .jar, .dev and other miscellaneous file formats. Now I will be clear - Carbonite backs up the source forms as detailed above. I'm just saying I want the whole package because some things like compilers and IDEs can become misplaced, hard to find, etc over time. Stuff disappears from the internet all the time, for example if you missed it one of the major original content portals on the internet - Geocities - closed recently.

Workarounds
So, I said Carbonite does by default back up all zip files. This suggests a relatively simple workaround, just put each important file inside a .zip file. Easier said than done?

We'll need some programmatic way of manipulating these files (otherwise it's literally back to the Windows GUI). 7-zip provides such a useful method.

http://dotnetperls.com/7-zip-examples

7za.exe is the command-line version, this one worked for me (though YMMV)
http://downloads.sourceforge.net/sevenzip/7za452.zip?use_mirror=puzzle

Like many programmers, I see myself as pragmatic (e.g. Rasmus Lerdorf, father of PHP - http://itc.conversationsnetwork.org/shows/detail3298.html so I'm going to just do something simple and easy for me.

My backup strategy for these files is basically to call something like the following on the Windows command line, which will create one big .zip file:

C:\Users\Peter>7za.exe a -r -tzip myfiles.zip *.exe

Since this is fundamentally a 3rd tier backup for me, I'm satisfied even though it will take me a little longer to recover the data, and it's possible I've missed something (or Carbonite changes their program's rules, but I reckon they aren't looking to start a war because they'll only lose customers).

Now again being lazy (though I should say this is in the spirit of automation, removing human error), I'd rather not type that into the command line every time (and if I forget something?), so let's turn this into a file called backup_via_carbonite.bat :

GOTO EndComment
This BAT-file zips up executable files,
web site favicons, development files, DLLS,
installers, compilers and other miscellaneous
files so they are backed up by Carbonite.

Please run it through YOUR OWN TESTING if
you plan to use it as part of your backup
strategy!

Notes: - * is a wildcard meaning match all
- "a" means create archive
- the -r recurses through the entire
folder structure
- myfiles.zip is the name of the
resulting .zip archive
- the -x!Downloads\* excludes files in
the Downloads folder, same for AppData

Written by Peter Schmidt
03/DEC/2010
:EndComment
7za.exe a -r -tzip myfiles.zip *.a *.bak *.cab *.com *.dev *.dll *.exe *.ico *.ini *.jar *.lib *.msi *.o *.win -x!Downloads\* -x!AppData\*

Now I can just double click backup_via_carbonite.bat from Windows, put it in my Startup directory, or in the Event Scheduler so it happens as close to automatically as I'd like it to.

My understanding of .zip is it is not a solid compression format and so should be more resilient to small amounts of data corruption, i.e. one flipped bit will not corrupt on average half the files, just one of them.

Now I did say video files. That's another set of extensions - easy to add but hard to discover. Here's a start:

GOTO EndComment
This bat-file zips up some video files
so they are backed up by Carbonite

Written by Peter Schmidt
03/DEC/2010
:EndComment
7za.exe a -r -tzip myvideos.zip *.flv *.mpeg *.mpg *.mp4 *.m4v *.qt *.wmv

Of course anyone with a sizable video library will know this won't scale, will consume a ridiculous amount of space and well...just be unwieldy and bad. It's possible on many connections that this file would never even be completely uploaded (or may change too frequently) - resulting in no backup at all!

I don't have the solution, except to say Windows 7 does provide the lovely feature of libraries. Try compositing all your videos into one library so you can use the Ctrl+A backup above. If you store your videos in separate folders or across multiple drives - you'll need to add each drive/folder to the library unless you've got a more creative workaround - good luck thinking =)

Final note: Ironically because Carbonite stores older versions of files (including the .zip file that will now be being regenerated automatically on schedule or when I restart Windows), this solution will end up costing them significantly more storage space, bandwidth and time than if they just gave me, the informed paying customer, the option to back up what I wanted conveniently.

Eventually I might get around to splitting this up into separate archives for separate folders, but I'm probably too pragmatic with too much other stuff to get on with...330MB is not too bad a zip file to upload dozens of times over.

Saturday
Oct092010

Electronic Voting's Issues

Just another part of my extended memory, someone said we should be going to all electronic voting machines in Australia.

For one very good reason, I believe the majority of electronic voting machines are fundamentally flawed - it is hard to give a reasonable guarantee that:

electronic records are written once and from that point read-only.

This is not an impossible problem, for example, modern database management systems can enforce such permissions (assuming there are no bugs affecting this functionality in the DBMS). Ideally this kind of functionality should be implemented in hardware, but that is generally more expensive.

In short, making such systems secure and reliable is a fundamentally hard problem.

Perhaps the best example of a voting machine done (almost) right, the Sequoia AVC Advantage, is well explained on Security Now! Episode 211 - Hacking Electronic Voting Machines:
Main Page - http://twit.tv/sn211
Transcripts - http://www.grc.com/securitynow.htm#211

The best documentary I've found explaining the problems with Diebold and many other machines is the HBO Hacking Democracy Special:
Google Video - http://video.google.com/videoplay?docid=7926958774822130737#
Main Site - http://www.hackingdemocracy.com/

The Princeton University findings are also good:
http://www.youtube.com/watch?v=aZws98jw67g

And the reason I remembered to write this post, a great post linked by the Electronic Frontier Foundation on Twitter:
http://www.freedom-to-tinker.com/blog/jhalderm/hacking-dc-internet-voting-pilot

Fundamentally in electronic mediums, things like trust are still being developed and are often at best fragile and relatively easily broken and overridden. That's the reason the heart of democracy still in my opinion requires a paper trail, literally a reliable, stable audit trail of every single vote - because paper is still a known medium that is much easier to protect and much harder to tamper with.

Saturday
Sep252010

jQuery - Freedom?!

Yes it should have been discovered sooner. But now that it has been...jQuery+jQueryUI is sheer brilliance.

I can immediately see this freeing me from the demands of stakeholders - I can just give them the ThemeRoller tool (since like most developers, I tend to have a terrible sense of UI design) and tell them to choose how they want the site to look and get them to send me the URL or files back. (Of course the hard part has always been getting the site to work and keeping it alive and secured from DDoS/XSS/CSRF and other attacks, but now I don't have to worry about things like the style of datepickers!)

So why is it so good? Simply put - an excellent pluggable framework for Javascript that unlocks the power of JS, gives the developer great flexibility, and provides better cross-browser support than I probably ever could!

An example related to my current work, building an ASP.NET MVC2 site on .NET 3.5 (c'mon people, .NET 4.0 has been out for a while now).

All I need to do is include in the Site.Master file something like:

<link href="<%= Url.Content("~/Content/jquery-ui-1.8.5.custom.css")%>" rel="stylesheet" type="text/css" />

<script type="text/javascript" src="<%= Url.Content("~/Scripts/jquery-1.4.2.min.js") %>"></script>

<script type="text/javascript" src="<%= Url.Content("~/Scripts/jquery-ui-1.8.5.custom.min.js")%>"></script>

obviously the actual URLs may be different depending on your project setup, the above come out to something like ../../Scripts/jquery-1.4.2.min.js, but that depends on the context of the request.

Whenever the stakeholders decide to redesign part of the site - it's update 3 files/references and voila! Whenever performance and security fixes come down the pipeline, same quick update! And in web development, it's all about taking DRY to the limit =)

So for an actual example, let's try making a datepicker:

  1. Add the javascript for the datepicker itself, for a basic Aussie date format:

    <script type="text/javascript">
      $(document).ready(function () {
          $(".datepicker").datepicker({
              dateFormat: 'dd/mm/yy',
        });
    });
    </script>

  2. Add the "datepicker" class to the relevant div or input type="text" area

    <input type="text" id="releaseDate" class="datepicker"/>


Of course for full details of what's possible, see the jQuery DatePicker page. http://docs.jquery.com/UI/Datepicker

But remember I said ASP.NET MVC2 ?
That's something like (simplified):

<div class="editor-field">
       <%: Html.TextBoxFor(model => model.Title) %>
</div>

i.e. using auto-generated view code (literally right-click on a controller ActionResult and click "Add View") which uses lambda expressions to reference model attributes, or something like that.

The key point is MVC2 abstracts the HTML using the built-in (and extensible) HTML helper classes. Seems like it's time to write one of those to take advantage of jQuery!

Update 5:29PM - An excellent post by Steve Michelotti covers this very well:
http://geekswithblogs.net/michelotti/archive/2010/02/05/mvc-2-editor-template-with-datetime.aspx

Tuesday
Aug172010

Google Wave killed to focus on the Aussie NBN?

Opinion

It is well known that Google recently killed their implementation of Wave. It is also well known that Australia’s current Labor government has currently promised to build a $43 billion Fibre-To-The-Home National Broadband Network (NBN). The first nodes are now operational in Tasmania, with (hopefully) much more to come.

It has recently been announced that the NBN will improve its coverage to 93% of the population, and deliver even higher speeds of up to 1000 Mbps (up from 100 Mbps). That’s the same speed promised by Google’s experimental network in the USA for 50,000 to 500,000 people, except it’s potentially on the scale of up to 93% of 22.4 million people. I don't think anyone at Google Australia missed this...the right kind of sufficiently pure fibre should be capable of 50Gbps or maybe even much more.

Even if only a small portion of Australians actually take up the offer it could easily be on par with the US experiment. If Google, with some of the cleverest engineers on the planet figures out killer apps for Aussie blokes and sheilas and their real families...there's an easy roadmap for taking the experiment to a level that's an order of magnitude larger.

However there’s a huge risk – an upcoming election on August 21st, 2010. If the Coalition wins the election, it has promised to kill the NBN, replacing it with a $6.3 billion patchwork of technologies. What does this mean for corporations like Google? Uncertainty - perhaps the single most destructive force for business. Why? Because as a business you can't easily justify the longer term investments that are often the most productive and profitable. For Google that might mean committing engineers to bandwidth-intensive next generation consumer and enterprise apps instead of whatever more mundane tasks need doing (like babysitting Wave?).

What would a Labor victory deliver? Governments tend to move slowly, by the time of the next election enough of the network will actually have been built (since we're already about 1 year into the estimated 8 year rollout period) that it will be effectively completed, almost no matter what happens in the election after.

So was the death of Google Wave a coincidence or a masterful strategic move?

Killing Wave frees up dozens, perhaps hundreds of Google employees in Sydney who were working on Wave, likely as their 80% project, to think about and work on architecting, designing and constructing the next generation of applications for the post-NBN world. And that means these Googlers and anyone else in the technology industry may have a huge experimental playground for building things like GoogleTV, videoconferencing, enterprise and cloud services, maybe even healthcare and educational applications or entirely novel things we can barely imagine.

Of course, it depends on how Australians vote on August 21st. Who knows what some of the best engineers and PhD’s could come up with on the next generation of network?

Smart people + passion + bandwidth = ?

Personally I’m excited to think of what might be in the pipeline. Whether this was a happy accident, a random coincidence or a master stroke of genius, I say bring on the innovation!

Saturday
Aug142010

Why the Coalition's $6.3bn broadband plan is a waste of money

According to The Australian, under a Coalition broadband policy, we'll be getting:

  1. "$6.31 billion over seven years"
  2. "$2.75bn to the construction of an optic fibre backhaul network but also relies on at least another $750 million from the private sector"
  3. "$1bn in grant funding for a rural and regional wireless network"
  4. "$1bn to build a wireless network in metropolitan Australia"
  5. "Satellite coverage for the remaining 3 per cent of the population will receive a $700m boost"

Why is this a waste?

Firstly the Coalition could have had another $6.3bn to spend on other things like roads, rail or healthcare.

Secondly they're building a "optic fibre backhaul network" - that sounds pretty similar to what MFS-WorldCom, New Edge Networks and others in the United States spent over $90 billion on in their technology boom, depending on who you believe much of that is still wasted "dark fibre".

Thirdly while I'll agree a wireless network makes sense in rural areas with very low populations, it doesn't whenever you get more than a few simultaneous users. On a properly built fiber network, you can get the full speed - Labor and the NBNCo are promising 100 Mbps minimum if you are willing to pay for it. That is a guarantee.

On a wireless network (and the Telstra/Optus HFC "Cable" networks) you have what is called contention - when other users use the same network the performance you get degrades. It's not uncommon for many users to degrade the performance of WiFi 802.11b networks from 11Mbps to 1Mbps or even lower making the network unusable (try this in a university library for example) even with the best algorithms to share the bandwidth.
And consider for a second - everyone on a wireless network must obey the rules - anyone with a misbehaving piece of equipment (or even someone who was deliberately being malicious) could interrupt service - not for one user but for dozens or hundreds - at once.

That makes applications like healthcare monitors for elderly citizens impossible - it's simply too risky. It also makes latency sensitive applications, like videoconferencing or online gaming, much less engaging (if you've had dropped telephone calls in the past - how often have you dialled the person back and how did you feel?)

Fourthly, even if you were *extremely* optimistic about the Coalition's wireless technology, it might be capable of a peak theoretical 1Gbps with the commercialisation of the still in progress 802.16m WiMAX. Now split that between 100 users per deployed node (10Mbps?). Many metropolitan and regional users today have ADSL2+ DSLAMs featuring up to 24Mbps. However you cut it, the proposed wireless technology either doesn't exist today or is inferior to existing wired technology, and vastly inferior to a fiber network.

Finally satellite coverage is receiving a $700 million boost. The NBNCo is planning to deploy 2 satellites for rural Australia at $500 million each, providing better bandwidth and redundancy in the event of one failing. There aren't that many makers of satellites, it sounds like the Coalition might end up wasting $200 million right off the bat and get just a single satellite.

Now I should be clear, I believe in the private sector moving forward with wireless deployment, it has great applications, especially in the mobile phone/laptop/netbook/tablet space. But wireless spectrum is still relatively scarce and relatively limited - and for the forseeable future it will remain so. There's are many more good reasons the IT industry has wholeheartedly slammed this Coalition broadband plan.