Daz's Brain Dump

The Lure of Fancy Phone Hardware

2012-01-12T19:17:00.001+11:00

My phone contract is about to expire. My HTC Desire is now 2 years old and not manufactured any more. It's running the latest Android (nearly), thanks to Cyanogen and it will get a taste of Ice Cream Sandwich (pending hardware drivers) in the month ahead.

So, thanks to the job I'm in, I moved myself to a Samsung Galaxy S II. All my apps were installed and my SD-Card with my music was installed. Straight away, I have twice the amount of everything. It had pre-production firmware, so was a bit flaky, but at the end of the day, I went back to the Desire in a day. It was smoother, and the display was really nice but nothing was notably faster. Benchmarks show a huge difference, but other than slickness, everything was about as responsive.

I also found it hard to hold. A bit thin and slippery, and the power button opposite the volume rocker annoyed me. Some of the external case solutions fix that though.

So, unless ics shapes up nicely, or Cyanogen looks like a good option, I'll be staying put for the time being.

KSM your KVM (or any VM)

2011-11-09T16:50:00.000+11:00

I run a linux desktop at work, with Fedora 16. My Corporate SOE environment is Windows XP (or Win 7 if you are lucky), which I run inside a kvm virtual machine. My laptop has 4GB RAM, so as you may guess, once my native desktop is busy, there's not much RAM left for Windows.

I've tried page compression (+1), and ballooning to dynamically adjust how much RAM windows consumes, which is effective, but not optimal. I'd heard of KSM (same page merging), but missed one key point, so never tried it. That point (for Windows guests) is:

Windows zero's it's unallocated memory on boot, so all that RAM can be coalesced down to one shared page.

This has allowed me to allocate the RAM required to the guest VM, knowing it's sharing nicely with the host. As it uses more RAM, paging starts (vi zram in this case), just like it would if I ram more apps native. It works very well. Right now, as a test, I have logged in but idle 2x XP instances, each with 1.5GB allocated, saving about 2GB (1GB each) in zero pages. Nice. When the AV stops it's first run on the VM, we'll be in action.

I think there are scenarios where that RAM can be allocated in a way that cannot be fullfilled (or swapped), in which case, I'm hosed, but it's sure better than wasting 1GB to have outlook and my payslip access (needs real ie) running.

I've also found kvm very stable. The virt-manager/spice/qxl frontent is a bit unstable, and sometimes dies, but the vm never does. Virtualbox tended to crash fairly often. When I use RDP to login, it is rock solid.

Why put 16GB of RAM in your PC

2011-09-28T09:57:00.000+10:00

I just built a new PC to use as a development box, so I can upskill myself in a few areas. As usual, I'm working on a budget, and here's what I ended up with:

AM3+ motherboard (basic, but all the peripherals I want, and 4 DIMM sockets)
Phenom X4 955
Entry level nvidia GT430 (well supported driver, not ideal for games, but a goot, quiet, cheap card)
2x 2TB 7200rpm drives in RAID1
16GB RAM

No SSD, no fancy CPU, but stacks of RAM. Why? All the benchmark sites show that my machine will be faster with a better CPU and an SSD, but that would have meant compromising in other areas. Here's what I see on a day-day basis with this box:

All 4 cpu's spend 95% of their time at 800MHz, and very rarely does more than one or 2 spend more than 10 seconds at 3200MHz. If they do, it's background processing. Stuff that I'm not waiting for, and I'm able to happily proceed on to other things while the CPU is busy. That CPU cost saving meant I could double my RAM. Total system power at idle is about 95W, 110W under normal load with up to 180 or 200 if busy (and overclocked). I also have a potential upgrade path if I want more CPU. AMD's bulldozer roadmap is AM3+ for at least 12 months, and although details are sketchy, it looks like a good opportunity to widen the number of CPU's up, but for me, I can't see any huge benefit. We'll wait and see there.

The pair of hard-drives perform very well. They don't boot as fast as an SSD, but I only boot once or twice a month. Read access is slower, but with 16GB of ram, after a short while, normal background activity (starting apps, linux prelink etc) has cached about 10GB of data in RAM - faster than any SSD, and the $saved meant I could mirror that drive for redundancy. The drive is using mdraid software mirroring, in a few chunks, which means if I need to, I can use that second drive by breaking the mirror and using LVM to manage the space. Re-creating the mirror takes about 1 hour, or less. I never have to be particularly concerned about needing to allocate a great chunk of space. I can just do it.

I have a windows VM in the background (to allow me to print to my Lexmark non-linux friendly printer, and run a few windows only apps). It's running al the time, with 2GB allocated. I don't even know it's there. If I want to fire up another similarly configured VM (or 3), I just do it. No performance issues, no RAM problems.

So, all in all, I've got a nicely balanced system for my needs.

Any downsides? Only one so far: If you copy a large chunk of data from the HDD to a slow, large device (like populating 20GB of music to a mobile phone), it can trigger some poor interactive behaviour in the desktop. What seems to happen is that the data source quickly builds up a huge outbound buffer in RAM, that has the potential to take an hour to flush to the slow device. Then some app (say your desktop) decides it needs to write to that card (or somewhere - not sure yet), and fsync it. Desktop stops till buffer is flushed, and that can be 20 minutes. I think that it's tuneable to some extent. The vfat filesystem can be made to flush more often, and rather than have the vm system use default values for amount of buffer, which are proportional to RAM size, you can push the buffer size down a bit, and also force writing to start earlier. A juggling act that I need to work on, but the phone has most of it's content now, so the odd album is not an issue, just that initial massive copy.

Cheers

Darren

Senseless Android

2011-08-31T21:55:00.001+10:00

Well, after some trepidation and a typo, I'm running a custom ROM on my HTC Desire. The potential show stopper was lack of exchange mail and calendar sync. It wasn't completely clear what level of functionality was there.
I'd rooted the phone a while ago, so had an easy path for backup and recovery that I tested in earnest at one stage, when exchange sync failed to work. Turns out it was a simple case of writing down the wrong activesync host name.
So here I am with cyanogenmod7 and clockwork recovery. HTC Sense had some neat features, but it turns out that it adds little value to my android experience. The mail was ok and so is what cyanogen provides. All in all, the 2.3.3 android is pretty good.
Perhaps we are approaching the dawn of a less fragmented world for android. Samsung and Motorola have pretty light customisation and HTC have given up on gingerbread on the Desire. I have to wonder why they're even bothering to be honest.
Beat me senseless. I quite enjoy it.

Update: For a really slick experience, have a look at miui. They've done a really tidy job.

Caravan Stability - where does the 10% rule come from

2011-06-03T09:47:00.003+10:00

This is not a Linux post :)

Youtube is full of videos of caravans getting out of control, much to the delight of Top Gear fans (and hosts). One of my pet projects is the design (and hopefully construction) of a large (6m) lightweight caravan, which will be the subject of another post (or three). Anyway, one of my concerns has been the stability of the end result. What makes it a good tow, will it be safe to tow.

One thing that I've wondered about is the origin of the 10% rule that is part of caravan culture. Nobody has directly explained where it might come from, but many rightly state that it's not a rule, just a recommendation. The rule says something to the effect that "You must have at least 10% of the van's weight on the towball to be stable". This rule just didn't seem altogether right as an absolute, both from the perspective of the load factor and the idea of absolute stability. In particular, my tow vehicle does not like heavy ball weights (poor load carrying springs), so I'm aiming to keep that as low as practical in my design.
What I have found is that the boffins say that essentially all conventionally coupled and proportioned caravans can become unstable. The system is like a double pendulum, and at some point of operation, it can just go nuts with the right input and oscillation. The live variable that changes a stable towing experience to unstable is speed. Something that settles quickly at 90km/h might become uncontrollable at 100km/h with the same disturbance. What makes one van stable at 100, and another unstable is very complicated, and the change in mode is very rapid. Most mathematical models need to represent the system with 4 degrees of freedom to even explain basic dynamics, and many more if you want to include the vehicle.

The critical things that can result in poor stability seem to be (in no particular order - numbers for reference only):

centre of mass not far enough in front of the axle
a relationship between radius of gyration and product of distances between ball to Cm and Cm - axle (Rgy must be less than product)
large moment of inertia (either massive van, or high mass around periphery)
large mass (particularly wrt the tow vehicle)
large overhang on tow vehicle
aerodynamics matter

These all make sense, but I think that is largely in the context of a normal caravan. The first two are strongly related, and probably the same thing. The first can be directly used to say that there must be some weight on the ball, as the ball is a lever holding the front of the van up. If the centre of mass is over the axle, then we could effectively have zero ball weight. A normal van like that is typically massively unstable. However, if we look at the interpretation of that from the perspective of #2 (distance between CM and axle is zero in this case), we need to make sure that the distance from the axle to the ball is large. Imagine a simple van represented by a massive blob of steel sitting on the axle. With a short drawbar, you can imagine it might be able to bounce around a bit (up and down as well as side-side), but with a very long drawbar, it will be reluctant to swing around (take it to infinity if you like). I can't see any reason why a zero ball weight van could not be stable with a long draw arm, but you may bang into practicalities like tow length and structural stuff.

The rest are mostly common sense (big weights keep moving once started, and big levers exert more torque).

So what would my van look like to achieve stability? I think it might have the following attributes:

overall weight as low as possible (minimalist approach - it's a caravan, not a house on wheels, just make it warm, dry, pleasant and avoid taking the whole box and dice)
front half (or more) of the structure would be very light, and avoid extra mass (small/no storage areas)
most of the weight would be centred around (or slightly forward) of the axle
most storage compartments would be clustered around the cenrtre of mass (low intertia)
axle would appear to be set well back. The front of it is really part of the drawbar, and it's weight relative to the whole van should be small, as if the front was a tent sitting on a long draw-bar
heavily sloping, streamlined front to move centre of effort from side winds back towards axle (but will make the front accomodation a bit like a boat, but that's no bad thing). This also give aerodynamic damping. As airflow becomes more oblique, drag increases, forcing the van in a straight line. It also reduces the force from trucks and the like

Here's a few references that are pretty interesting:
Vehicle Stability, Dean Karnopp, Marcel Dekker Inc 2004
Collyn Rivers paper on Vehicle Dynamics

Update 24/6/2011: Well bugger me - just stumbled across the Evergreen Element caravan in the US - Almost exactly how the idea brewing in my head over the last 4 years was panning ou. 24ft van with a tare of about 1400kg .... and they thought I was dreaming.

Next best thing since more RAM

2011-06-02T17:03:00.000+10:00

A while ago, I raved about using USB flash as swap. On a laptop, RAM is king, as unless you have a fancy SSD or hybrid drive, you've probably got a 5400 rpm drive that will rapidly become your bottleneck. For you machine to perform, you need to cache that drive as aggressively as possible, and avoid swapping to it whenever you can.

The USB stick approach has worked well for me, but I still don't like it dangling out of my docking station (and not having it on the road). I've recently found something that is just as good. It's called compcache.

What it does is makes a swap block device in RAM, and compresses that, with overflow to real physical swap. It kind of sounds counter-intuitive (like the USB), but it works well. Say you have a machine with 4G RAM. compcache will expose itself (by default) as a 1G swap. In reality, filling that 1G swap probably only consumes 250MB RAM (4:1 compression is what I see), so you get 750MB extra disk cache. Importantly, that swapping does not hit a physical disk. That's nearly an extra GB for free. All those unused anonymous pages are zipped away for immediate retrieval. It uses more CPU, but that is well and truly compensated for by saved IO.

Once you use all that compressed swap, lower priority physical devices take up the slack.

It also makes sense at the other end of the spectrum, where you are very RAM (and I/O) constrained, like embedded devices. A lot of those devices would traditionally not have swap configured, but now, they can leverage some virtual memory without a physical disk, or speed penalties.

It may not be easy to setup (perhaps a custom kernel), but it's easy enough with modern (2.6.35+) kernels:

http://code.google.com/p/compcache/

Beware the rare earth magnet

2011-02-14T11:56:00.004+11:00

One of the things I love on my Android phone is Google Sky Maps. I re-installed it a while ago, after getting my OTA update to Android 2.2.

I went to fire it up to amaze some friends while camping, and I thought I'd had one to many beers. It wouldn't work. Some googling seemed to imply that there was a calibration issue with the compass with this release, requiring at least a re-calibration. This helped, but things were still wrong. It would point north (ish), but not follow my rotations very well, and every time I started a compass app, the whole process needed repeating.

I stumbled upon some discussions asking about phone magnetisation, and it wrang a bell with me, and my symptoms. The phone spent some time (a few days) in the car against a magnetic name badge, which may have been able to magnetise something inside the phone.

But how to degauss? I found an old CRT monitor (of the variety that makes the right "degauss on powerup" noise - that sweeping deep sound). Put the phone behind it, and power cycled the monitor once or twice. Tada - the compas is dead on again.

Update: I still need to re-calibrate frequently (as per common thread for Android 2.2), but once done, it's all good. I suspect the device is not saving calibration data, but a quick flick of the wrist in a figure 8 backflip(ask youtube), and it's working properly. Just looks a bit lame, that's all.

Cheers

Darren

Update: 10-Mar-2011 - found a fix for the forgetting calibration issues:
http://forum.xda-developers.com/archive/index.php/t-747398.html
basically, you need to root your phone (easy - warranty voided though) and delete some config files. All good as new now.

Doh - busted again by calibrating g-sensor. Looking at the file, a new one is re-created, with wrong owner:

system system 46 2011-03-10 17:06 AK8973Prms.txt

#chown compass.compass /data/misc/AK8973Prms.txt

fixes it again (needs compass opening and re-calibrate). While the ownership is wrong, I bet the saving fails silently when compass closed. I wonder what logcat shows (nothing)

Equations of motion and your MTB

2010-03-02T16:56:00.001+11:00

Short and sweet this one. Popped into my head the other day, so it pops out here...

How do the MTB pro's make it look so easy? They seem to just glide over obstacles with such little effort. I think there are 2 things:

Skill
Speed

I'm struggling with both (but improving), and hopping even small logs mid trail is a distinct set of front-back lifts most of the time. Sometimes, however, it's fluid and easy. Thinking about those easy ones, I was generally travelling faster, and I think this is part of the key.

Why does speed help? Lets assume that you are clearing a log on a flat surface (no ramp). To take other technique things away (that can let you clear higher objects), pretend we are seated (and remain seated) on a uni-cycle MTB, which is not recommended by health care professionals. Lets also assume that the log height is round about the height you could clear doing a standing bunny hop (which in this case is how high you could raise your centre of gravity by jumping- probably not all that far)

Going back to basic physics (projectile motion), Once airborne, your flight path is a parabola - constant horizontal velocity and constant acceleration in the vertical. There are a whole set of these parabolas that will let you clear the log (just touching). The ones where horizontal velocity is lower have shorter bases and higher peaks. To just make the distance, you have to get way more height than if you are going fast and just skim the top

So there you have it - speed helps. Suck it up and hop that log with a bit more speed and commitment and it may be easier than you think (or you may smash yourself and bike to pieces - don't blame me - this is physics, not legal advice).

Happy riding.

Darren

Why put your linux swap on a USB stick

2010-03-02T15:36:00.001+11:00

What! Why on earth would you want to do that?

Glad you asked. Here's why (the how is later). My daily workhorse (at work) is a Toshiba Tecra A9, running Fedora 12 (64 bit). I recently upgraded to 3GB RAM (tried 4, but the bios seems to hide most of the 4th Gig in IO space etc). It has a small 80GB hard drive and that is its Achilles Heel.

Moving to 3GB encouraged more multi-tasking, and enabled use of VM's etc, which means I'm actually more likely to start madly using swap space. The problem is that as soon as this happens, things grind to a halt. The poor hard drive is not too bad at sequential IO, but as soon as it gets busy, it gets slow.

I'd been reading about SSD's (in the context of a new machine), and was wondering how to best use a smaller SSD (large ones cost too much for me at the minute). I'd read about things like overlay filesystems, Windows ReadCache etc. What I thought I wanted to achieve was a filesystem cache on the SSD, but no clear solution for that existed. Then I realised that by making linux more swappy (swappiness is tunable), I could better use my RAM for that filesystem cache, and the SSD could be the swap space.

I'd found a few reasonably priced small SSD's, but I have limited expansion options (Cardbus and USB - no express cards etc). When reading about Windows ReadyCache, the talk was that random IO was more important than raw throughput (and in fact, many devices that are fast in a straight line suck when concurrent read/write is happening). It also aligned with the problem I wanted to eliminate, which was those long seek times

So, as an experiment, I found a cheap USB stick (4G SanDisk Cruzer) that had the ReadyCache logo. I partitioned it up as swap, and gave it a go. It seemed to work well. I did some research about block alignment etc, and decided that I'd just use the whole device as swap. This should give good alignment, and certainly while mkswap was checking the device, we got good IO throughput.

Did it work? You bet it did. I added the swap as a higher priority device than the existing swap, increased the swappiness factor, and gave the box some work. Almost straight away the machine felt better. Typically as soon as the swap was in use at all, I'd get interactivity issues, but now things were smooth as a baby's bum. It's worked so well, I've left it in place.

What are the downsides? There are a few:

My swap file is dangling out the back of the laptop, which alarms me somewhat. My docking station provides a relatively protected USB slot, and a bit of tape stops the odd accident.
When push comes to shove, the IO is not that great on the cheap USB drive, but under normal, gradual swap activity, you don't notice, as writes are delayed slightly, and it's only when you trigger massive sudden swap thrashing that it gets slow, but no worse than the HDD
Not sure how long the flash will last - I suspect that the write pattern is not optimal, but assuming good "whole of device" wear levelling, it may not be an issue. I suspect that wear leveling is not across the whole device though, but more on that later

Why does it work? Those USB's can be so slow...

Using sysstat, I can see that typically, swap write rates are low, and once you've written something to swap, it stays there and never needs to be re-written (so repeated swapout is low cost), so total write load is really relatively small most of the time. Once your working set is established, it's mostly read access.
read access has very low latency. I'll get some numbers, but when the machine is busy, the HDD service times can get into the second range, but the flash stays very low. Once you working
Most swap ins are incremental reads and watching vmstat, I see just small bursts of IO easily satisfied by the USB drive in reasonable time.
By increasing swappiness (now that you have a better swap device), the kernel can use more RAM for filesystem cache, so it's a win-win situation. Faster filesystem access and faster VM system.

What about write endurance on the USB. That's a bit of an unknown, but I have some theories, and at the end of the day, I don't care - it's a $20 investment that will be retired after a year or so anyway. Here's what I think:

If the device is able to level the wear across all it's blocks and the swap activity uses whole disk erase blocks, the scenario is good. I write out a few GB per day, which means that under ideal circumstances, every block on the device is written 2-3 times per week, so the flash should last for years.

If the disk wear-levels in smaller chunks, assuming that a full filesystem will have data spread everywhere (which swap is probably not - does any one know? I'll hunt through the kernel and see at some stage) then I might re-write the same blocks at the front of the device once or twice a day. I tend to have the same apps running for long periods, so I suspect that large chunks of the device are written once a day. Even so, you get a lot of days before the endurance limit is hit. Just put a reminder to replace the flash with the laptop, or buy myself a new one at christmas...

I'll write another post soon describing how I set it up, and collect some numbers and graphs.

Give it a bash. If you can't add more RAM to your machine (more RAM is always better than swap alone), and you have a slow disk (older laptops), it may pay off for you. It has for me.

Cheers

PS - since writing this, I googled with different keywords, and found SwapBoost (http://zelut.org/projects/misc/swapboost.sh) - same idea, swap in USB.