One Bitcoin Address effectively works like a Public/Private Key-Pair, except that the trapdoor function used is not RSA. Given the Public Key (aka Address), it is possible to Audit that Address, i.e. to listen for payments made to it. And, there are specific cases where a Public Key can be associated with a known identity.
For example, an Address may have been published openly, to accept donations or vending-machine payments. Because the Public Keys can be tested for equality, any additional occurrence of the same one, will be known to belong to the same identity.
Further, while the Private Key associated with an Address is needed to Send a Payment from that Address, how many BTC any Address holds is decided by the system of Peers, more than by the one Wallet. Therefore, if a third party can obtain a Private Key, he or she can use the network in order to “Sweep” that Key, which means, to order any BTC held by that Private Key Sent to some other Address. This can work without physical access to the version of the Wallet which has actually been stored on a given Computer.
What this might mean for me, if the HD of my older computer was to die, is that if additionally, I had an offline copy of its Private Keys, I could Sweep those Keys into my new Wallet, on my new computer, without having to revive the old computer necessarily.