The Fraud
Inventory can "shrink" due to plain old larceny, phony receiving reports, and unauthorized write-offs of old (and fake) accounts receivable.
The Flaw
Depending on the industry you're in, there is bound to be some natural shrinkage in inventories over time due to minor errors of one sort or another. But large discrepancies are almost always due to fraud. Even during a physical count, a fraudster employee can use empty boxes, etc., to fool an auditor into thinking there are more units on the shelf than there actually are.
The Fix
At the risk of repeating the same advice, a formal policy of separation of duties, strict supervision, voucher accounting and all relevant internal controls must exist and be enforced at all times.
Physical counts should be frequent, sometimes on a surprise basis, and thorough. Nothing should be taken for granted.
Reconciling shipment records with sales invoices periodically is a good way to test inventory. Frequent inspection of the perpetual inventory records is a must. Analysis of financial statements to mine for margins, turnover rates, increases in costs of goods sold and delinquent receivables should be done monthly.