Reference
Methodology
How a page becomes a dated fact, and what is deliberately withheld.
Five steps, and only the first one is irreversible.
- Capture
- The page is fetched and the bytes are kept. This is the only step that cannot be redone later. A page not captured today is gone.
- Reading
- The capture is parsed into plans, prices, and features. A reading is stamped with the version of the parser that produced it, so a better parser can re-read old captures without erasing what the old one said.
- Change
- Two consecutive clean readings are compared. Differences become typed changes, each pointing at both readings.
- Fact
- Changes become dated intervals: what was true, and when we learned it. Both dates are kept, because a historical page is often read years after it was published.
- Measurement
- Facts are counted across a market, over a fixed cohort, and published only if coverage is high enough.
Four things this record will not publish.
A reading that might be wrong
Each reading is marked clean, partial, or failed. That is a statement about whether the page was read correctly, not about whether its prices are right. Only clean readings are compared, because a garbled parse and a large price cut produce the same diff.
A measurement without a cohort
Coverage improves over time, so counting whoever happens to be present each year produces a rising line that means nothing. Historical measurements use only companies present across the whole window, and nothing is published below 60% coverage or five companies.
A guess about what a feature is
Feature names are matched against a hand-curated list. Nothing is matched approximately. A term that cannot be matched is set aside for a human rather than filed under a near-miss, and it is left out of the counts rather than bundled into "other".
A confidence score
No probability is attached to a claim about the world. A pattern reports how many signals support it, how many contradict it, over how long, and what would refute it.
The four limits, stated up front.
- Feature counts describe pricing pages, not products.
- A company that ships single sign-on without listing it is counted as not having it.
- Some pages cannot be read automatically.
- The attempt and its failure are recorded. Partial and failed readings stay out of counts until a later capture can be read cleanly.
- Prices are captured in one region.
- A page served in a different currency is recorded as such and is never compared against another region's.
- Some historical dates are ranges.
- Where a change can only be placed between two captures, it is shown as a period, not a date.
Where the record has gaps, it says so. A quiet week reports the count it excluded and the rule that excluded each one, because "nothing moved" and "nothing cleared the bar" are different facts that look identical on a screen.