We are now performing full index data lookups on BuiltWith top sites (over two million websites) on a weekly basis.
What does this actually mean?
What improvements does this bring?
This allows us to uncover more technologies and provide a more accurate BuiltWith experience. The following improvements will be found within Trends, TrendsPro reports and standard technology lookups –
Websites using tag management systems load additional third party analytics and tracking technologies differently to their default installation. Full index provides us with the ability to track them regardless of how they are implemented in the original HTML.
Website owners tweak their code to run as fast as possible, especially mission critical websites where 1 second extra of load time reduces engagement. These websites sometimes have totally customized installations where we would previously have missed third party scripts loading because they would not match the majority of detection techniques we use.
Technical Challenges of Full Index
At our last run 2.4 million top sites create 92.4 million records. Some of these requests are for cached content or rendering errors with the site (a single image being downloaded many times for example) but show the scaling required to provide a level of detail unprecedented in web technology research.
We’ve greatly improved technology lookup, Trends and TrendsPro accuracy and coverage by doing full website page loads.