What do Freud, Hemingway and web crawlers have in common? Icebergs.

While Freud’s theory referred to the tip of the iceberg as the conscious mind and Hemingway as the part of a story he literally wrote down, the largest and by far the most interesting aspects of both the the subconscious ideas of the human mind or the conscious creations of a prolific writer, are deemed to be hidden to the untrained eye. It takes a bit of effort and sometimes a few years of training to see beyond the tip and understand the new universe of ideas that can be revealed when diving under the surface.

What is interesting about data is that it may be hidden both as an involuntary or a conscious choice. Developers hide data on purpose as it is not immediately relevant to the objective of their website, others show it but in a way that is not usually considered data since it's not structured as such. 

Many companies make only some indispensable data-points visible such as the title, price, category and description of a product; however, their API (Application Programming Interface) may hide information as relevant as historic sales, delivery times ,contact information of the supplier and much more. Likewise, comment sections, article descriptions, news articles and other unstructured sources may be data-gold mines for a trained eye, or machine, that is able to convert raw text into valuable information.

Whatever the case is, today’s technology allows us to automatically extract information, both visible and invisible, structured or unstructured, from websites at astonishing speeds. Automated machines are now capable of diving into the depths of any surface and scrape whatever data is available for grabs, incidentally freeing humans to focus on what they really excel at.

Here at Iceberg Data our mission is to let the machines do the heavy lifting by diving into the depths of the iceberg that is the internet, retrieve information that otherwise would not be available in any other source and allow humans to take advantage of the universe of data that hides in plain sight.