Looking for Keys in the Dark: Navigating the Unmeasurable Future
Why organizations struggle to invest beyond the reach of their metrics
I was meeting with a client recently who was trying to rationalize how AI would impact their business metrics when the old story about the man and his keys came to mind.
You might know it - a policeman comes upon someone searching for his keys under a streetlight. After helping him look for a few minutes, the officer asks, "Are you sure you dropped them here?"
"Oh no," the man replies, gesturing to the darkness beyond the light. "I dropped them over there."
Confused, the officer asks, "Then why are you looking here?"
"Because," the man says, "this is where the light is."
This isn't just a silly parable - it's the story of why many organizations will miss the AI wave. In our data-driven world, metrics are the streetlights by which companies navigate - they form the foundation of how we plan, invest, and reward projects. These metrics illuminate the path forward, showing us what's working and what isn't. The problem comes when what matters most lies beyond their reach.
Like the streetlights in the story, our metrics can only illuminate what we've learned to measure, leaving vast territories in darkness. And in the beginning of any technological wave, that's exactly where we find ourselves - in unmeasured territory, trying to navigate before we've built new ways to measure. The challenge isn't that our metrics are wrong - it's that the future often begins in places we haven't yet learned to illuminate.
Building Streetlights: How Measurement Follows Innovation
Take the web, for instance. Today, e-commerce accounts for $5.7 trillion in global sales, with sophisticated analytics tracking every click, hover, and purchase. Our largest tech companies are built on the foundation of online interactions, each measurable, trackable, and optimizable. But if you were to travel back to the web's early days in the mid-90s, you'd find yourself in near-total darkness.
Back then, we didn't have the rich metrics we take for granted today. No session tracking, no unique visitor counts, no conversion rates. All we had was "hits" - simple lines in server access logs that recorded when someone requested a resource. And "someone" could be a person viewing your homepage, or just their browser automatically loading an image. The more images you added to your page, the more hits you got, regardless of actual human engagement.
These primitive measurements weren't a failure - they were all anyone had. The tools for measurement developed alongside the technology itself. IP addresses seemed like a reasonable way to track unique visitors until we realized a single corporate firewall could make hundreds of employees look like one user. Cookies wouldn't arrive until 1994, and it would take years to develop the conventions and tools that gave us the sophisticated tracking we have today.
The winners of the early web weren't those who waited for better metrics - they were those who moved forward despite not being able to measure. They understood that in the beginning of any technological wave, you're operating in darkness. The streetlights get built later.
This is where we are with AI - at the "hits" stage of measurement. We're still figuring out what metrics actually matter, still building the vocabulary we'll use to discuss impact and success. And that's exactly as it should be. But here's the real challenge: it's precisely when a company is most successful by its existing metrics that it becomes hardest to venture into unmeasurable territory.
When Success Breeds Caution
To understand how this plays out, let's travel back to a moment when the Internet wave was building. It's 1993, and somewhere in America, a tourism department is about to miss the internet revolution...
The head of marketing is presenting their latest success at the Las Vegas Travel Convention. "Five thousand travel agents visited our booth," she reports with pride. "These agencies collectively influence eighty thousand trips annually, representing over four hundred million in economic impact. Every agent received our new four-color catalog, and we're already seeing increased bookings. The numbers speak for themselves."
The room nods in appreciation. This is what success looks like - measurable, quantifiable, proven. Someone's Motorola MicroTAC pager chirps in the background, a small reminder of how technology is changing communication. But no one pays it much attention. They're focused on the metrics that matter.
Then a junior marketing coordinator, fresh from a technology conference, raises her hand. "I just got back from this amazing conference," she begins, her voice carrying both excitement and uncertainty. "They're saying that in the future, people might not need travel agents at all. They'll use something called the World Wide Web to research and book their own trips."
The room stirs with interest. This isn't the first they've heard of the internet - they even have their own experimental website.
"Tell us more," the marketing head leans forward. "How would this actually work?"
"Well, people could see photos of hotels, read about attractions, maybe even make reservations directly. The technology is still early, but they're saying it could transform the whole industry."
Several heads nod. It's an intriguing vision. Then someone asks the practical question: "How many visitors does our website get now?"
"About 50 last month," another staff member reports. "It's growing, but..."
"Fifty," the marketing head repeats thoughtfully. "Against eighty thousand trips from our agent network." She's not dismissive - she's genuinely wrestling with the implications. "Even if we believe this is the future, how do we justify shifting resources there? What metrics do we use? How do we know if it's working?"
The room falls quiet. Everyone can sense the potential, but the path forward is unclear. How do you measure the impact of something that's barely begun? How do you justify investment in a channel that reaches tens when your current approach reaches thousands?
"Listen," the marketing head finally says, "let's keep the website running, but we know what works today. We have the numbers right here. Let's focus on what we can measure, what we can prove."
The meeting moves on. The website remains a small experiment, underfunded and understaffed. Under the streetlight of traditional metrics, everything is clear, quantifiable, safe. But somewhere in the darkness beyond, the future is taking shape, unmeasured and unseen.
History Doesn't Repeat, But It Rhymes
That 1993 tourism meeting feels almost quaint now. Today, 80% of travel arrangements happen online. The idea that we once needed travel agents to know hotel availability or book a flight seems as outdated as those paper tickets and printed brochures. What seemed impossible to measure then has become the foundation of an entire industry's metrics.
But here's where it gets interesting: thirty years later, in that same tourism department, we're having the exact same conversation about AI.
Picture the meeting room in 2024. The head of digital marketing is presenting their latest results. "Our content strategy is working," she announces. "Website traffic is up 20% year over year, we're ranking first for key destination searches, and our social media reach has never been stronger. The numbers speak for themselves."
The room nods in appreciation. This is what digital success looks like - measurable, quantifiable, proven. Someone's Apple Watch pings with a notification, a small reminder of how technology keeps evolving. But no one pays it much attention. They're focused on the metrics that matter.
Then a junior analyst, fresh from an AI conference, raises her hand. "I've been seeing some fascinating demos," she begins, echoing that same mix of excitement and uncertainty from 1993. "People are starting to use AI to plan their entire trips. They're having long conversations about destinations, getting recommendations, comparing options - all without visiting a single website."
The room stirs with interest. This isn't their first exposure to AI - they've seen the headlines, watched the demos, maybe even played with ChatGPT.
"Tell us more," the head of digital leans forward. "How would this actually work?"
"Well, instead of searching websites, people will just have conversations with AI about what they're looking for in a vacation. The AI will understand both our destination and what they want, making personalized recommendations."
Several heads nod. It's an intriguing vision. Then someone asks the practical question: "How many people are actually planning travel this way now?"
"That's just it," the analyst replies. "We don't know. We can't track these conversations. We don't know what AI systems are saying about our destination, or how many people are asking about us."
"And there are all these concerns about AI hallucinating facts," another staff member adds. "What if it gives wrong information? What about the lawsuits we're hearing about?"
The head of digital is thoughtful. She's not dismissive - she's genuinely wrestling with the implications. "Even if we believe this is coming, how do we justify investing in something we can't measure? How do we know if what we're doing is working? What happens to all our carefully tracked metrics?"
The room falls quiet. Everyone can sense the potential, but the path forward is unclear. How do you measure the impact of conversations you can't see? How do you optimize for AI perception?
"Listen," the head of digital finally says, turning back to her dashboard, "we know what works. We have the numbers right here. Let's focus on what we can measure, what we can prove."
The meeting moves on. The AI preparation remains a vague future concern, lost in the shadow of proven metrics and measurable success. Under the streetlight of current analytics, everything is clear, quantifiable, safe. But somewhere in the darkness beyond, another future is taking shape.
Into the Darkness
This pattern of technological change teaches us something vital: the future always begins in darkness. The early web pioneers couldn't measure engagement, but they built anyway. The first e-commerce sites couldn't track conversions, but they sold anyway. The metrics came later, but by then, the pioneers had already claimed the territory.
Today's AI pioneers face the same challenge. We can't yet measure AI's understanding of our content, track the impact of AI conversations, or quantify the value of being well-represented in AI's knowledge. But that's exactly as it should be. New territories are always dark until we build the streetlights. The only question is whether we're willing to venture there first.