Academic institutions systemically promote exactly the sort of short-term optimization of which, ironically, the private sector is often accused

An interesting insight from On the use of a life by Colin Percival.  Percival is a talented entrepreneur in the cyber security space.   In this essay he is responding to someone's comment wondering whether there might have been some better use of Percival's talent. 

He responds in part:

Of course, the question at hand isn't whether I've done anything useful, but rather whether this was the most useful way I could have spent these years. Judging by the reference to the Millennium Problems, I imagine that the specific alternative they had in mind was a research career; indeed, between my Undergraduate studies in number theory under the late Peter Borwein and my Doctoral studies in Oxford I might have considered seriously working on the Birch and Swinnerton-Dyer conjecture had my life taken a different path. (A very different BSD from the one with which I am currently involved!)

So why am I not an academic? There are many factors, and starting Tarsnap is certainly one; but most of them can be summarized as "academia is a lousy place to do novel research". In 2005, I made the first publication of the use of shared caches in multi-threaded CPUs as a cryptographic side channel, and in 2006 I hoped to continue that work. Having recently received my doctorate from Oxford University and returned home to Canada, I was eligible for a post-doctoral fellowship from Canada's National Sciences and Engineering Research Council, so I applied, and... I didn't get it. My supervisor cautioned me of the risks of doing work which was overly novel as a young academic: Committees don't know what to make of you, and they don't have any reputational prior to fall back upon. Indeed, I ran into this issue with my side channel attack: Reviewers at the Journal of Cryptology didn't understand why they were being asked to read a paper about CPU design, while reviewers at a computer hardware journal didn't understand why they were being asked to read about cryptography. It became clear, both from my own experiences and from advice I received, that if I wanted to succeed in academia I would need to churn out incremental research papers every year — at very least until I had tenure.

In many ways, starting my own company has given me the sort of freedom which academics aspire to. Sure, I have customers to assist, servers to manage (not that they need much management), and business accounting to do; but professors equally have classes to teach, students to supervise, and committees to attend. When it comes to research, I can follow my interests without regard to the whims of granting agencies and tenure and promotion committees: I can do work like scrypt, which is now widely known but languished in obscurity for several years after I published it; and equally I can do work like kivaloo, which has been essentially ignored for close to a decade, with no sign of that ever changing.

Is there a hypothetical world where I would be an academic working on the Birch and Swinnerton-Dyer conjecture right now? Sure. It's probably a world where high-flying students are given, upon graduation, some sort of "mini-Genius Grant". If I had been awarded a five-year $62,500/year grant with the sole condition of "do research", I would almost certainly have persevered in academia and — despite working on the more interesting but longer-term questions — have had enough publications after those five years to obtain a continuing academic position. But that's not how granting agencies work; they give out one or two year awards, with the understanding that those who are successful will apply for more funding later.

In short, academic institutions systemically promote exactly the sort of short-term optimization of which, ironically, the private sector is often accused. Is entrepreneurship a trap? No; right now, it's one of the only ways to avoid being trapped.

Fair enough.  No disagreement.

But I think there is more to the answer, and true for everyone.  It seems to me that this is a Hayekian Problem of Knowledge challenge.   In hindsight when some ambiguities and uncertainties have been removed, we can speculate whether someone might have made a better decision or choice.

But at any given moment when a decision must be made, or can be made, we are in the middle of a flowing and shifting stream.  We typically lack much in terms of context or completeness of knowledge.  And whatever we decide in that moment inevitably shifts the portfolio of choices even a short way down stream.   One choice leads to another and a mile or so later we are clambering up safely onto a smooth beach on one side of the river, desperately clinging to dense foliage on the other, or plunging over a waterfall whose existence was unknown some minutes earlier.

For Hayek, part of the Knowledge Problem was solved by the pricing mechanism of a freeish market.  Price contains broad knowledge which can be incorporated into local decision making. 

Similarly with the question about Percival.  Did his choices lead to an optimal outcome?  In some respects it is a nonsensical question.  What were his goals, his costs, his rewards, his constraints, his risks?  Only by beginning to understand those fundamentals can we even begin to conceptualize an answer.  And it still ignores the reality that each decision and experience dynamically interacts with downstream decisions and experiences.  

We think we want to become a fighter pilot but in training become fascinated with the limits of structural design and we end up chasing an adjacent goal of improved design.  At what which point in the decision stream do we optimize?

Another way to consider it is that any outcome arising from Percival's decision must tautologically be the best outcome.  He was the one making the decision and as long as his decisions were consonant with his best estimate of knowledge and probabilities and not intentionally self-destructible or random, then the outcome must be optimal for the limits and conditions at the time.

Anything other than these two answers can only be speculative.