Lots of energy around the concept of Rack Scale Design (Intel’s nomenclature) in systems design these days. Instead of depositing a cpu, memory, I/O, and storage on a single motherboard, why not have a rack of each, interconnected over a high-bandwidth, low-latency network – PCIe is favored today – and use software to define bundles of resources as “servers” on which the usual software runs.
The economic payoff comes in two flavors. First, it should be possible to raise resource utilization, so more work can be done for a given investment. Second, it should be possible to invest more frequently in the specific technologies – GPU, for instance – where progress is most rapid, without forcing new investment in longer-lived tech.
Broken assumptions
But the RSD concept breaks a lot of assumptions that are baked into system architectures. Storage, for example, assumes that the storage stack – software and physical I/O paths – are static. But in a networked rack scale datacenter, why would they?
That’s the problem tackled in a new paper from Microsoft Research, Treating the Storage Stack Like a Network, by Ioan Stefanovici, Microsoft Research,
Bianca Schroeder, University Of Toronto, Greg O’shea, Microsoft Research, and Eno Thereska, Confluent, Imperial College London.
The paper’s
. . . main contribution is experimenting with applying a well known networking primitive, routing, to the storage stack. IO routing provides the ability to dynamically change the path and destination of an IO, like a read or write, at runtime. Control plane applications use IO routing to provide customized data plane functionality for tenants and data center services.
Areas where this could prove valuable include
- Load balancing writes to less-busy storage.
- Ensure reads always come from the latest data.
- Supporting SLAs in multi-tenant systems.
- Supporting per-tenant cache isolation.
Now the hard part
We’ve been routing networks for decades, so how hard could this be? Pretty hard.
The basic problem is that data networks deal with copies of data, while storage is dealing with originals. Another wrinkle: applications have varying expectations of storage that need to be respected.
So the question of where routing switches are placed in the storage stack has important implications for data integrity, storage latency, and system performance.
What they did
Building on the earlier IOFlow storage architecture, they
. . . designed and implemented sRoute, a system that enables IO routing in the storage stack. sRoute’s approach builds on the IOFlow storage architecture. IOFlow already provides a separate control plane for storage traffic and a logically centralized controller with global visibility over the data center topology.
After examining storage semantics across a number of systems, the team concluded that I/Os can be classified into three types:
- Endpoint – I/O goes to a specific file.
- Waypoint – I/O goes to an intermediate destination, such as a cache or specialized processor.
- Scatter – I/O goes to muliple sites, for replication or erasure coding purposes.
Sounds like data networking, no?
The StorageMojo take
I won’t try to summarize the entire 26 page paper. Suffice it to say that the authors demonstrate important use cases, such as tail latency control, replication, file cache control, and performance debugging – vital in a fully distributed infrastructure.
They also show that routing I/O can offer significant advantages. Nonetheless, a number of open issues remain.
One of the most interesting from an architecture view is that this could force network and storage management integration. If the storage wants to go from A to B, but the network management won’t allow that, you can see the problem.
But that could drive the second coming of the SAN. Take that, hyperconvergence!
Courteous comments welcome, of course.
To make the most obvious comment maybe of all, the problem of deconstructed designs where normally electrically closely physically coupled components are made removable or remote, directly or switched, is the accumulation of edge case signals that the chipset and in any truly resilient design also the upper software stack needs to be able to handle.
Consider the commonly deployed “data diode” appliances in which a communication signal is allowed to traverse a network node in one direction only. I’m not familiar with the equipment parameters as I have only recently become aware of individual vendors including Boeing and must tiptoe my way towards any familiarity from the clearance disadvantage of British citizenship, but the essential theory of operation struck me as almost immediately relevant example here: to ensure further packets are sent, the diode appliance must send TCP ACKs to the dispatcher network. This presents the question of behaviour in failure modes: if the receipt of a datagram is corrupted, e.g. in a later decode stage, and the diode appliance has already acknowledged the receipt of the datagram, how to negotiate with the dispatcher network for replay? I’m not going to attempt even a summary logic state dissembly but it sets my hair on end in considering the advantages of the diode appliance caching the transmission for replay. Because I can’t even figure out how to classify the cache data. And if the cache data is a miss, to what extent is the diode appliance now a classification risk at best, if it intelligently refreshes the cache? I’m extremely rusty on the formal classification approach required here, but I suspect that the data diode appliance has approval based on theoretical transparency and the moment it is a actor in the data path even so “simply” the full stringency of qualifications apply.
My example is a extreme one in the obvious demands that security in classified network systems place upon it.
But that’s a good highlight because the problem is the same as asking a computer system to remove components at the ends of arbitrary separation.
In both instances complexity is difficult to appreciate because of the possibility to strip protocol context from the path.
The data diode appliance at least for the above vignette is apparently (to limit of discussion stated) much simpler than the layers of the storage stack in any practically useful storage system.
I’m grossly simplifying everything but the data diode appliance has a function that can break expected protocol behaviour in a way you can see is analogous to a disruption of a layer of the signals fabric used to assemble components flexibly in almost all current advanced storage systems not only in the ones switching lower layer paths.
I’m convinced privately that the pain ( potential in sleepless nights and real and acute in the event of failure modes increasingly hard to debug, something that has to be a familiar convern to readers here, now the kind of proprietary firmwares of controller cards should be a historical memory only) requires a response from the customers not vendors.
I’m disinterested to see more competition in a highly competitive market that’s​true relief after the recent hegemony, but which is now instead forcing mergers and acquisitions that diminish choice.
At least not if the features I think are essential to the future are the competitive assets to be fought over.
That only holds the customers hostage again.
The capability I require is for the data object envelope to encapsulate the live request status and requesting object and pathway.
This is a security requirement but it sets the stage for recovery of data transmission state in the event of failure in a arbitrarily complicated stack.
I’m at liberty to say only a little bit more than this, fairly equally because of my partnership policy as well as the underlying novelty of my efforts mean that there’s only so much to show. I’m personally adding the fact that I’m not far enough along with this to be personally comfortable I’ve not been reinventing the wheel or overlooked a obvious showstopper. That’s my excuse, forgive me! I’m surprised to find nothing yet in the literature almost as surprised as finding my partner’s seeing the point of my interest so quickly. But I suspect that that is the effect of storage becoming a standalone budget item in the past year. I broke it out from the regular financials, a excercise I can recommend if you are similarly able: this enabled me to restate storage costs as a security and continuity business function. I felt it was necessary to do this because of storage becoming the predominant demand client for all systems we purchased. This effort completely rewrote the nature of intercourse with vendors and the users who have been converted to treating storage as a profit center required to book income to have a future. Later on, we’ll expose the function to competitive bidding to provide it turnkey. If I hadn’t torn storage apart from the rest of the company financially, we’d see the internal efforts dangerously favoured by the numbers. This presumes that you can invite bids to supply future capability in contract, not only delivery to your needs of present. I hope I will be allowed to say something about that adventure when it begins to be a understood process. But until then I can say this is my attempt to hedge against talent acquisition deals that estop product development along the roadmap part of the representations. Random hypothetical example would be a Mangstore like vendor promising mainframe systems support in the next major rollouts, but a HDS buying the talent and IP and protecting their existing mainframe direct attached and sysplex offers. This is my current fear for the latter-day storage stack losing the competition just finally breathing life into departments everywhere. I’m deeply concerned about losing the opportunity to redefine the industry in ways portended by companies like Liquid in oversight lapses that forget to bring about the performance to the unnamed oft silent majority of unlimited budget customers who if on mainframe systems are the last to see benefits but possibly the most eager purchasers. I’m immediately impressed with the equipment possibilities of PCE/e switching at the lowest signals that could isolate the new ability for compatibility reasons. I can think of who would be great sites prepared to allocate the resources to program I/O to utilise potential in a few probably not forsee ways. (But recall that CICS was a remote file and resource method ultimately still is) and the talent does exist to continue products in such spaces but it’s rare that any innovators reach out, despite so much important virtualization being basic knowledge sets even at Sysop levels, life being closer to the metal. (And a longstanding neighbours I was acquainted with upon meeting in the elevator, why was I grumpy, well a NT4.0 network rollout did suck, “I/O locks I bet.” It was indeed. “My first job was to write a FS for the new machine.. ” so I made a friend.)
My apologies for the length of my comment Robin, the subject touched so many concerns which gained new priority from seemingly nowhere inside just a year, despite presaged so long by the few like yourself.
I think the consequences are not now linear algebra and the edges of abstraction now begin to pick argument with the best planning. I’m surprised by how much I changed my appreciation without actually being fully self conscious of the process. I think the influences may be not unique on delineation, but I must reflect first. I’m aware my thinking is a potential for friction with the cooperation of vendors and suppliers I must talk to, and it’s early days still getting the language and common interest settled. I’m unapologetic however if I showed my hand, it would be a scary one, in which so much that only appears as features of high end storage systems is addressed most easily in core application logic that no longer requires a array or similar deviations from the simplest way of attachment of storage. I’m seeing the hardware and license bills and the cost of development support, i.e long term hiring of employees able to write software that meets the requirements, is a genuine alternative. The hardware and software costs only look vaguely justified if you sell me wholly managed services which are a complexity in the process of sale alone which places profit against peace of mind by the very inherent nature of it all. I will be flippant and conclude the transaction costs of a imaginary RESTful web presentation of storage, in which the synch/asynchronous boundary layer of application consumption in any, let alone HAcontexts, is a language feature in present and constant flux. This is the reason I understand the importance of the latest JavaScript systems and stacks command such high salary and extensive discussion: the vital logic is running​free today among the forefront of the beneficiaries of easy scaling systems. Who are those who are least likely to be responsible for the HA effectiveness. In reality the most important aspects of system programming have been stolen wholesale and stuffed in the front end baby basket and the market and the management is chasing the abduction only unsure how to call out what is going on because only in some very vague but deeply nagging way does anyone appreciate the crime. I’m screaming like a disposessed new mother – bring back the future of our systems already!Yes you grabbed the right babe, but you’re just too young to raise it right to fulfill the promise we all not need but depend on!
Meanwhile and I wish you covered the topic Robin, but I realised only recently the meaning of the dot net ecosystem open source policy: the same could not be created possibly ever as a matter of necessity even, but what it is is the foundation of any alternative to the present client ecosystems: if I wanted to expose the T10 metadata to Word, well this is not a pipe dream suddenly. I’m reading blogs about writing new primitives in the intermediate language. So I’m also reading of the possibility to extend low lying components of a storage system as the underlying types that languages manipulate. With portable bit wrangling implementation. This could be the creation of usage aware storage primitives: extension of a array or class object and even sorting the result of a retrieval could be observable to the metal, if metal means just the lowest manipulation of the stored bits. If a new object registered itself with say MTS and is remote, code in the CLR hypothetically could be serialising to a buffer and also copying the resource to the machine local to the calling method. Readers of your blog I’m sure know about the potential uses of alternative data streams in NTFS. I’m proposing to finally give them a job. Because very small data especially random data not only is but will continue to be a huge cost to access, the same tiny datum to describe a on disk format that might be requested of many locations now, should be a sequence of all known relevant or simultaneously called data, collated by the additional stream. This creates a new raid like potential that I can see being helpful to processor efficiency as we grow the core count. I think that at least for some possibilities, storage can present to the OS and vitally to application authors, as a salve not a injury for performance.
My comment is set against the impression that I received lately on reflection, that the opportunity to do truly new business with storage — I mean new business programming as we never could before and so creating entirely new business revenue, is in a unknown but direct relationship with the length of time during which the former abuse of customers who require performance of storage can be put to trial by the sheer size of the arbitrage that was completely sufficient to bring unprecedented capital to invest in new companies and new designs where the costs of viable product delivery is significant.
I’m seeing there’s a lot of challenges to the opportunity:
I’m seeing the holders of the most exploited customer accounts acquire to mitigate risk of displacement by the better technology moving up the value stack to e.g. Sysplex support.
I’m seeing the cost of development of hardware become both eroded and moated. Eroded when you get the capability of basic components and systems engineering such as Micron are assembling. Moated because the problem of performance and capabilities moves to software or firmware and very high hurdle rates to employ individuals capable- I think I couldn’t get who I want without the future being very well assured. Indeed the wall of unemployed capital seeks homes , suggesting that the sums necessary to do epochal engineering are available. But that capital still cares about how quickly it’s rewarded so invalidating it for the support of the characters I think fit a actual startup that can break a market open. You have to be patient with many different factors that I as customer need nurtured not skunk worked.
The domain knowledge that sets the entry barrier to be competitive in a startup market is often very specific within the domain. Hence serial founder CTOs e.g . But the potential for applying the talent to funny cats when we need a Model T, is clear risk.
This storage market is often very well insulated from the real world, because of factors like the bulk of the consumer capacity shipping to OEMs and responding to Windows launches. Or the recently noted nuttiness of short stroke capacity effects on business economics.
But frankly I see the entire industry as hamstrung by terrible integration, and next to suicidal marketing if you even can find any. If you can tell me what the latest marketing campaigns are like, call Robin not for the discussion but to get hiking until the air revives you. If you can tell me what advertising goes on..
Oh I should like that I suppose, being my business…but still getting outdoors is the last hope you have 🤣
But go ahead and read Thom Hogan and keep on reading what he says until the cows come home.
Thom was appointed marketing director at Osbourne just after the infamous mistake. Thom knows valley marketing from the start if it.
Really just read what Thom Hogan says.
I’m going to risk a summary:
The camera companies are so inept at making the amazing equipment usable that at least some very big names cannot be ruled out of the commission of commercial suicide.
You can’t even transfer the picture you just took to anywhere useful without pain.
Sorry storage world, you don’t see any issues like this because you’re even worse.
But that avalanche of 4K data ain’t a coming.
I’m concluding the chance of storage utopia is so diminished by the above factors alone, only something very left field stands a chance to make the promised facilities come thru.
I’m not arguing about the present available equipment, it’s good. But if the whole picture is as it is, to remain as it is now, we need a new creative director.
I spotted a logic gap in my description of the process of settling storage requests where a layer of protocol is abrogated. My description looks like the desire for a transaction FS. The real world need is to question operation when the transaction status is itself in flight and lost or corrupted. I’m concerned about losing sequential performance which is likely the biggest constraint on hardware, that we can’t work around. And latency. Is it faster to write back to the same location a transaction log at the nearest logical step, or to round trip from the requesting object and hold or hope the called object is in a cache. I’m thinking of the possibility of nvram approaching ram equivalence and the continued increasing cost of access to any cheaper bulk storage. Against the dissipation of iot style processing into the very atmosphere of computing systems, a pixie dust sprinkle of capability I am not sure anyone has good predictive handle on yet. Consider the assignment to individual data requests, even a simple core. This kind of helper core to my mind makes sense and a very different idea of a transaction FS arises as a result. Will we be smart at concatenation of requests to alleviate the random access surcharge to the point of a future where the iops demand crosses the available cores in a reasonable effort of hardware. A much simpler core than the present available CPUs may be. I believe that the eight cores in my phone could be squared to provide useful function. I’m thinking about assistive processing, even very simple logic such as listening to the intended transmission metadata and keeping tabs. I think a million suchlike​cores could fit in 4U soon. With downward pressure on the raw iops demand, again also from the application of dedicated tiny cores, maybe accelerating language features, at least my hand waving logic suggests to me some purpseful investigation of the possibility. This goes so closely with my comment on Microsoft opening​the language runtime and designs. Without that, I wouldn’t have been down this path to begin with. I’m probably ridiculously biased, the merit I see has to be available for a regular programmer to use. It must sell a product. I am not going there now but I also believe that the product must benefit small business and not the audience reading here. I’m so far away from my business roots precisely because, from founding in the mid nineties, more than I thought possible my job was to build access to the technology supply only even mentioned to big corporations. If we had a thriving small business world, I am privately convinced that we would be at far less risk of totalitarian politics. I’m a British Subject (note that correct title) and I am sufficiently read to know there’s a need to change in American business first, not because of the supposed supplication of British enterprise to American, buy it arises from cultural heritage, in which the passion inscribed in the genetics of waves of early American refugees from religious and secular persecution, created a exegesis of rigor and profession of new ways of life, which were never embraced among my remaining compatriots, the instigation to siezed opportunity some say first offshored to cheap experiment in the new world and latterly scorned first as trivial infatuation and then as offensive mercantilism. I’m no apologist for my country, the fact that we follow too often has many explanations. But in computing, American fervour for efficiency, the roots of Silicon valley pioneers the strictest of religious communities in some very notable persons, America found the perfect place for secular prosecution of a better life. We forget that at our peril, doing so seems to invite the least compatible of leaders with the pioneers spirits.
Flippant comment that’s been definitely cause of much thought lately: the onboard storage of this phone is exceeding several measures the last workstation I configure with a local array could manage. In 2005, the RAID 10 15KRPM SCSI subsystem was a GBP £5000 line item, before discounts. If energy was not a cost, I would still run the four cores of 3.9GHz Xeon processor, which wasn’t​ challenged by any use we could find, excepting 3D or database oltp styles, with a modest ssd.
But I am a generational lucky lucky so and so, and so is my company. Had I cut my teeth and been writing critical software for Linux and open source software, instead of COM+ and DCOM / CORBA and mainframes (and VMS locally) I fully believe that we would be chasing the ideals of”scalability” and totally unprepared to take advantage via the immense value added to local machine resources so cheaply by the latest storage technology.
Not for nothing was SQL server a good port to Linux: almost the only thing that had to be ported was the Microsoft Transaction Server, which had existed for a variety of UNIX systems before.
My favourite anecdatum is the story of Plenty Of Fish, the popular dating site which is well documented by the creator. Literally he wrote what became the biggest such site for a bet- to do so in spare time. One employee was eventually hired, but the last year prior to sale was the sole year the creator applied himself full time. Microsoft technologies are openly cited as the reason why this was possible.
I’m loathe to let any generalisation be drawn from this anecdotal datum. Indeed, resistance is not futile but rewarding.
However I am a generational direct beneficiary of my experience with Microsoft technologies, that independent the Azure services which are being modified by us: replaced really, just transition from at first using the shrink wrapped offering for immediacy and substituting the custom service we refine. This new service equally benefits from the infrastructure support, just the same as the Microsoft offerings do.
I’m not going to describe the issue surrounding a extremely nice fluke, however. This is not my point.
I’m trying to get attention to the unrecognised threat of reinventing the wheel:
This computer industry thrives on establishing new advances that make commodities out of old technology.
I’m just seeing far too much wheel design in current computing business.
And the market looks like a great story of many custom wheel shops selling the fanciest alloy spinners.
But if this goes on, the chance of a retrieval of the free common ground by a financially highly aggressive player keeps accreting.
In the same way I tell friends and customers equally, customers very vocally indeed, that the value of the most important data about anyone even the President is nearly zero I am warning about the current computing generation risking the very office we hold most important (regardless of whether we feel great who holds the office who would want the personal data of the President compromised..)
To explain: the marginal advantage of any data for even a very large number of advertisement placements is bounded by the fact that the cost of a impression has become not viable to measure individually.
So the database is always appended and the value applied by the owners of the store keeps decreasing at a accelerating rate.
Only a root and branch reassessment of the business will have any chance of addressing the privacy concerns. (Fwiw my business is required required to understand the dynamics but takes no actual part in this kind of transaction.)
Now take the personal data like it was the latest JavaScript library or stack (I think stack singular is actually plural since the complexity is so great now, so JavaScript Stacks, not JavaScript Stack) and because of the predominantly JavaScript Stacks skilled workforce able to implement the latest application demanded of them, both the technology and the methods used by the technology to enact critical business, becomes a moving target.
This means that the value of critical infrastructure is being reduced to the same as the next JavaScript Stacks nobody not in the right twittersphere has heard of.
I’m not actually criticising the skills and abilities and talent and not even detracting from the technology supply either.
In fact recently I see a good awareness of the security and architecture issues that I’ve never seen in a popular language to this extent. This arises from the dangerous exposure and immature environment, but the awareness is definitely and positively there. I’m critical of the way things are, not demeaning the talent. We’ve not yet washed away generations wholesale,, merely confused and conflated aspects and atoms of the systems.
I’m simply pressing the button marked Do Something Already, when I am unable to comprehend the systems in place to ensure a respository of business continuity knowledge and information. I’m not sure if I even see any.
This causes the value placed on the JavaScript Stacks to be pinned to the lowest level possible.
Of which undoubtedly individual talents capture a very big proportion. I don’t begrudge the amounts, but the haphazard ways to allocate resources us dangerous. With no actual plans to mitigate disaster…
Commodity software is paid for by the so called horizontal scaling.
Storage benefits from the demands.
But the transaction volume of Twitter could probably be supported on one commodity Intel server, not a fancy Kunlun, to name a part of note, why should a Chinese company server be the one that comes to mind…why did they make one for such a hard to persuade market.. maybe the demand is local instead and the feeling of the potential for efficiency such machines provide worries me were establishing a inverse Server Gap, alongside the accumulation of the least computationally efficient talent. Hmm…
I’m seeing analogue development in storage.
If we don’t move quickly to commodities the functions we can get performed*cheaper* by someone else, the opportunity will exist to obtain a stranglehold upon a critical narrow (to use Rockefeller’s maximum, control the narrows) and cripple our businesses.
This worries me deeply.
I fear were learning the wrong skills. China has energy advantage, having built hydroelectric supply starving neighbours of water dangerously, they have the manpower advantage, negating and describing the fallacy of the scalability promotion, and the feeling is that I have seen they are learning the optimal skills to assault out computing capability in devastating fashion, using the very same multiplier effects we fill pages of glossy marketing nonsense with, against us before any one of us sees the picture clearly.
Right now, storage companies are able to integrate with the biggest infrastructure of all, Windows, and utilise the springboard that is, to secure the scale of the dreams of every brogrammer start-up, only in a fashion serious planning can rely on.
Instead I only see the advanced features of storage on hopeful roadmaps and still entirely disconnected from the utility a genuine programmer wants to use to sell new products that stand chance to redefine the industry in ways which we became accustomed would leave the competition just unaware that they had been done for if they even felt the cut. I’m worried that this is the reality only we are the victims of our own advantage of technology. I forget the Asimov short story how by being too advanced the solar system is lost, but if I can find it, that’s the last comment I have and I hope I’ve not strained your patience too greatly to bear. I’m simply hoping that in another few short years, I may feel I can retire and not lay awake at night as I do now. Storage has been leading a renaissance, but the awareness of the mechanism of the renaissance itself I believe places the culmination of success in jeapardy. I want to be wrong, and I am talking steps to ensure I shall be proven wrong. But I hope I am not alone in reviewing the information and chances in pursuit of security.
Storage semantics:
Endpoint, waypoint, scatter…
I’m frustrated with the futility of semantics here which to my mind suffer from the apotheosis of the adjectival form.
This semantic describes incomplete steps or partial states, because​the classification is valuable only for in flight data.
And data at rest is a binary performance problem only: it’s in the right place or it isn’t.
My weekend reading took me from maths type theory relevant to compiler design as far as a touch of quantum mechanics and the well known quotation of Einstein’s”spooky action at a distance”.
This summarises my view of the above semantics very well.
Because the value of the descriptions relies upon this same kind of spooky action at a distance.
In this case the apply action is that of some kind of data management software such as cache or tier instructions, or intelligence in a application.
I’m not sure at all that a reliable storage system should be aware of the kind of semantic description posited here. For a variety of reasons including the formal proof of integrity.
I’m suspicious of marketing being too intertwined with engineering to produce a expression like this, which describes no actual action that I nor any customer can inspect.
What we want is the data to be provided as efficiently as possible.
The method, if one is not reading a technical paper in depth, ought not to be front of house discussion.
Instead we need to know how the necessity of the required data location (RAM, and other stores) is satisfied.
How can a system recognise and meet this simple and binary inquiry, “is my data in the most accessible place it can be?”.
Without profiling every application, the only means that proposes itself to me, I have stated: stream access context including the exact method calling the data, alongside the on storage object. By including in this stream up to data performance observation, invoke assistant processors to inspect the performance and context information from the nearest to the store repository possible, likely a job for tiny cores not yet sold, and provide methods to instigate storage actions invisibly to concurrent access to localise the data appropriately and update the high level controller or even the application language runtime when done. This is the sort of task susceptible to all manner of well understood optimization including in flight optimization and parallelism.
I’m too easily distracted by tangential issues (sorry Robin!) But I think this is another case of muddying the waters in a way we have to avoid for our general mutual welfare. Don’t let someone else go just do something about it whilst we’re all in a twist making up new words to describe the action we desire. I have always suspected that if the Inuit relied on infrequent snow fall for drinking needs, the proliferation of words for snow in their language would never have occurred.
PS for clarity only, the possibility of my proposal is made fortunate by virtue of sequential access being cheap. Secondary streams are freely offset or very cheaply interleaved.
This proposal furthermore consolidated the potential for optimal fitting of knowledge about the stored data under machine learning friendly conditions. I’m not a fan of the lack of insight into ML despite very recent tentative progress in proofs of observation, but the black box nature of ML seems greatly preferable to reliance upon proprietary algorithms. I’m thinking about when I was flatly informed that the cost of installing Oracle’s database at multinational companies was commonly in nine figures before any comma. This wasn’t necessary to persuade me that hand tuning large data schema is not the future of mankind that I want to bet on. I’m skeptical the potential exists to apply similarly inefficient tuning in large data stores generally. I’m concerned that the value of the systems on offer is increasingly tied to pot luck as the assumptions fit or do not. And I want to retain the right to my own metadata insights and not pass them on as bounty for the supplier of my storage to use elsewhere or for boasting purposes that may inadvertently cause my company name to be a lever in a sales process for no reason but fluke, which is a legal nightmare scenario of a different order. But a scenario I am able to imagine just as how employee references are strongly advised against lately by our counsel and caused me to expensively find a viable personal form alternative to do the fair thing.
I still think we’re far from being doomed, and the worst case scenario leaves plenty of time to revert from. It’s just how easily we all get dragged down decade long cul dear sacs in this industry, a d my age that I think about hoping to see some rationalisation if not conclusive resolution of probably the most important function a business must solve. Storage is too important to leave to the storage companies. I think programmers absolutely capable to take it from here. If not distracted too much.
Ouch, I believe I dumped my thoughts from the year since I last commented, Robin.
Sorry I really could have distilled the above significantly, and helpfully separated my mind concerning storage accounting.
This simple move, to strip storage system costs out of business units, and place the account at least on the same level as business units, probably is the single best action I’ve instigated.
In fact any thoughts about storage have stemmed from the fresh emphasis. I assume the similar is possible and can result in different insights in each situation the excercise is done.
For instance, the costs of procurement of the Hitachi systems for a very modest Z sysplex, drove that function to a contract supply. I’m not going to mention who, there’s few names in that game and I am envious of the business metrics they enjoy.
But the driver is the in house knowledge required to understand the sales process of such a system.
I’m not clear enough above, in saying that if you retain the talent, the miracle of the current storage market, is the reality of being able to consider addressing needs in house.
What I don’t make clear is the timeframe of such endeavour is dictated by the contract supply lengths.
And I don’t touch on the advantages a contract supplier has.
Start with the five year plus terms that make sense.
Now as a tenant, obviously not using the whole system capacity, the supplier can just throw already amortized hardware at your needs.
So the supply deal is likely good for a decade.
Therefore the horizon and planning you are to undertake is longer than a decade.
I’m deeply concerned about not the business or technical aspects, now, but instead the quality of our pension plan providers.
I was far too harsh above just how I wrote my observation of the way web systems engineering happens presently.
I let my worries carry the argument negatively.
I’m impressed in reality with how young professionals too often are lumped with the delivery of high availability systems which must be provided from tools that are at best only recently adapting to such demands.
This means raw talent is plentiful.
Which is raising all boats salary scale wise, in modern web programming.
I’m proposing instead, provided your organisation has the requisite skills and experience and willpower to invest, to be looking for the new talent to augment the intellectual assets sitting here lamenting the modern world.
Buy you can’t hire the talent into a far more traditional workplace and expect that they will accept less than the gold plated terms which the old timers earned.
This would be a recipe for disaster already.
Equally important, the salary rates do reflect the instability of employment and you may think it possible to have a balance and reduce the rates in exchange for the stability.
I wouldn’t accept that however, because a long commitment to a project that I could say little about, would hurt my employability.
This is my life: I’m tied to the company I could founded, for better or worse.
Had I not been fully aware of the possibility of long periods of doldrums, I would have sold out long ago. At about the ten years mark, I announced to my late co-founder had I hired myself I would fire myself, had I applied for the job, I would quit. That’s the hard reality of such plans and I think not unusual.
I’m mulling over the argument negatively to my mind, that salaries are close to peak sustainable, and so if you are a young hotshot buy just old enough to think about family and the like, locking in the past inflation might be a smart move. But my late co-founder refused ever to permit me to pitch on a negative premise like that. So I’m going back to the drawing board and asking what is the reality of the present organisation being attractive if presented to my much younger self. I loathe the attitude of seemingly common belief held by investors, that you programmers can afford to work for noodles profit levels. This is so anathema to my own success’ derivation: I started a company very young but I had a unprecedented two hundred years of multinational board level talent (semi retired) guiding me. To throw young professionals or worse new graduates out to find out how this business works alone, is cruel, and I had such a row with the founder of a angel type find now notable I had my last emails returned by the MTA. I probably was too fervent. I know I was.
But who marked my email on a send mail blacklist, is very up on recent rounds of a young company, with substantial fortune magazine list customers, who recently aired in public their complete lack of exposure to critical storage system characteristics.
I wonder if the expense of my efforts to have access to the sales channels of much larger companies was a necessary experience.
Buy it really didn’t have to be done, were I not the details facto CIO from the start. I was advised by my partner who has been a final decision maker for a global corporation, that I had to make the calls for two reasons, first familiarity with the system being written, but most importantly because the budgets for enforcement of contract terms simply did not exist. This changed forever the moment we had onwards consumers of the supplied hardware etc: the so called box model of contracts would compress to expose our supplier directly to the complaint of our customer, in the event of failure modes not our making. Normal people around me were more impressed with the possibility of steep discounts off the supposedly already reasonable quotes from a well known web first vendor. Simple levers like this paid for the entire staff pension plan. I’m so against teaching business foundation as if it is a term project, because simply the learning follows the reality, and partners asking if you will be causing their grandchildren to be able to attend the college of their choice or not, is the first and last lesson.
I’m genuinely fearful of storage becoming a bogus priesthood all over again.
I’m even more worried about the casual notes that growing and new companies sound, often not going to think about the subject because of S3 taking the burden. This is dangerous because a company’s storage needs never shrink, but systems writing vital data change. This results in the risk surface exponentially growing by default. And defaults are well avoided for their scary ability to become​the standard practice or reliance by which real business decisions get avoided.
My approach to accounting is straight out of Nicholas Nassim Taleb’s playbook.
Only black swans are very common in business.
Rife in new business.
So I built a model of the probabilities of various failure modes, from outright wholesale data loss, to slow burn litigation causing problems by missing data required in Cort discovery. I applied costs of manpower and agent time (specialist recovery, law firms) and modelled for the impact on sales. I made harsh assumptions in some cases: two significant data losses in any three years would spell contract cancellations, as well as blowing out new business. I included the likelihood of senior resignations and hiring opportunity difficulty. I naturally collected data on the advertising costs expended by companies similarly affected. Of course big advertising buyers actually account that expense partly as reputation insurance, the trade press will be at least careful to check sources before publication if you book bulk adverts with them. But few enjoy such treatment and it’s the wrong assumption if you are not the client the agency trade press coo over.
I guess this is the result of having access to the brains of a capital finance PhD and former CFO of a fortune 500 during my formative years. (My story is one of pure luck, not family or suchlike privilege, entirely time and place, the time being sat in a bar during a bomb scare (1991, Victoria, London) the place stood next to a dept. Trade director pushed out too young and not knowing what to do next, wanting to prove it in the real world, himself a lifer in government, and a rushed nervous beer caused me to spout a elevator pitch before I heard the phrase by some decade yet then.)
But I see the possibility of storage becoming so critical, let alone the content addressed object types, to business functions, that if it is left subsumed under the same budgets for providing desktops and tablets, the outlook for far too many good stories is bleak.
I hope to be able to make my thinking public by the year end, with the context of my business, which still has no public website since the beginning… I was quite pushed to make a LinkedIn profile, and my colleagues are not yet persuaded. Our average age is almost pensionable, and the social networks that matter to us are still telephone Rolodex entries. What we do is not the most popular thing if you are a agency taking the vast commission for buying adverts with systems long ago forgotten about how this was promised to be automated. Instead of savings off the rich fifteen percent of gross, as high as sixty is captured now. The retiring CEO of a top network just said that online media is criminal, when he said he would not seek re-election to the board. No qualifications for the word. The officially accepted rate of fraud in online in America alone is eight billions per annum. Nobody believes it’s so low. Only this march did Facebook allow for customers to have data for the adverts they bought. I can’t think how much storage will be suddenly needed when finally people, shareholders because it’s so significant, demand audit logging. I am hoping to be running out with big buckets when this rain comes.
I think the opportunity exists for both customer accounts to gain insight into their business they never had, by simply accounting separately and vendors should build a approach to try to capture the results, which unlikely will be less demand.
My next exploration of the subject will be talking to activist investors. Because I found so much in the process of value. I’ll look into whether any hold storage tech stocks too… This is the kind of pitch I spent my life working on: finding the non executive director who would share a concern if I can present only positive approaches. My company never was able to sell by knocking on the front door. Heck we couldn’t get SUN servers back in the day, until I met by accident the sales director of a FTSE 100 VAR when I went to hang out with my friend a tailor on saville row… I was never close to anyone at EDS, but I remember distinctly, word for word in my last employer job, the way a former soldier was selling a large company director on the phone: it was a drill. I half heard a objection… “No, you’re not hearing the objective, this is that, that is where you are, our ability is x, time y, cost z, and I can act in today plus 72 hours, boots on ground, if you say yes, does this mean I am moving already?” After that I realised that I had to make my sales as succinctly, that real business men would not be offended by rapid clarification, unlike the way I was taught to try another angle or benefit, and the appreciation for action only increases the higher power you speak with. So long as (the total opposite of how I write) you waste no time. If it is urgent you can claim that only in brevity. I think the model which I roughed out last year, has the imperatives and brevity and the ready audience. But the vendors are talking about how they are all important to consider the way I’m writing: long form and dealing with too much explanation of factors that make understanding possible but dangerously assume the effort to understand is present and available. I had not written my thoughts long hand until now, which contributes to the verbose discussion. If I was pitching the idea, I would have a critical path analysis of the target customers knowable business and get the model down good for answering queries if shown via a linked spreadsheet. It’s not that there’s​immediate danger of the storage industry falling to bits, but the time is now to get talking about customer scenario details and not however​the product works. If my first advice was not wrong, then that is the first aspect of the c suite pitch I would ditch from opening stages. If anyone is curious what else I factored which there’s no space for inclusion here (and I’ve been most graciously tolerated by Robin and won’t rejoin with another long comment, I’m about done for the year anyhow!) I’m always happy to discuss when approached by genuine interest. I’m not actually a hermit or recluse, despite having barely contact with the visible parts of the computing world for many years. Occasionally just by being under exposed, the same subjects have turned out unusually interesting points for debate and the most frequent response I get has been definitely that the way I see the subjects has been different enough to be of value, often by counterpoint alone, in attracting new customers to hear from admittedly narrow so hence hit or miss but deeply considered evaluation. I’m certainly not a natural maverick, absolutely not as a character, but my history just always demands of me a less trodden approach. I’m not ever commenting for the sake of my business wishes, nor ever do I make claims based on what I may say. But I can say that some fascinating and significant deals stemmed from my public analysis, in the past, which is a genuine pleasure to have that experience especially because my entire view of public comment is the desire to freely submit what I can’t make use of, to be the opposite of Aesop’s dog with the hay. I’m even more verbose unfortunately because I try to never speak from the pulpit of specifics and be as general as I can. Rather the way traders ought not give stock tips, that’s more to protect the trader than the asking investor, but the idea is the similar anyhow. I’m long past my reasonable welcome and shall demur before I mention the two thousand IT folk who will be saved come… If you can stomach my actual devotionally pious point, the storage industry has too much of the most important economy resting on its shoulders. That’s responsibility, but it sets the tone for talking afresh with customers. I work in a industry where the top five accounts spend about half the dollars and the biggest, Samsung, spent more than the rest of the top ten by some measurements. When I sell a smaller advertiser, the outcome of which so very much is beyond anything either side of the call can control, truly matters. Small business especially is long excluded from the ability to advertise companies could count on even twenty years ago. I’m daily confronted by demand of a desperate nature:”how can I but without getting screwed by the big sophisticated players?” I beg you not to let storage become as similar experience. I’m fully aware also just how much the storage industry has to mature to address the technical factors that make progression with my now twenty two year old plan possible. I’m almost like the guys in The Big Short, and if you’ve not seen it, please do. Their predicament is absolutely similar to mine in a also impressive size market. But I need the storage product I don’t think will be built, to enable us to give some more level field to get small biz back in the game again. But that’s literally my entire life story so not carrying on further. As said, more than happy to discuss with any genuine interest if you even find my profile linked above. I greatly up with companies sending out technical journals for the asking, and bagged the most I could from about nine years old. We know what the IBM journal created. This just isn’t done any longer. It was being stopped while I was at school. I quit school (and the only real privilege I lucked into) to go do something about the problem… I’m certain that the American economy suffered directly. And leadership is not a thing you can do secretly. If you are not giving away knowledge, then you have insufficient in the building. I speaking up because I see such strong links, and regardless of strength, the lack of attention in this field leaves plenty to be established anew. In a way it’s a sad indictment just how valuable the time Robin takes, is to the industry, by providing this blog. Yes that is absolutely as flattering as you imagined you read. But emulation is the better praise, so go publish a journal my company can get printed at profit from the adverts already! That’s my evil intent dealt with for another decade or so…
If I can get about to writing this up formally, I hope to have a great recruitment story to boot. Not for now, but the other poorer explained part of my above comments, was that I’ve been assembling the required skills to implement storage system characteristics and components, from other disciplines. It’s entirely possible, on paper anyhow. I will know maybe for sure, in about another two years…