Tuesday, March 24, 2009

Blackwave.tv, innovative storage for internet video storage and delivery.

Recently I was introduced to an emerging storage company, Blackwave Inc. As I listened to a presentation by CTO, Michael Kilian, what impressed me was his clear conviction and articulation of why the company came into in existence. It was not because it had cool technology (that was looking for a problem to solve) but because the company had identified a tangible need and applied technology to develop a meaningful solution. (The technology by the way is intriguing). Blackwave was founded in 2006, currently has about 40 people and has shipped product. It is nascent but highly energetic.

The company’s value proposition is based on the premise that conventional file systems and associated traditional hardware platforms are not sensitive to the requirements of unique data types such as internet video. This application indifference leads to degraded performance such as longer latencies and stifled throughput. The team at Blackwave believe that this performance gap has created a substantial opportunity in the rapidly growing internet video distribution and content distribution market.

The real world opportunity that the team identified provided them with the inspiration and guidance for the technology solution they have subsequently developed. The epiphany was the realization that a storage solution optimized for video and digital content delivery must have unique requirements that traditional storage was unable to deliver. Their claim is that they are able to deliver video from the disk to the internet more efficiently, at a higher scale and with a greater simplicity which results in lower capital, operational and management costs.
Similar flashes of understanding are commonly experienced by a number of innovative but nascent data storage companies but rarely by today’s storage establishment or if it is it tends to be suppressed in protection of the status quo.

Some of the Blackwave guiding principles are;
  • The need to deliver streaming video to thousands of clients simultaneously.
  • The fact that all content is not equal hence different content requires different performance such as content specific bit rates.
  • Performance must be predictable to maintain video SLA’s and avoid periodic degradation.
  • The system must have the intelligence to decide how to best deliver content based on its popularity.
  • Be able to scale systems transparently to the application and with minimum management impact.
  • Eliminate the unnecessary sophistication and cost of traditional transactional storage and deliver the required performance at the right price.
  • Coordinate distributed systems to act as a federated system.

With these principles as the guiding backdrop, the development team created a solution tailored to the internet video market that is an integration of commodity hardware and unique software.

The Blackwave product suite consists of three models the 8120, 8150 and 8170 each increase in capacity and power and are positioned to meet local, regional distribution or global distribution applications.

Each platform contains the same basic modules.

  • Storage Server: This is a 4U rack that support 24x1TB SATA drives. An embedded quad processor supports the 10GE connectivity used as the backend system interconnect. Multiple storage servers can be stacked to a currently maximum supported capacity of 96TB.
  • Access Server: A 1U server (minimum of 2 per system) with dual quad-core processors delivering 10Gb/s Ethernet connections which enables connection to the internet and to the storage servers. Each access server is capable of serving 5Gb/s of Progressive HTTP downloads with additional Access Servers supporting Windows Media Streaming or Flash Media Streaming
  • Note, Storage and Access Servers can be added independently.
  • Interconnect: Two integrated switches. A 16 port, non-blocking switch for data and a 24 port switch for management and upload.

A fully configured 4x4 systems occupies 22U, has a capacity of 96TB, aggregated bandwidth of 20Gb/s and can support up to 30,000 users, depending on the bitrate requirements of the content.

Some of the interesting system innovations include:

  1. Inode and content data are not stored on the same physical disk. This technique eliminates or at least minimizes actuator thrash.
  2. System is sensitive to the video lifecycle and associated changing performance characteristics. What this means is that the system determines how to layout the data and dynamically adjusts as use patterns are established to deliver the required performance.
  3. Performance optimization must be achieved without administrator intervention.
  4. System based decisions include the placement of data on the disk with the content that requires the highest bandwidth performance located on the outer tracks of the platters.
  5. Throughput (content bitrate) requirements dictate the number of disk the data will be stripped across and resilience requirements dictate the number of replications.
  6. Bitrate throttling is applied to match the content delivery to the actual bitrate required. This granular provisioning of transport resource prevents unnecessary and inefficient allocation of bandwidth and increases the systems ability to service a large number of concurrent users. Even the most modest of over provisioning of resources can cause a starvation of a request.

While concurrency is a key attribute, predictability of service is equally important. When a request is received, the data management software determines if the request can be serviced by the available resource and without impacting any of the existing requests. If the answer is yes the request is serviced, if no, the request is rejected. This is in contrast to other systems where the willingness to respond to requests without understanding if sufficient resources are available can lead to a degradation in expected QoS. This is an example of the starvation issue referenced in point 6 above.

The internet video storage solution from Blackwave is tailored for the growing video distribution market. It is a clear illustration of how user sensitivity and an early identification of unique market requirements can drive the rapid development of a dedicated solution with the attributes needed to solve specific customer issues. However as noted earlier, the company is in an early formative stage and time will tell if their market execution matches their initial development success.

Online data storage is increasingly moving away from the old paradigm that a single basic architecture will satisfy all data types and satisfactorily meet all use cases. Storage, management and content delivery expectations and associated solutions are becoming more closely aligned with data types and usage patterns.

For the end user, if the cost of performance and an ability to predictably meet QoS expectations is a concern, then do not accept a solution that is a compromise. Look for a solution that is designed to meet your requirements, if not be prepared to accept a compromise in performance and a potential impact on the service you deliver.

A thought for my friends in the vendor community. If your solution does not address a real world issue that exhibits demonstrable customer pain then rethink your development plans. Do not expect your product to resonate with your customers if its value is not obvious and you have to convince them that there is a need.


Christopher said...

I work for a company that is starting to research two items to help in our business: Digital Asset Management and Content Distribution.

We create a lot of video and have no real organization for its storage and management. We are also looking to create a TV platform for it's distribution.

Any thoughts on top tier companies for either?

[email protected]