QAware | Blog: Apache Solr as a compressed, scalable, and high performance time series database
68.000.000.000 time correlated data objects: How to store such amount of data on your laptop computer and retrieve any point within a few milliseconds? We answered that question at FOSDEM 2015 Conference in Brussels.A relational database management system (RDBMS) like Oracle, MySQL or Microsoft SQL Server and a normalized data-schema does not work well on 68 Billion data objects in a time series. They have some unacceptable drawbacks for us
- long import duration,
- slow query and retrieval of data objects,
- high amount of hard dive space and
- are limited in scalability due to RDBMS.
There are open source time series databases available, including InfluxDB, OpenTSBD, RDDTool or SciDB and many more but neither of them fully complies to our major requirements.
- fast imports and queries
- storing arbitrary metadata on time series as well on data objects
- minimal hard drive space
- everything should run on a laptop computer without performance drawbacks
We decided to create our own solution instead of using a solution that only complies to 50 percent of our requirements. We realized how easy it is, to build a perfect matching solution when choosing the right tools.
Read full article from QAware | Blog: Apache Solr as a compressed, scalable, and high performance time series database
No comments:
Post a Comment