An optimal algorithm for the distinct elements problem

DM Kane, J Nelson, DP Woodruff - Proceedings of the twenty-ninth ACM …, 2010 - dl.acm.org
Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on …, 2010dl.acm.org
We give the first optimal algorithm for estimating the number of distinct elements in a data
stream, closing a long line of theoretical research on this problem begun by Flajolet and
Martin in their seminal paper in FOCS 1983. This problem has applications to query
optimization, Internet routing, network topology, and data mining. For a stream of indices in
{1,..., n}, our algorithm computes a (1±ε)-approximation using an optimal O (1/ε-2+ log (n))
bits of space with 2/3 success probability, where 0< ε< 1 is given. This probability can be …
We give the first optimal algorithm for estimating the number of distinct elements in a data stream, closing a long line of theoretical research on this problem begun by Flajolet and Martin in their seminal paper in FOCS 1983. This problem has applications to query optimization, Internet routing, network topology, and data mining. For a stream of indices in {1,...,n}, our algorithm computes a (1 ± ε)-approximation using an optimal O(1/ε-2 + log(n)) bits of space with 2/3 success probability, where 0<ε<1 is given. This probability can be amplified by independent repetition. Furthermore, our algorithm processes each stream update in O(1) worst-case time, and can report an estimate at any point midstream in O(1) worst-case time, thus settling both the space and time complexities simultaneously.
We also give an algorithm to estimate the Hamming norm of a stream, a generalization of the number of distinct elements, which is useful in data cleaning, packet tracing, and database auditing. Our algorithm uses nearly optimal space, and has optimal O(1) update and reporting times.
ACM Digital Library
Showing the best result for this search. See all results