Surprisingly space efficient trie in Golang(11 bits/key; 100 ns/get).
Slim - surprisingly space efficient data types in Golang
Slim is collection of surprisingly space efficient data types, with corresponding serialization APIs to persisting them on-disk or for transport.
- 1. Index on-disk key-values - 2. Sparse index - 3. Range scan - Versions - Protobuf data structures - Regenerating protobuf code
Why slim
As data on internet keeps increasing exponentially, the capacity gap between memory and disk becomes greater.
Most of the time, a data itself does not need to be loaded into expensive main memory. Only the much more important information, WHERE-A-DATA-IS, deserve a seat in main memory.
This is what slim does, keeps as little information as possible in main memory, as a minimized index of huge amount external data.
-
SlimIndex: is a common index structure, building on top ofSlimTrie.
-
SlimTrieis the underlying index data structure, evolved from [trie][].
Features:
- Minimized: 11 bits per key(far less than an 64-bits pointer!!).
- Stable: memory consumption is stable in various scenarios. The Worst case converges to average consumption tightly. See benchmark.
- Loooong keys: You can have VERY long keys(16K bytes), without any waste of memory(and money). Do not waste your life writing another prefix compression:). ([aws-s3][] limits key length to 1024 bytes). Memory consumption only relates to key count, not to key length.
- Ordered: like [btree][], keys are stored. Range-scan will be ready in 0.6.0.
- Fast: ~150 ns per Get(). Time complexity for a get is O(log(n) + k); n: key count; k: key length.
- Ready for transport: a single proto.Marshal() is all it requires to serialize, transport or persisting on disk etc.
Performance and memory overhead
- 3.3 times faster than the [btree][].
- 2.3 times faster than binary search.
- Memory overhead is about 11 bit per key.
The data struct in this benchmark is a slice of key-value pairs with a SlimTrie serving as the index. The slim itself is built in the filter mode, to maximize memory reduction and performance. The whole struct slimKV is a fully functional kv-store, just like a static btree.
type slimKV struct {
slim *trie.SlimTrie
Elts []*KVElt
}
type KVElt struct {
Key string
Val int32
}You can find the benchmark code in benchmark;
Read more about Performance
Synopsis
1. Index on-disk key-values
One of the typical usages of slim is to index serialized data on disk(e.g., key value records in a SSTable). By keeping a slim in memory, one can quickly find the on-disk offset of the record by a key.
Show me the code ......
package index_test
import ( "fmt" "strings"
"github.com/openacid/slim/index" )
type Data string
func (d Data) Read(offset int64, key string) (string, bool) { kv := strings.Split(string(d)[offset:], ",")[0:2] if kv[0] == key { return kv[1], true } return "", false }
func Example() {
// Accelerate external data accessing (in memory or on disk) by indexing // them with a SlimTrie:
// data is a sample of some unindexed data. In our example it is a comma // separated key value series. // // In order to let SlimTrie be able to read data, data should have // a Read method: // Read(offset int64, key string) (string, bool) data := Data("Aaron,1,Agatha,1,Al,2,Albert,3,Alexander,5,Alison,8")
// keyOffsets is a prebuilt index that stores key and its offset in data accordingly. keyOffsets := []index.OffsetIndexItem{ {Key: "Aaron", Offset: 0}, {Key: "Agatha", Offset: 8}, {Key: "Al", Offset: 17}, {Key: "Albert", Offset: 22}, {Key: "Alexander", Offset: 31}, {Key: "Alison", Offset: 43}, }
// SlimIndex is simply a container of SlimTrie and its data. st, err := index.NewSlimIndex(keyOffsets, data) if err != nil { fmt.Println(err) }
// Lookup v, found := st.Get("Alison") fmt.Printf("key: %q\n found: %t\n value: %q\n", "Alison", found, v)
v, found = st.Get("foo") fmt.Printf("key: %q\n found: %t\n value: %q\n", "foo", found, v)
// Output: // key: "Alison" // found: true // value: "8" // key: "foo" // found: false // value: "" }
2. Sparse index
Create an index item for every 4(or more as you wish) keys.
Let several adjacent keys share one index item reduces a lot memory cost if there are huge amount keys in external data. Such as to index billions of 4KB objects on a 4TB disk(because one disk IO costs 20ms for either reading 4KB or reading 1MB).
Show me the code ......
package index_test
import ( "fmt" "strings"
"github.com/openacid/slim/index" )
type RangeData string
func (d RangeData) Read(offset int64, key string) (string, bool) { for i := 0; i < 4; i++ { if int(offset) >= len(d) { break }
kv := strings.Split(string(d)[offset:], ",")[0:2] if kv[0] == key { return kv[1], true } offset += int64(len(kv[0]) + len(kv[1]) + 2)
} return "", false }
func Example_indexRanges() {
// Index ranges instead of keys: // In this example at most 4 keys shares one index item.
data := RangeData("Aaron,1,Agatha,1,Al,2,Albert,3,Alexander,5,Alison,8")
// keyOffsets is a prebuilt index that stores range start, range end and its offset. keyOffsets := []index.OffsetIndexItem{ // Aaron +--> 0 // Agatha | // Al | // Albert |
// Alexander +--> 31 // Alison |
{Key: "Aaron", Offset: 0}, {Key: "Agatha", Offset: 0}, {Key: "Al", Offset: 0}, {Key: "Albert", Offset: 0},
{Key: "Alexander", Offset: 31}, {Key: "Alison", Offset: 31}, }
st, err := index.NewSlimIndex(keyOffsets, data) if err != nil { panic(err) }
v, found := st.RangeGet("Aaron") fmt.Printf("key: %q\n found: %t\n value: %q\n", "Aaron", found, v)
v, found = st.RangeGet("Al") fmt.Printf("key: %q\n found: %t\n value: %q\n", "Al", found, v)
v, found = st.RangeGet("foo") fmt.Printf("key: %q\n found: %t\n value: %q\n", "foo", found, v)
// Output: // key: "Aaron" // found: true // value: "1" // key: "Al" // found: true // value: "2" // key: "foo" // found: false // value: "" }
3. Range scan
Slim can also be used as a traditional in-memory kv-store: Building a slim with Opt{ Complete: Bool(true) }, it won't strip out any information(e.g., it won't eliminate single-branch labels) and it will functions the same as a btree. This snippet shows how to iterate key values.
Show me the code ......
package triebloom filterimport ( "fmt"
"github.com/openacid/slim/encode" )
func ExampleSlimTrie_ScanFrom() { var keys = []string{ "", "
", "a", "ab", "abc", "abca", "abcd", "abcd1", "abce", "be", "c", "cde0", "d", } values := makeI32s(len(keys))codec := encode.I32{} st, _ := NewSlimTrie(codec, keys, values, Opt{ Complete: Bool(true), })
// untilD stops when encountering "d". untilD := func(k, v []byte) bool { if string(k) == "d" { return false }
_, i32 := codec.Decode(v) fmt.Println(string(k), i32) return true }
fmt.Println("scan (ab, +∞):") st.ScanFrom("ab", false, true, untilD)
fmt.Println() fmt.Println("scan be, +∞):") st.ScanFrom("be", true, true, untilD)
fmt.Println() fmt.Println("scan (ab, be):") st.ScanFromTo( "ab", false, "be", false, true, untilD)
// Output: // // scan (ab, +∞): // abc 4 // abca 5 // abcd 6 // abcd1 7 // abce 8 // be 9 // c 10 // cde0 11 // // scan [be, +∞): // be 9 // c 10 // cde0 11 // // scan (ab, be): // abc 4 // abca 5 // abcd 6 // abcd1 7 // abce 8 }</code></pre>
</details>
Filter mode and KV mode.
Slim can be built into either a filter(like
but with key order preserved.) or a real kv-store(likebtree) There is anoptioninNewSlimTrie(..., option)to control the building behavior. Ref: [OptComplete
- To use slim as a kv-store, set the option to
then there won't be false positives.InnerPrefix
- To use it as a filter, set
,LeafPrefixto false(CompleteimpliesInnerPrefix==trueandLeafPrefix==true). Then slim won't store any single branch label in the trie it builds.InnerPrefix==trueWith
, it does not reduce a single label branch that leads to an inner node.LeafPrefix==trueWith
, it does not reduce a single label branch that leads to a leaf node.-c-> 2 -x-> 3 -y-> $E.g.:
<pre><code class="lang-">// Complete InnerPrefix: true LeafPrefix: true ^ -a-> 1 -b-> $
-z-> $ InnerPrefix: true LeafPrefix: false ^ -a-> $-c-> 2 -x-> 3 -y-> $-z-> $ InnerPrefix: false LeafPrefix: true ^ -a-> 1 -b-> $-c-> 3 -y-> $-z-> $ InnerPrefix: false LeafPrefix: false ^ -a-> $-c-> 3 -y-> $-z-> $</code></pre>yThe memory consumption in filter mode and kv mode differs significantly. The following chart shows memory consumption by 1 million var-length string, 10 to 20 byte in different mode:
| - | size | gzip-size | | :-- | --: | --: | | sample data size | 15.0M | 14.0M | | Complete:true | 14.0M | 10.0M | | InnerPrefix:ture | 1.3M | 0.9M | | all false | 1.3M | 0.8M |
<!-- ## FAQ -->
Try it
Install
<pre><code class="lang-sh">go get github.com/openacid/slim/trie</code></pre>
Change-log: Change-log
Versions
A newer version
being compatible with an older versionxmeansycan load data serialized byx. Butxshould never try to load data serialized by a newer versiony.v0.5.is compatible with0.2.,0.3.,0.4.,0.5.*.v0.4.is compatible with0.2.,0.3.,0.4..v0.3.is compatible with0.2.,0.3.*.v0.2.is compatible with0.2...proto<!-- TODO add FAQ -->
Who are using slim
<span> <span> ![][baishancloud-favicon] </span> <span> [baishancloud][] </span> </span>
Slim internal
Protobuf data structures
Slim uses [protobuf][] to define its on-disk data structures and as its serialization engine. All
files use proto3 syntax.array/bitmap.protoThe protobuf definitions are in the following files:
:Bits– a bitmap with rank index, used as the building block for sparse arrays.array/array.proto
:Array32– a 32-bit sparse array backed by bitmaps and offset tables.trie/slim.proto
:Bitmap,VLenArray, andSlim– the core trie structures.Slimstores node-type bitmaps, inner-node label bitmaps, short-bitmap tables, inner/leaf prefixes, and serialized leaf values.proto.Marshal()These structures are serialized with
and deserialized withproto.Unmarshal()from the [github.com/golang/protobuf][protobuf-go] package (v1.3.1).*.pb.goRegenerating protobuf code
The generated Go files (
) should not be edited by hand. To regenerate them after modifying a.protofile:protoc
- Install
(the Protocol Buffers compiler). See [protoc installation][protoc-install].trie/slim.proto<pre><code class="lang-sh">go install github.com/golang/protobuf/protoc-gen-go@latest</code></pre>
- Install the Go protobuf plugin:
<pre><code class="lang-sh"># array package (has a go:generate directive in array/gen.go) go generate ./array/...
- Re-generate the Go source:
# trie package cd trie && protoc --protopath=. --goout=. slim.proto</code></pre>
> Note:
was originally built withprotoc-gen-go> v1.2.0. When regenerating, make sure the generated code is compatible > with the module dependencygithub.com/golang/protobuf v1.3.1.slimFeedback and contributions
Feedback and Contributions are greatly appreciated.
At this stage, the maintainers are most interested in feedback centered on:
- Do you have a real life scenario that
supports well, or doesn't support at all?pseudo-gopathDo any of the APIs fulfill your needs well? Let us know by filing an issue, describing what you did or wanted to do, what you expected to happen, and what actually happened:Or other type of [issue][new-issue].
- [bug-report][]
- [improve-document][]
- [feature-request][]
<!-- ## Contributing --> <!-- The maintainers actively manage the issues list, and try to highlight issues --> <!-- suitable for newcomers. -->
<!-- [> TODO dep CONTRIBUTING <] --> <!-- The project follows the typical GitHub pull request model. See CONTRIBUTING.md for more details. -->
<!-- Before starting any work, please either comment on an existing issue, --> <!-- or file a new one. -->
<!-- [> TODO <] --> <!-- Please read [CONTRIBUTING.md][] --> <!-- for details on our code of conduct, and the process for submitting pull requests to us. --> <!-- https://gist.github.com/PurpleBooth/b24679402957c63ec426 -->
<!-- ### Code style -->
<!-- ### Tool chain -->
<!-- ### Customized install -->
<!-- Alternatively, if you have a customized go develop environment, you could also --> <!-- clone it: -->
<!-- <pre><code class="lang-sh">--> <!-- git clone git@github.com:openacid/slim.git --> <!--</code></pre> -->
<!-- As a final step you'd like have a test to see if everything goes well: -->
<!-- <pre><code class="lang-sh">--> <!-- cd path/to/slim/build/pseudo-gopath --> <!-- export GOPATH=$(pwd) --> <!-- go test github.com/openacid/slim/array --> <!--</code></pre> -->
<!-- Another reason to have a
in it is that some tool have their --> <!-- own way conducting source code tree. --> <!-- E.g. git-worktree --> <!-- checkouts source code into another dir other than the GOPATH work space. -->vendor/<!-- ## Update dependency -->
<!-- Dependencies are tracked by dep. --> <!-- All dependencies are kept in
dir thus you do not need to do anything --> <!-- to run it. -->dep<!-- You need to update dependency only when you bring in new feature with other dependency. -->
<!-- - Install
-->dep init<!-- <pre><code class="lang-">--> <!-- curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh --> <!--</code></pre> -->
<!-- - Download dependency -->
<!-- <pre><code class="lang-">--> <!-- dep ensure --> <!--</code></pre> -->
<!-- > dep uses Gopkg.toml Gopkg.lock to track dependency info. --> <!-- > --> <!-- > Gopkg.toml Gopkg.lock is created with
. --> <!-- > --> <!-- > dep creates avendor` dir to have all dependency package there. -->Authors
- ![][刘保海-img-sml] [刘保海][] marshaling
- ![][吴义谱-img-sml] [吴义谱][] array
- ![][张炎泼-img-sml] [张炎泼][] slimtrie design
- ![][李文博-img-sml] [李文博][] trie-compressing, trie-search
- ![][李树龙-img-sml] [李树龙][] marshaling
See also the list of [contributors][] who participated in this project.
License
This project is licensed under the MIT License - see the LICENSE file for details.
[刘保海]: https://github.com/liubaohai [吴义谱]: https://github.com/pengsven [张炎泼]: https://github.com/drmingdrmer [李文博]: https://github.com/wenbobuaa [李树龙]: https://github.com/lishulong
[刘保海-img-sml]: https://avatars1.githubusercontent.com/u/26271283?s=36&v=4 [吴义谱-img-sml]: https://avatars3.githubusercontent.com/u/6927668?s=36&v=4 [张炎泼-img-sml]: https://avatars3.githubusercontent.com/u/44069?s=36&v=4 [李文博-img-sml]: https://avatars1.githubusercontent.com/u/11748387?s=36&v=4 [李树龙-img-sml]: https://avatars2.githubusercontent.com/u/13903162?s=36&v=4
[contributors]: https://github.com/openacid/slim/contributors
[dep]: https://github.com/golang/dep [protobuf]: https://github.com/protocolbuffers/protobuf [protobuf-go]: https://github.com/golang/protobuf [semver]: http://semver.org/
[protoc-install]: http://google.github.io/proto-lens/installing-protoc.html [dep-install]: https://github.com/golang/dep#installation
[CONTRIBUTING.md]: CONTRIBUTING.md
[baishancloud]: http://www.baishancdnx.com [baishancloud-favicon]: http://www.baishancdnx.com/public/favicon.ico [golang-standards-project-layout]: https://github.com/golang-standards/project-layout
[bug-report]: https://github.com/openacid/slim/issues/new?labels=bug&template=bug_report.md [improve-document]: https://github.com/openacid/slim/issues/new?labels=doc&template=doc_improve.md [feature-request]: https://github.com/openacid/slim/issues/new?labels=feature&template=feature_request.md
[new-issue]: https://github.com/openacid/slim/issues/new/choose
[benchmark-get-png]: docs/trie/charts/benchget20190603.png
[trie]: https://en.wikipedia.org/wiki/Trie [btree]: https://github.com/google/btree [aws-s3]: https://aws.amazon.com/s3/ [red-black-tree]: https://en.wikipedia.org/wiki/Red%E2%80%93black_tree
