Topics

#devops #go #golang #grammar #parser #static-code-analysis #tree-sitter

Language breakdown

No language data available.

▸ Files click to expand

README

sibyl 2

The missing logical layer in codebases.

中文文档

Status

| Name | Badge | |----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Latest Version | GitHub release (latest by date) | | Unit Tests | | | Docker Image | | | Perf Tests | | | Code Coverage | | | Code Style | |

Overview

2023.08.03

>

At present, the implementation of function relationships is too rough.

If you want an accurate function call relationship diagram, you should prioritize using srctx.

>

This is because it is difficult for us to obtain an accurate call relationship without relying on compilation (or something like compilation).

In the future, the focus of Sibyl2 will be on extracting more factual information from the language rather than relationship information.

sibyl2 is a service for extracting, managing and offering metadata of your code in codebase.

Although Git is a widely used platform for version control and collaboration, it does not have the capability to analyze and interpret the logic of code.

Assuming my program now wants to know what is on line 35 of the extract_test.go file. If using GitLab, it is possible to find the corresponding text information, which may look like this:

func TestExtractString(t *testing.T) {
    fileResult, err := ExtractFromString(javaCodeForExtract, &ExtractConfig{
        LangType:    core.LangJava,
        ExtractType: extractor.TypeExtractFunction,
    })
    if err != nil {
        panic(err)
    }
    for _, each := range fileResult.Units {
        core.Log.Debugf("result: %s", each.GetDesc())
    }
}

However, my program does not know what this text represents. This is even more difficult for programs written in other languages (such as Java or Python).

With sibyl2 API, you can get:

{
  "repo_id": "sibyl2",
  "rev_hash": "e995ef44372a93394199ea837b1e2eed375a71a0",
  "path": "extract_test.go",
  "signature": "sibyl2||TestExtractString|*testing.T|",
  "tags": [],
  "func": {
    "name": "TestExtractString",
    "receiver": "",
    "namespace": "sibyl2",
    "parameters": [
      {
        "type": "*testing.T",
        "name": "t"
      }
    ],
    "returns": null,
    "span": {
      "start": {
        "row": {
          "$numberLong": "34"
        },
        "column": {
          "$numberLong": "0"
        }
      },
      "end": {
        "row": {
          "$numberLong": "45"
        },
        "column": {
          "$numberLong": "1"
        }
      }
    },
    "extras": {},
    "lang": "GOLANG"
  }
}

And even its relationships?

{
  "name": "TestExtractString",
  
  ...,
  
  "calls": [
    "object|*Function|GetDesc||string",
    "object|*Symbol|GetDesc||string",
    "sibyl2||ExtractFromString|string,ExtractConfig|extractor.FileResult,error",
    "object|*Call|GetDesc||string",
    "object|*Clazz|GetDesc||string"
  ],
  "reverseCalls": []
}

And even all the relationships?

Also class, module and something else. See Reference for details.

In addition, sibyl2's unified logic layer can also convert different language logics into the same data structure for storage, which is friendly to OLAP and other tools.

You can use various mature analysis tools (such as clickhouse, superset, metabase, etc.) to further deconstruct your code repository based on sibyl2. For example, you can view the distribution of methods with the @Test annotation throughout the repository.

Applying some global analysis:

Comparing the differences in methods, classes, and modules across each version:

In summary, sibyl2 aims to build a layer of common logic on top of your code repository in a simple way, helping businesses better understand, analyze, and use the data in their code repository.

Deployment

With Docker (recommended for start up)

We have provided an official compose file. Just:

copy and paste to your own docker-compose.yml file
docker-compose up

With an existed MongoDB

download our binaries from the release page.
add a sibyl-server-config.json file:

{
  "binding": {
    "dbtype": "MONGO",
    "mongodbname": "sibyl2",
    "mongouri": "mongodb+srv://<USERNAME>:<YOURPASSWORD>@XXXXXXXX.mongodb.net/test"
  }
}

./sibyl server

With nothing

It can also run with no middleware and database installed, if you just want to take a try.

Just ./sibyl server without config file.

Usage

Upload

./sibyl upload --src . --url http://127.0.0.1:9876

You can upload from different machines (just correct the url). Usually it only takes a few seconds.

Now everything is ready.

Access with Mongo URI

There are many mature visualization tools that support MONGO URI, such as official compass and metabase. With this, accessing all your data is easy.

Our docker-compose file includes metabase, allowing you to connect to MongoDB and start analyzing your data by simply opening 127.0.0.1:3000.

Currently, our data is divided into three collections:

fact_func: Function Information
fact_clazz: Class Information
rel_funcctx: Function Context Information

Access with sibyl2 clients

Of course, at the most time, developers access data programmatically. You can access all the data via different kinds of languages, to build your own tools:

For example, git diff with logical?

// assume that we have edited these lines
affectedFileMap := map[string][]int{
    "pkg/core/parser.go": {4, 89, 90, 91, 92, 93, 94, 95, 96},
    "pkg/core/unit.go":   {27, 28, 29},
}
for fileName, lineList := range affectedFileMap {     strLineList := make([]string, 0, len(lineList))     for _, each := range lineList {         strLineList = append(strLineList, strconv.Itoa(each))     }
affectedFunctions, _, err := apiClient.BasicQueryApi.         ApiV1FuncctxGet(ctx).         Repo(projectName).         Rev(head.Hash().String()).         File(fileName).         Lines(strings.Join(strLineList, ",")).         Execute() 	     for _, eachFunc := range affectedFunctions {         // get all the calls details?         for _, eachCall := range eachFunc.Calls {             detail, _, err := apiClient.SignatureQueryApi.                 ApiV1SignatureFuncGet(ctx).                 Repo(projectName).                 Rev(head.Hash().String()).                 Signature(eachCall).                 Execute()             assert.Nil(t, err)             core.Log.Infof("call: %v", detail)         }     } }

| Language | Link | |------------|------------------------------------------------------| | Golang | https://github.com/opensibyl/sibyl-go-client | | Java | https://github.com/opensibyl/sibyl-java-client | | JavaScript | https://github.com/opensibyl/sibyl-javascript-client |

See more examples about how to use for details.

Languages support

| Languages | Function | Function Context | Class | |------------|----------|------------------|-------| | Golang | Yes | Yes | Yes | | Java | Yes | Yes | Yes | | Python | Yes | Yes | Yes | | Kotlin | Yes | Yes | Yes | | JavaScript | Yes | Yes | Yes |

Based on tree-sitter, it's very easy to add an extra language support.

Performance

We have tested it on some famous repos, like guava. And that's why we can say it is " fast enough".

See https://github.com/williamfzc/sibyl2/actions/workflows/perf.yml for details.

| Language | Repo | Cost | |----------|----------------------------------------------------|------| | Golang | https://github.com/gin-gonic/gin.git | ~1s | | Java | https://github.com/spring-projects/spring-boot.git | ~50s | | Python | https://github.com/psf/requests | ~1s | | Kotlin | https://github.com/square/okhttp | ~1s |

Contribution

This project split into 3 main parts:

/cmd: Pure command line tool for general usage
/pkg/server: All-in-one service for production
others: Shared api and core

Workflow:

core: collect files and convert them to Unit.
extract: classify and process units into functions, symbols.
api: higher level analyze like callgraph

Issues / PRs are welcome!

Further work

More effective and stable uploader
More languages support
More fact level data we can collect
Provide a standard data layer for AI models

About its role in DevOps, see About This Project: Code Snapshot Layer In DevOps for details.

References

Inspired by semantic of GitHub.

basic grammar: https://tree-sitter.github.io/tree-sitter/creating-parsers#the-grammar-dsl
language parser (for example, golang): https://github.com/tree-sitter/tree-sitter-go/blob/master/src/parser.c
symbol: https://github.com/github/semantic/blob/main/docs/examples.md#symbols
stack graphs: https://github.blog/2021-12-09-introducing-stack-graphs/

License

Apache License Version 2.0, see LICENSE

🔗 More in this category