a simplified machine learning container platform that helps teams get started with an automated workflow
dama
A simplified machine learning container platform that helps teams get started with an automated workflow.
DISCLAIMER: dama is currently in alpha due to the lack of security and scaling, but still fun to try out!
Server Configuration
Server default configurations in config.yml These configurations are loaded by default if not overridden inconfig.yml.
expire: "1300" deployexpire: "86400" uploadsize: 2000000000 envsize: 20 https: listen: "0.0.0.0" port: "8443" debug: false verifytls: false db: db: 0 maxretries: 20 docker: endpoint: "unix:///var/run/docker.sock" cpushares: 512 memory: 1073741824 gotty: tls: false
These configurations need to be set in your environment variables.
# Server admin username and password DamaUser # example: DamaUser="tim" DamaPassword # example: DamaPassword="9e9692478ca848a19feb8e24e5506ec89"
# Redis database password if applicable DBPassword # example: DBPassword="9e9692478ca848a19feb8e24e5506ec89"
All configurations types
images: ["perlogix:minimal"] # required / string array expire: "1300" # string deployexpire: "86400" # string uploadsize: 2000000000 # int envsize: 20 # int https: listen: "0.0.0.0" # string port: "8443" # string pem: "/opt/dama.pem" # required / string key: "/opt/dama.key" # required / string debug: false # bool verifytls: false # bool db: network: "unix" # required / string address: "./tmp/redis.sock" # required / string db: 0 # int maxretries: 20 # int docker: endpoint: "unix:///var/run/docker.sock" # string cpushares: 512 # int memory: 1073741824 # int gotty: tls: false # bool
CLI Configuration
These environment variables need to be exported in order to use dama-cli.DAMASERVER # example: export DAMASERVER="https://localhost:8443/" DAMAUSER # example: export DAMAUSER="tim" DAMAKEY # example: export DAMAKEY="9e9692478ca848a19feb8e24e5506ec89"
CLI Flags
Usage: dama [options]-new Create a new environment from scratch and delete the old one -run Create environment and run with dama.yml -file Run with dama.yml in different directory -env Create an environment variable or secret for runtime -img Specify a docker image to be used instead of the default image -dl Download file from workspace in your environment to your local computer -up Upload files from your local computer to workspace in your environment -deploy Deploy API and get your unique URI -show-api Show API details: URL, Health and Type -show-images Show images available to use
CLI Examples
dama -new dama -run dama -run -file ../dama.yml dama -env "AWSACCESSKEYID=123,AWSSECRETACCESSKEY=234" dama -deploy dama -run -img tensorflow:lite dama -show-images dama -show-api dama -up data.csv dama -dl model.pkldama.yml File
This a simpledama.yml to setup your environment and run a Flask API.
image: "perlogix:minimal" port: "5000" pip: | Flask==0.12.2 scikit-learn==0.19.1 numpy==1.14.2 scipy==1.0.0 python: | from flask import Flask, request, jsonify from sklearn import datasets from sklearn.modelselection import traintest_split from sklearn.ensemble import RandomForestClassifier from sklearn.externals import joblib
X, y = datasets.loadiris(returnX_y=True) Xtrain, Xtest, ytrain, ytest = traintestsplit(X, y, testsize=0.3, randomstate=42) model = RandomForestClassifier(random_state=101) model.fit(Xtrain, ytrain) print("Score on the training set is: {:2}".format(model.score(Xtrain, ytrain))) print("Score on the test set is: {:.2}".format(model.score(Xtest, ytest))) model_filename = 'iris-rf-v1.0.pkl' print("Saving model to {}...".format(model_filename)) joblib.dump(model, model_filename) app = Flask(name) MODEL = joblib.load('iris-rf-v1.0.pkl') MODEL_LABELS = ['setosa', 'versicolor', 'virginica']
@app.route('/predict') def predict(): sepallength = request.args.get('sepallength') sepalwidth = request.args.get('sepalwidth') petallength = request.args.get('petallength') petalwidth = request.args.get('petalwidth') features = [[sepallength, sepalwidth, petallength, petalwidth]] label_index = MODEL.predict(features) label = MODELLABELS[labelindex[0]] return jsonify(status='complete', label=label) if name == 'main': app.run(debug=False, host="0.0.0.0", threaded=True)
cURL API in sandbox or deploy
curl -ks https://localhost:8443/api/
Even simpler environment setup with model training.
image: "perlogix:tensorflow" checkout: "https://github.com/aymericdamien/TensorFlow-Examples.git" cmd: | cd TensorFlow-Examples/examples/3_NeuralNetworks python neural_network.py
All YAML configuration option types.
project # string - proejct name env # string array - env variables checkout # string - git checkout master branch time_format # string - python time format used in container as env variable TIMESTAMP setup_cmd # string - run setup /initial command before cmd or python cmd # string - run BASH Linux command python # string - run inline Python pip # string - install pip packages image # string - define container image for environment port # string - port to expose for web service git: url # string - git URL branch # string - git branch sha # string - git SHA aws_s3: file # string - file to push or pull dir # string - directory to push or pull bucket_push # string - push file or dir to S3 bucket_pull # string - pull file or dir from S3
Dockerfiles
Add these lines to your Dockerfiles for your CLI to connect via websocketsRUN cd /usr/bin && curl -L https://github.com/yudai/gotty/releases/download/v1.0.1/gottylinuxamd64.tar.gz | tar -xz CMD ["/usr/bin/gotty", "--reconnect", "-w", "/bin/bash"]
Build
make build
To Do
- [ ] Tokenize environment variables in DB - [ ] Write test suite - [ ] Provide Vagrant and Docker images - [ ] Add scheduler / resource manager for multi-host container serving - [ ] Rewrite auth middleware - [ ] Swap out stdlib flags package for third-party package - [ ] These docs stink!
