Nulladmin.com aitokenizer

Aitokenizer is a Nulladmin.com microservice which performs input tokenization for AI workloads. It supports various encodings used by OpenAI, Anthropic and others.
More information: https://nulladmin.com/aitokenizer/
Contact Info: contact@nulladmin.com
Version: 1.0.0
BasePath:/v1
Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0.html

Access

Methods

[ Jump to Models ]

Table of Contents

Cl100kBase

Claude

Default

Gpt2

O200kBase

P50kBase

P50kEdit

R50kBase

Cl100kBase

Up 
post /encode/cl100k_base
Upload text file for cl100k_base encoding (encodeCl100kBasePost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

Claude

Up 
post /encode/claude
Upload text file for claude encoding (encodeClaudePost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

Default

Up 
get /server/health
(serverHealthGet)

Return type

_server_health_get_200_response

Example data

Content-Type: application/json
{
  "health" : "Ready"
}

Example data

Content-Type: application/json
{
  "health" : "Unavailable"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK _server_health_get_200_response

502

Service Unavailable _server_health_get_502_response
Up 
get /server/version
(serverVersionGet)

Return type

_server_version_get_200_response

Example data

Content-Type: application/json
{
  "Revision" : "330dd98",
  "ServiceName" : "aitokenizer",
  "Version" : "1.0.0",
  "BuildDate" : "2024-03-24T01:00:43Z",
  "OSArch" : "linux/amd64",
  "Branch" : "main",
  "GoVersion" : "go1.21.0"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK _server_version_get_200_response

Gpt2

Up 
post /encode/gpt2
Upload text file for gpt2 encoding (encodeGpt2Post)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

O200kBase

Up 
post /encode/o200k_base
Upload text file for BPF encoding (encodeO200kBasePost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

P50kBase

Up 
post /encode/p50k_base
Upload text file for p50k_base encoding (encodeP50kBasePost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

P50kEdit

Up 
post /encode/p50k_edit
Upload text file for p50k_edit encoding (encodeP50kEditPost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

R50kBase

Up 
post /encode/r50k_base
Upload text file for r50k_base encoding (encodeR50kBasePost)

Consumes

This API call consumes the following media types via the Content-Type request header:

Form parameters

filename (required)
Form Parameter — The name of the file being uploaded default: null format: binary

Return type

String

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Example data

Content-Type: application/json
{
  "message" : "message"
}

Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

Responses

200

OK String

400

Bad Request _encode_o200k_base_post_400_response

403

Forbidden _encode_o200k_base_post_400_response

500

Internal Server Error _encode_o200k_base_post_400_response

Models

[ Jump to Methods ]

Table of Contents

  1. _encode_o200k_base_post_400_response -
  2. _server_health_get_200_response -
  3. _server_health_get_502_response -
  4. _server_version_get_200_response -

_encode_o200k_base_post_400_response - Up

message (optional)
String Error message

_server_health_get_200_response - Up

health (optional)
String Service status

_server_health_get_502_response - Up

health (optional)
String Service status

_server_version_get_200_response - Up

ServiceName (optional)
String Name of the service
Version (optional)
String Program version. Typically this is the git tag of the current commit, e.g. "1.0.0'.
Revision (optional)
String Build revision, e.g. the short git commit hash
Branch (optional)
String Git branch
BuildDate (optional)
String Build date
GoVersion (optional)
String Go version
OSArch (optional)
String Machine architecture