Forward nginx access logs to Spinlink using Vector for AI crawler analytics.
Vector is a lightweight log collector that forwards your nginx logs to Spinlink.
Using the install script (recommended):
curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash
Ubuntu / Debian:
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo bash sudo apt-get install -y vector
Amazon Linux / RHEL / CentOS:
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.rpm.sh' | sudo bash sudo yum install -y vector
Verify installation:
vector --version
Spinlink requires nginx logs in JSON format. Add the following to your nginx.conf inside the http block:
log_format spinlink_json escape=json '{'
'"time":"$time_iso8601",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"status":$status,'
'"body_bytes_sent":$body_bytes_sent,'
'"request_time":$request_time,'
'"http_referrer":"$http_referer",'
'"http_user_agent":"$http_user_agent",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"request_method":"$request_method",'
'"request_uri":"$request_uri",'
'"query_string":"$query_string",'
'"host":"$host"'
'}';
access_log /var/log/nginx/access.json spinlink_json;Test and reload nginx:
sudo nginx -t && sudo systemctl reload nginx
Create the Vector configuration at /etc/vector/vector.toml:
# Spinlink Nginx Log Collector
[sources.nginx_access_logs]
type = "file"
include = ["/var/log/nginx/access.json"]
read_from = "end"
[sources.nginx_error_logs]
type = "file"
include = ["/var/log/nginx/error.log"]
read_from = "end"
[transforms.parse_access_logs]
type = "remap"
inputs = ["nginx_access_logs"]
drop_on_error = true
source = '''
parsed, err = parse_json(.message)
if err != null {
log("Failed to parse JSON: " + to_string(err), level: "warn")
abort
}
. = parsed
.log_type = "access"
'''
[transforms.parse_error_logs]
type = "remap"
inputs = ["nginx_error_logs"]
source = '''
.log_type = "error"
.raw_message = .message
'''
[sinks.spinlink]
type = "http"
inputs = ["parse_access_logs", "parse_error_logs"]
uri = "https://api.spinlink.io/api/ingestion/logs/nginx"
encoding.codec = "json"
method = "post"
[sinks.spinlink.request.headers]
X-API-Key = "YOUR_API_KEY_HERE"
Content-Type = "application/json"Replace YOUR_API_KEY_HERE with your Spinlink API key.
sudo systemctl enable vector sudo systemctl start vector sudo systemctl status vector
You should see Active: active (running) in the output.
View Vector logs to confirm it's processing traffic:
sudo journalctl -u vector -f
Generate test traffic and check for errors:
curl http://localhost sudo journalctl -u vector | grep -i error
If you see 401 Unauthorized, verify your API key. If you see 404 Not Found, check the endpoint URL.
Open PowerShell as Administrator:
Invoke-WebRequest https://packages.timber.io/vector/0.53.0/vector-x64.msi -OutFile vector-0.53.0-x64.msi msiexec /i vector-0.53.0-x64.msi
Add the same JSON log format to your nginx.conf (typically C:\nginx\conf\nginx.conf):
access_log C:/nginx/logs/access.json spinlink_json;
Create C:\Program Files\Vector\config\vector.yaml:
sources:
nginx_access_logs:
type: file
include:
- C:\nginx\logs\access.json
read_from: end
transforms:
parse_access_logs:
type: remap
inputs:
- nginx_access_logs
drop_on_error: true
source: |
parsed, err = parse_json(.message)
if err != null {
abort
}
. = parsed
.log_type = "access"
sinks:
spinlink:
type: http
inputs:
- parse_access_logs
uri: https://api.spinlink.io/api/ingestion/logs/nginx
encoding:
codec: json
method: post
request:
headers:
X-API-Key: YOUR_API_KEY_HERE
Content-Type: application/jsonUse NSSM to install Vector as a service:
# Download NSSM Invoke-WebRequest https://nssm.cc/release/nssm-2.24.zip -OutFile nssm.zip Expand-Archive nssm.zip -DestinationPath C:\nssm # Install and start service C:\nssm\nssm-2.24\win64\nssm.exe install Vector "C:\Program Files\Vector\bin\vector.exe" "--config" "C:\Program Files\Vector\config\vector.yaml" Start-Service Vector
vector validate /etc/vector/vector.toml sudo journalctl -u vector --no-pager | tail -50
sudo usermod -a -G adm vector sudo systemctl restart vector
# Verify nginx is writing JSON logs tail -5 /var/log/nginx/access.json # Check Vector is processing logs sudo journalctl -u vector | grep "nginx_access_logs" # Check for HTTP errors sudo journalctl -u vector | grep -E "(ERROR|401|404|500)"
# Install script (recommended) curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash -s -- -y # Ubuntu/Debian sudo apt-get update && sudo apt-get upgrade vector # Amazon Linux / RHEL sudo yum update vector
sudo systemctl stop vector sudo systemctl disable vector sudo apt-get remove vector # Debian/Ubuntu sudo yum remove vector # RHEL/Amazon Linux sudo rm /etc/vector/vector.toml
For information, see the Security & Compliance documentation.
Need help? Contact support