Refer to the information below when troubleshooting Gitaly and Gitaly Cluster.
Troubleshoot Gitaly
The following sections provide possible solutions to Gitaly errors.
See also Gitaly timeout settings,and our advice on parsing the gitaly/current file.
Check versions when using standalone Gitaly servers
When using standalone Gitaly servers, you must make sure they are the same versionas GitLab to ensure full compatibility:
- On the left sidebar, at the bottom, select Admin Area.
- Select Overview > Gitaly Servers.
- Confirm all Gitaly servers indicate that they are up to date.
Find storage resource details
You can run the following commands in a Rails consoleto determine the available and used space on a Gitaly storage:
Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics# For Gitaly ClusterGitlab::GitalyClient::ServerService.new("<storage name>").disk_statistics
Use gitaly-debug
The gitaly-debug
command provides "production debugging" tools for Gitaly and Gitperformance. It is intended to help production engineers and supportengineers investigate Gitaly performance problems.
To see the help page of gitaly-debug
for a list of supported sub-commands, run:
gitaly-debug -h
Commits, pushes, and clones return a 401
remote: GitLab: 401 Unauthorized
You need to sync your gitlab-secrets.json
file with your GitLabapplication nodes.
500 and fetching folder content
errors on repository pages
Fetching folder content
, and in some cases 500
, errors indicateconnectivity problems between GitLab and Gitaly.Consult the client-side gRPC logsfor details.
Client side gRPC logs
Gitaly uses the gRPC RPC framework. The Ruby gRPCclient has its own log file which may contain helpful information whenyou are seeing Gitaly errors. You can control the log level of thegRPC client with the GRPC_LOG_LEVEL
environment variable. Thedefault level is WARN
.
You can run a gRPC trace with:
sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check
If this command fails with a failed to connect to all addresses
error,check for an SSL or TLS problem:
/opt/gitlab/embedded/bin/openssl s_client -connect <gitaly-ipaddress>:<port> -verify_return_error
Check whether Verify return code
field indicates aknown Linux package installation configuration problem.
If openssl
succeeds but gitlab-rake gitlab:gitaly:check
fails,check certificate requirements for Gitaly.
Server side gRPC logs
gRPC tracing can also be enabled in Gitaly itself with the GODEBUG=http2debug
environment variable. To set this in a Linux package installation:
Add the following to your
gitlab.rb
file:gitaly['env'] = { "GODEBUG=http2debug" => "2"}
Reconfigure GitLab.
Correlating Git processes with RPCs
Sometimes you need to find out which Gitaly RPC created a particular Git process.
One method for doing this is by using DEBUG
logging. However, this needs to be enabledahead of time and the logs produced are quite verbose.
A lightweight method for doing this correlation is by inspecting the environmentof the Git process (using its PID
) and looking at the CORRELATION_ID
variable:
PID=<Git process ID>sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID=
This method isn't reliable for git cat-file
processes, because Gitalyinternally pools and re-uses those across RPCs.
Repository changes fail with a 401 Unauthorized
error
If you run Gitaly on its own server and notice these conditions:
- Users can successfully clone and fetch repositories by using both SSH and HTTPS.
- Users can't push to repositories, or receive a
401 Unauthorized
message when attempting tomake changes to them in the web UI.
Gitaly may be failing to authenticate with the Gitaly client because it has thewrong secrets file.
Confirm the following are all true:
When any user performs a
git push
to any repository on this Gitaly server, itfails with a401 Unauthorized
error:remote: GitLab: 401 UnauthorizedTo <REMOTE_URL>! [remote rejected] branch-name -> branch-name (pre-receive hook declined)error: failed to push some refs to '<REMOTE_URL>'
When any user adds or modifies a file from the repository using the GitLabUI, it immediately fails with a red
401 Unauthorized
banner.Creating a new project and initializing it with a READMEsuccessfully creates the project but doesn't create the README.
When tailing the logson a Gitaly client and reproducing the error, you get
401
errorswhen reaching the /api/v4/internal/allowed endpoint:# api_json.log{ "time": "2019-07-18T00:30:14.967Z", "severity": "INFO", "duration": 0.57, "db": 0, "view": 0.57, "status": 401, "method": "POST", "path": "\/api\/v4\/internal\/allowed", "params": [ { "key": "action", "value": "git-receive-pack" }, { "key": "changes", "value": "REDACTED" }, { "key": "gl_repository", "value": "REDACTED" }, { "key": "project", "value": "\/path\/to\/project.git" }, { "key": "protocol", "value": "web" }, { "key": "env", "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" }, { "key": "user_id", "value": "2" }, { "key": "secret_token", "value": "[FILTERED]" } ], "host": "gitlab.example.com", "ip": "REDACTED", "ua": "Ruby", "route": "\/api\/:version\/internal\/allowed", "queue_duration": 4.24, "gitaly_calls": 0, "gitaly_duration": 0, "correlation_id": "XPUZqTukaP3"}# nginx_access.log[IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby"
To fix this problem, confirm that your gitlab-secrets.json fileon the Gitaly server matches the one on Gitaly client. If it doesn't match,update the secrets file on the Gitaly server to match the Gitaly client, thenreconfigure.
If you've confirmed that your gitlab-secrets.json
file is the same on all Gitaly servers and clients,the application might be fetching this secret from a different file. Your Gitaly server'sconfig.toml file
indicates the secrets file in use.If that setting is missing, GitLab defaults to using .gitlab_shell_secret
under/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret
.
Repository pushes fail with 401 Unauthorized
and JWT::VerificationError
When attempting git push
, you can see:
401 Unauthorized
errors.The following in server logs:
{ ... "exception.class":"JWT::VerificationError", "exception.message":"Signature verification raised", ...}
This combination of errors occurs when the GitLab server has been upgraded to GitLab 15.5 or later but Gitaly has not yet been upgraded.
From GitLab 15.5, GitLab authenticates with GitLab Shell using a JWT token instead of a shared secret.You should follow the recommendations on upgrading external Gitaly and upgrade Gitaly before the GitLabserver.
Repository pushes fail with a deny updating a hidden ref
error
Due to a changeintroduced in GitLab 13.12, Gitaly has read-only, internal GitLab references that users are notpermitted to update. If you attempt to update internal references with git push --mirror
, Gitreturns the rejection error, deny updating a hidden ref
.
The following references are read-only:
- refs/environments/
- refs/keep-around/
- refs/merge-requests/
- refs/pipelines/
To mirror-push branches and tags only, and avoid attempting to mirror-push protected refs, run:
git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/*
Any other namespaces that the administrator wants to push can be included there as well via additional patterns.
Command-line tools cannot connect to Gitaly
gRPC cannot reach your Gitaly server if:
- You can't connect to a Gitaly server with command-line tools.
- Certain actions result in a
14: Connect Failed
error message.
Verify you can reach Gitaly by using TCP:
sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT]
If the TCP connection:
- Fails, check your network settings and your firewall rules.
- Succeeds, your networking and firewall rules are correct.
If you use proxy servers in your command line environment such as Bash, these can interfere withyour gRPC traffic.
If you use Bash or a compatible command line environment, run the following commands to determinewhether you have proxy servers configured:
echo $http_proxyecho $https_proxy
If either of these variables have a value, your Gitaly CLI connections may be getting routed througha proxy which cannot connect to Gitaly.
To remove the proxy setting, run the following commands (depending on which variables had values):
unset http_proxyunset https_proxy
Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories
You might see the following in Gitaly and Praefect logs:
{ ... "error":"rpc error: code = PermissionDenied desc = permission denied", "grpc.code":"PermissionDenied", "grpc.meta.client_name":"gitlab-web", "grpc.request.fullMethod":"/gitaly.ServerService/ServerInfo", "level":"warning", "msg":"finished unary call with code PermissionDenied", ...}
This information in the logs is a gRPC callerror response code.
If this error occurs, even thoughthe Gitaly auth tokens are set up correctly,it's likely that the Gitaly servers are experiencingclock drift.
Ensure the Gitaly clients and servers are synchronized, and use an NTP timeserver to keep them synchronized.
Gitaly not listening on new address after reconfiguring
When updating the gitaly['configuration'][:listen_addr]
or gitaly['configuration'][:prometheus_listen_addr]
values, Gitaly maycontinue to listen on the old address after a sudo gitlab-ctl reconfigure
.
When this occurs, run sudo gitlab-ctl restart
to resolve the issue. This should no longer benecessary because this issue is resolved.
Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node
If this error occurs even though file permissions are correct, it's likely that the Gitaly node isexperiencing clock drift.
Ensure that the GitLab and Gitaly nodes are synchronized and use an NTP timeserver to keep them synchronized if possible.
Health check warnings
The following warning in /var/log/gitlab/praefect/current
can be ignored.
"error":"full method name not found: /grpc.health.v1.Health/Check","msg":"error when looking up method info"
File not found errors
The following errors in /var/log/gitlab/gitaly/current
can be ignored.They are caused by the GitLab Rails application checking for specific filesthat do not exist in a repository.
"error":"not found: .gitlab/route-map.yml""error":"not found: Dockerfile""error":"not found: .gitlab-ci.yml"
Git pushes are slow when Dynatrace is enabled
Dynatrace can cause the /opt/gitlab/embedded/bin/gitaly-hooks
reference transaction hook,to take several seconds to start up and shut down. gitaly-hooks
is executed twice when userspush, which causes a significant delay.
If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace.
gitaly check
fails with 401
status code
gitaly check
can fail with 401
status code if Gitaly can't access the internal GitLab API.
One way to resolve this is to make sure the entry is correct for the GitLab internal API URL configured in gitlab.rb
with gitlab_rails['internal_api_url']
.
Changes (diffs) don't load for new merge requests when using Gitaly TLS
After enabling Gitaly with TLS, changes (diffs) for new merge requests are not generatedand you see the following message in GitLab:
Building your merge request... This page will update when the build is complete
Gitaly must be able to connect to itself to complete some operations. If the Gitaly certificate is not trusted by the Gitaly server,merge request diffs can't be generated.
If Gitaly can't connect to itself, you see messages in the Gitaly logs like the following messages:
{ "level":"warning", "msg":"[core] [Channel #16 SubChannel #17] grpc: addrConn.createTransport failed to connect to {Addr: \"ext-gitaly.example.com:9999\", ServerName: \"ext-gitaly.example.com:9999\", }. Err: connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "pid":820, "system":"system", "time":"2023-11-06T05:40:04.169Z"}{ "level":"info", "msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: remote error: tls: bad certificate\"", "pid":820, "system":"system", "time":"2023-11-06T05:40:04.169Z"}
To resolve the problem, ensure that you have added your Gitaly certificate to the /etc/gitlab/trusted-certs
folder on the Gitaly serverand:
- Reconfigure GitLab so the certificates are symlinked
- Restart Gitaly manually
sudo gitlab-ctl restart gitaly
for the certificates to be loaded by the Gitaly process.
Gitaly fails to fork processes stored on noexec
file systems
Because of changes introduced in GitLab 14.10, applying the noexec
option to a mountpoint (for example, /var
) causes Gitaly to throw permission denied
errors related to forking processes. For example:
fork/exec /var/opt/gitlab/gitaly/run/gitaly-2057/gitaly-git2go: permission denied
To resolve this, remove the noexec
option from the file system mount. An alternative is to change the Gitaly runtime directory:
- Add
gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'
to/etc/gitlab/gitlab.rb
and specify a location withoutnoexec
set. - Run
sudo gitlab-ctl reconfigure
.
Commit signing fails with invalid argument: signing key is encrypted
or invalid data: tag byte does not have MSB set.
Because Gitaly commit signing is headless and not associated with a specific user, the GPG signing key must be created without a passphrase, or the passphrase must be removed before export.
Gitaly logs show errors in info
messages
Because of a bug introduced in GitLab 16.3, additional entries were written to theGitaly logs. These log entries contained "level":"info"
but the msg
string appeared to contain an error.
For example:
{"level":"info","msg":"[core] [Server #3] grpc: Server.Serve failed to create ServerTransport: connection error: desc = \"ServerHandshake(\\\"x.x.x.x:x\\\") failed: wrapped server handshake: EOF\"","pid":6145,"system":"system","time":"2023-12-14T21:20:39.999Z"}
The reason for this log entry is that the underlying gRPC library sometimes output verbose transportation logs. These log entries appear to be errors but are, in general,safe to ignore.
This bug was fixed in GitLab 16.4.5, 16.5.5, and 16.6.0, which prevents these types of messages frombeing written to the Gitaly logs.
Troubleshoot Praefect (Gitaly Cluster)
The following sections provide possible solutions to Gitaly Cluster errors.
Check cluster health
Introduced in GitLab 14.5.
The check
Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster.
gitlab-ctl praefect check
The following sections describe the checks that are run.
Praefect migrations
Because Database migrations must be up to date for Praefect to work correctly, checks if Praefect migrations are up to date.
If this check fails:
- See the
schema_migrations
table in the database to see which migrations have run. - Run
praefect sql-migrate
to bring the migrations up to date.
Node connectivity and disk access
Checks if Praefect can reach all of its Gitaly nodes, and if each Gitaly node has read and write access to all of its storages.
If this check fails:
- Confirm the network addresses and tokens are set up correctly:
- In the Praefect configuration.
- In each Gitaly node's configuration.
- On the Gitaly nodes, check that the
gitaly
process being run asgit
. There might be a permissions issue that is preventing Gitaly fromaccessing its storage directories. - Confirm that there are no issues with the network that connects Praefect to Gitaly nodes.
Database read and write access
Checks if Praefect can read from and write to the database.
If this check fails:
See if the Praefect database is in recovery mode. In recovery mode, tables may be read only. To check, run:
select pg_is_in_recovery()
Confirm that the user that Praefect uses to connect to PostgreSQL has read and write access to the database.
See if the database has been placed into read-only mode. To check, run:
show default_transaction_read_only
Inaccessible repositories
Checks how many repositories are inaccessible because they are missing a primary assignment, or their primary is unavailable.
If this check fails:
- See if any Gitaly nodes are down. Run
praefect ping-nodes
to check. - Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persistto the database, leading Praefect to think nodes are unhealthy.
Check clock synchronization
Introduced in GitLab 14.8.
Authentication between Praefect and the Gitaly servers requires the server times to bein sync so the token check succeeds.
This check helps identify the root cause of permission denied
errors being logged by Praefect.
For offline environments where access to public pool.ntp.org
servers is not possible, the Praefect check
sub-command fails thischeck with an error message similar to:
checking with NTP service at and allowed clock drift 60000ms [correlation_id: <XXX>]Failed (fatal) error: gitaly node at tcp://[gitlab.example-instance.com]:8075: rpc error: code = DeadlineExceeded desc = context deadline exceeded
To resolve this issue, set an environment variable on all Praefect servers to point to an accessible internal NTP server. For example:
export NTP_HOST=ntp.example.com
Praefect errors in logs
If you receive an error, check /var/log/gitlab/gitlab-rails/production.log
.
Here are common errors and potential causes:
- 500 response code
ActionView::Template::Error (7:permission denied)
praefect['configuration'][:auth][:token]
andgitlab_rails['gitaly_token']
do not match on the GitLab server.
Unable to save project. Error: 7:permission denied
- Secret token in
praefect['configuration'][:virtual_storage]
on GitLab server does not match thevalue ingitaly['auth_token']
on one or more Gitaly servers.
- Secret token in
- 503 response code
GRPC::Unavailable (14:failed to connect to all addresses)
- GitLab was unable to reach Praefect.
GRPC::Unavailable (14:all SubCons are in TransientFailure...)
- Praefect cannot reach one or more of its child Gitaly nodes. Try runningthe Praefect connection checker to diagnose.
Praefect database experiencing high CPU load
Some common reasons for the Praefect database to experience elevated CPU usage include:
- Prometheus metrics scrapes running an expensive query. If you have GitLab 14.2or above, set
praefect['configuration'][:prometheus_exclude_database_from_default_metrics] = true
ingitlab.rb
. - Read distribution caching is disabled, increasing the number of queries made to thedatabase when user traffic is high. Ensure read distribution caching is enabled.
Determine primary Gitaly node
To determine the primary node of a repository:
In GitLab 14.6 and later, use the praefect metadata subcommand.
In GitLab 13.12 to GitLab 14.5 with repository-specific primaries,use the gitlab:praefect:replicas Rake task.
With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.To determine the current primary Gitaly node for a specific virtual storage:
(Recommended) Use the
Shard Primary Election
Grafana chart on theGitlab Omnibus - Praefect
dashboard.If you do not have Grafana set up, use the following command on each host of eachPraefect node:
curl localhost:9652/metrics | grep gitaly_praefect_primaries
View repository metadata
Introduced in GitLab 14.6.
Gitaly Cluster maintains a metadata database about the repositories stored on the cluster. Use the praefect metadata
subcommandto inspect the metadata for troubleshooting.
You can retrieve a repository's metadata by its Praefect-assigned repository ID:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id>
When the physical path on the physical storage starts with @cluster
, you canfind the repository ID in the physical path.
You can also retrieve a repository's metadata by its virtual storage and relative path:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage <virtual-storage> -relative-path <relative-path>
Examples
To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1
To retrieve the metadata for a repository with virtual storage default
and relative path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git
:
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git
Either of these examples retrieve the following metadata for an example repository:
Repository ID: 54771Virtual Storage: "default"Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"Primary: "gitaly-1"Generation: 1Replicas:- Storage: "gitaly-1" Assigned: true Generation: 1, fully up to date Healthy: true Valid Primary: true Verified At: 2021-04-01 10:04:20 +0000 UTC- Storage: "gitaly-2" Assigned: true Generation: 0, behind by 1 changes Healthy: true Valid Primary: false Verified At: unverified- Storage: "gitaly-3" Assigned: true Generation: replica not yet created Healthy: false Valid Primary: false Verified At: unverified
Available metadata
The metadata retrieved by praefect metadata
includes the fields in the following tables.
Field | Description |
---|---|
Repository ID | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories. |
Virtual Storage | Name of the virtual storage the repository is stored in. |
Relative Path | Repository's path in the virtual storage. |
Replica Path | Where on the Gitaly node's disk the repository's replicas are stored. |
Primary | Current primary of the repository. |
Generation | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. |
Replicas | A list of replicas that exist or are expected to exist. |
For each replica, the following metadata is available:
Replicas Field | Description |
---|---|
Storage | Name of the Gitaly storage that contains the replica. |
Assigned | Indicates whether the replica is expected to exist in the storage. Can be false if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased. |
Generation | Latest confirmed generation of the replica. It indicates: - The replica is fully up to date if the generation matches the repository's generation. |
Healthy | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. |
Valid Primary | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if: - It is stored on a healthy Gitaly node. |
Verified At | Indicates last successful verification of the replica by the verification worker. If the replica has not yet been verified, unverified is displayed in place of the last successful verification time. Introduced in GitLab 15.0. |
Command fails with 'repository not found'
If the supplied value for -virtual-storage
is incorrect, the command returns the following error:
get metadata: rpc error: code = NotFound desc = repository not found
The documented examples specify -virtual-storage default
. Check the Praefect server setting praefect['configuration'][:virtual_storage]
in /etc/gitlab/gitlab.rb
.
Check that repositories are in sync
Is some cases the Praefect database can get out of sync with the underlying Gitaly nodes. To check thata given repository is fully synced on all nodes, run the gitlab:praefect:replicas Rake task on your Rails node.This Rake task checksums the repository on all Gitaly nodes.
The Praefect dataloss command only checks the state of the repository in the Praefect database, and cannotbe relied to detect sync problems in this scenario.
Relation does not exist errors
By default Praefect database tables are created automatically by gitlab-ctl reconfigure
task.
However, the Praefect database tables are not created on initial reconfigure and can throwerrors that relations do not exist if either:
- The
gitlab-ctl reconfigure
command isn't executed. - Errors occur during the execution.
For example:
ERROR: relation "node_status" does not exist at character 13
ERROR: relation "replication_queue_lock" does not exist at character 40
This error:
{"level":"error","msg":"Error updating node: pq: relation \"node_status\" does not exist","pid":210882,"praefectName":"gitlab1x4m:0.0.0.0:2305","time":"2021-04-01T19:26:19.473Z","virtual_storage":"praefect-cluster-1"}
To solve this, the database schema migration can be done using sql-migrate
sub-command ofthe praefect
command:
$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migratepraefect sql-migrate: OK (applied 21 migrations)
Requests fail with 'repository scoped: invalid Repository' errors
This indicates that the virtual storage name used in thePraefect configuration does not match the storage name used ingitaly['configuration'][:storage][<index>][:name] setting for GitLab.
Resolve this by matching the virtual storage names used in Praefect and GitLab configuration.
Gitaly Cluster performance issues on cloud platforms
Praefect does not require a lot of CPU or memory, and can run on small virtual machines.Cloud services may place other limits on the resources that small VMs can use, such asdisk IO and network traffic.
Praefect nodes generate a lot of network traffic. The following symptoms can be observed if their network bandwidth hasbeen throttled by the cloud service:
- Poor performance of Git operations.
- High network latency.
- High memory use by Praefect.
Possible solutions:
- Provision larger VMs to gain access to larger network traffic allowances.
- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.
gitlab-ctl reconfigure
fails with error: STDOUT: praefect: configuration error: error reading config file: toml: cannot store TOML string into a Go int
This error occurs when praefect['database_port']
or praefect['database_direct_port']
are configured as a string instead of an integer.
Profiling Gitaly
Gitaly exposes several of the Go built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listeningon port 9236
of the GitLab server:
Get a list of running
goroutines
and their backtraces:curl --output goroutines.txt "http://<gitaly_server>:9236/debug/pprof/goroutine?debug=2"
Run a CPU profile for 30 seconds:
curl --output cpu.bin "http://<gitaly_server>:9236/debug/pprof/profile"
Profile heap memory usage:
curl --output heap.bin "http://<gitaly_server>:9236/debug/pprof/heap"
Record a 5 second execution trace. This impacts the Gitaly performance while running:
curl --output trace.bin "http://<gitaly_server>:9236/debug/pprof/trace?seconds=5"
On a host with go
installed, the CPU profile and heap profile can be viewed in a browser:
go tool pprof -http=:8001 cpu.bingo tool pprof -http=:8001 heap.bin
Execution traces can be viewed by running:
go tool trace heap.bin