Commit graph

54 commits

Author SHA1 Message Date
silverwind
24e64fe372
Replace interface{} with any (#25686) (#25687)
Same perl replacement as https://github.com/go-gitea/gitea/pull/25686
but for 1.20 to ease future backporting.
2023-07-04 23:41:32 -04:00
wxiaoguang
18f26cfbf7
Improve queue and logger context (#24924)
Before there was a "graceful function": RunWithShutdownFns, it's mainly
for some modules which doesn't support context.

The old queue system doesn't work well with context, so the old queues
need it.

After the queue refactoring, the new queue works with context well, so,
use Golang context as much as possible, the `RunWithShutdownFns` could
be removed (replaced by RunWithCancel for context cancel mechanism), the
related code could be simplified.

This PR also fixes some legacy queue-init problems, eg:

* typo : archiver: "unable to create codes indexer queue" => "unable to
create repo-archive queue"
* no nil check for failed queues, which causes unfriendly panic

After this PR, many goroutines could have better display name:

![image](https://github.com/go-gitea/gitea/assets/2114189/701b2a9b-8065-4137-aeaa-0bda2b34604a)

![image](https://github.com/go-gitea/gitea/assets/2114189/f1d5f50f-0534-40f0-b0be-f2c9daa5fe92)
2023-05-26 07:31:55 +00:00
wxiaoguang
ec2a01d1e2
Fix regression: access log template, gitea manager cli command (#24838)
Close #24836

![image](https://github.com/go-gitea/gitea/assets/2114189/95b025d2-f25f-4246-a08a-fe44ecb787a9)

![image](https://github.com/go-gitea/gitea/assets/2114189/c3afe1fa-2a23-420d-a016-3b67dcd04cd5)
2023-05-22 09:38:38 +08:00
wxiaoguang
4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
zeripath
7565e5c3de
Implement systemd-notify protocol (#21151)
This PR adds support for the systemd notify protocol. Several status
messagess are provided. We should likely add a common notify/status
message for graceful.

Replaces #21140

Signed-off-by: Andrew Thornton <art27@cantab.net>

---------

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: ltdk <usr@ltdk.xyz>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-15 22:20:30 +00:00
silverwind
938b591994
Remove most path-based golangci exclusions (#24214)
They are non-obvious and do not survive refactor.

Will replace with `//nolint` comments after CI results are in.
2023-04-19 22:08:01 -04:00
flynnnnnnnnnn
e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
delvh
0ebb45cfe7
Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)
Found using
`find . -type f -name '*.go' -print -exec vim {} -c
':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;`

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-24 20:29:17 +01:00
zeripath
943753f560
Support Proxy protocol (#12527)
This PR adds functionality to allow Gitea to sit behind an
HAProxy and HAProxy protocolled connections directly.

Fix #7508

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-08-21 19:20:43 +01:00
wxiaoguang
6784a707d1
Fix graceful doc (#20883) 2022-08-21 02:50:27 -04:00
JonRB
cb37c6ba5c
call builtinUnused() if internal SSH is disabled (#20877)
The graceful manager waits for 4 listeners to be created or to be told that they are not needed. If it is not told about them it will indefinitely and timeout. 

This leads to SVC hosts not being told of being in the readyState but on Unix would lead to the termination of the process.

There was an unfortunate regression in #20299 which missed this subtly and in the case whereby SSH is disabled the `builtinUnused()` is not called.

This PR adds a call to `builtinUnused()` when not using the builtin ssh to allow `createServerWaitGroup.Done()` to be called. 

In addition it was noted that the if/else clauses for timeout informing of the SVC host were in the wrong order. These have been swapped.

Fix #20609
2022-08-20 22:09:41 +01:00
Frank
1f0638747b
Preserve unix socket file (#20499)
By default Gitea will always unlink any sockets that are provided using the `LISTEN_FDS` environment variable. This is because it uses this variable to handle passing when it is doing a graceful restart. However, this same mechanism is used by systemd - which explicitly expects that passed in sockets should not be unlinked by the receiving process. 

This PR adjusts Gitea's graceful restart mechanism to use an additional environment variable which tracks if a listening socket was opened by Gitea - and therefore should be unlinked on shutdown by Gitea.

Fix #20490

Co-authored-by: zeripath <art27@cantab.net>
2022-08-13 22:31:33 +01:00
Wim
cb50375e2b
Add more linters to improve code readability (#19989)
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
wxiaoguang
a9dc9b06e4
Fix signal loop in graceful manager (#19943) 2022-06-12 21:50:18 +08:00
wxiaoguang
c18d8d6968
Remove legacy +build: constraint (#19582)
Go 1.17 and later use modern `//go:build` constraints, the old `// +build:` constraints should be removed.
2022-05-02 23:22:45 +08:00
zeripath
c88547ce71
Add Goroutine stack inspector to admin/monitor (#19207)
Continues on from #19202.

Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them. 

This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.

The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.

If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process. 

The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.

In this way we should be able to more easily examine goroutines that have been stuck.

A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-31 19:01:43 +02:00
zeripath
5fe764b1eb
Add pprof labels in processes and for lifecycles (#19202)
Use pprof labelling to help identify goroutines with stacks.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-25 12:47:12 +00:00
zeripath
83c9035971
Immediately Hammer if second kill is sent (#18823)
Currently Gitea will wait for HammerTime or nice shutdown if kill -1 or kill -2
is sent. We should just immediately hammer if there is a second kill.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-19 16:36:25 +00:00
6543
54e9ee37a7
format with gofumpt (#18184)
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
zeripath
c96be0cd98
Make SSL cipher suite configurable (#17440) 2021-11-20 01:12:43 -05:00
Lunny Xiao
4a57c9ea17
Fix some lints (#17337)
Fix some linting problems.
2021-10-17 20:47:12 +01:00
zeripath
429258cff3
Use pointer for wrappedConn methods (#17295)
Fix #17294

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-12 21:55:18 +01:00
Eng Zer Jun
f2e7d5477f
refactor: move from io/ioutil to io and os package (#17109)
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-22 13:38:34 +08:00
John Olheiser
648464b504
Add bundle download for repository (#14538)
* Add bundle download

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix fmt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix build tags

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Download specific commit

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2021-08-24 11:47:09 -05:00
Stanley Hu
3dba75fb97
Support HTTP/2 in Let's Encrypt (#16371)
Modify the tlsConfig.NextProtos for Let's Encrypt and built-in HTTPS server in order to support HTTP/2.

Co-authored-by: 6543 <6543@obermui.de>
2021-07-13 18:17:46 +01:00
zeripath
daa5a23548
Set self-adjusting deadline for connection writing (#16068)
* Set self-adjusting deadline for connection writing

In #16055 it appears that the simple 5s deadline doesn't work for large
file writes. Now we can't - or at least shouldn't just set no deadline
as go will happily let these connections block indefinitely. However,
what seems reasonable is to set some minimum rate we expect for writing.

This PR suggests the following algorithm:

* Every write has a minimum timeout of 5s (adjustable at compile time.)
* If there has been a previous write - then consider its previous
deadline, add half of the minimum timeout + 2s per kb about to written.
* If that new deadline is after the minimum timeout use that.

Fix #16055

* Linearly increase timeout

* Make PerWriteTimeout, PerWritePerKbTimeouts configurable

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-11 00:25:25 +03:00
zeripath
3183a465d7
Make modules/context.Context a context.Context (#16031)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 02:18:11 -04:00
zeripath
ba526ceffe
Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow level queue, reduce goroutines etc (#15693)
* move shutdownfns, terminatefns and hammerfns out of separate goroutines

Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.

This may help reduce the background select/poll load in certain
configurations.

* The LevelDB queues can actually wait on empty instead of polling

Slight refactor to cause leveldb queues to wait on empty instead of polling.

* Shutdown the shadow level queue once it is empty

* Remove bytefifo additional goroutine for readToChan as it can just be run in run

* Remove additional removeWorkers goroutine for workers

* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher

* Add shutdown flusher to CUQ

* move persistable channel shutdown stuff to Shutdown Fn

* Ensure that UPCQ has the correct config

* handle shutdown during the flushing

* reduce risk of race between zeroBoost and addWorkers

* prevent double shutdown

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 16:22:26 +02:00
zeripath
1a5659943e
Add timeout to writing to responses (#15831)
In #15826 it has become apparent that there are a few occasions when a response can
hang during writing, and because there is no timeout go will happily just block
interminably. This PR adds a fixed 5 second timeout to all writes to a connection.

Fix #15826

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-14 13:26:03 +01:00
zeripath
bdc1697729
Drop back to use IsAnInteractiveSession for SVC (#15749)
* Drop back to use IsAnInteractiveSession for SVC

There is an apparent permission change problem when using
IsWindowsService to determine if the SVC manager should be
used.

This PR simply drops back to using IsAnInteractiveSession as
this does not change behaviour.

Fix #15454

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Yes staticcheck I know this is deprecated

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Just leave me alone lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-05-07 11:27:31 +02:00
zeripath
113c1557ff
Fix regression from #14623 - use debug SVC handler only on interactive sessions (#15210)
Unfortunately #14623 changed from the deprecated IsInteractiveSession to
IsWindowsService without recognising that they are the complement of
each other.

This means that Windows SVC control is not working correctly. This PR
adds some Tracing statements but also fixes the bug.

Fix #15159

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-03-31 22:48:48 +03:00
zeripath
c03f530212
Make internal SSH server host key path configurable (#14918)
* Make SSH server host key path configurable

* make it possible to have multiple keys

* Make gitea.rsa the default key

* Add some more logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-08 03:43:59 +01:00
Lunny Xiao
a1c9e8f266
Fix windows build error (#14263)
* fix build

* take flash error message back and fix more windows lint error

* performance optimization

* own step to check lint for windows

Co-authored-by: 6543 <6543@obermui.de>
2021-01-06 09:38:00 +08:00
zeripath
2f1353a2f3
Move install pages out of main macaron routes (#13195)
* Move install pages out of main macaron loop

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update templates/post-install.tmpl

Co-authored-by: Lauris BH <lauris@nix.lv>

* remove prefetch

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-19 17:03:08 -04:00
zeripath
702e98c5ec
Set TLS minimum version to 1.2 (#12689)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-09-02 23:37:49 +01:00
zeripath
74bd9691c6
Re-attempt to delete temporary upload if the file is locked by another process (#12447)
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.

Fix #12339

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
2020-08-11 21:05:34 +01:00
zeripath
c5b08f6d5a
Pause, Resume, Release&Reopen, Add and Remove Logging from command line (#11777)
* Make LogDescriptions race safe

* Add manager commands for pausing, resuming, adding and removing loggers

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Placate lint

* Ensure that file logger is run!

* Add support for smtp and conn

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add release-and-reopen

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-05 20:07:07 -04:00
zeripath
4a04740daf
Handle panics that percolate up to the graceful module (#11291)
* Handle panics in graceful goroutines

Adds a some deferred functions to handle panics in graceful goroutines

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Handle panic in webhook.Deliver

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Handle panic in mirror.syncMirror

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-05-14 20:06:00 -04:00
zeripath
0e799c26ba
Prevent panic during wrappedConn close at hammertime (#11219)
* Prevent panic during wrappedConn close at hammertime

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/graceful/server.go

* Fix extraneous debug in goldmark.go

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-04-26 21:01:06 -03:00
zeripath
37892be635
Restore graceful restart on SIGHUP (#10224)
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-02-11 13:29:45 +08:00
zeripath
c512bfd09c
Prevent double waitgroup decrement (#10170)
* Prevent double waitgroup decrement
2020-02-07 09:08:09 +00:00
6543
169bb55ec6
[UI] Show pull icon on pull (#10061)
* make fmt
* show pull icon on pull
2020-01-29 18:54:34 +00:00
zeripath
c01221e70f
Queue: Make WorkerPools and Queues flushable (#10001)
* Make WorkerPools and Queues flushable

Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Change to for loop

* Add IsEmpty and begin just making the queues composed WorkerPools

* subsume workerpool into the queues and create a flushable interface

* Add manager command

* Move flushall to queue.Manager and add to testlogger

* As per @guillep2k

* as per @guillep2k

* Just make queues all implement flushable and clean up the wrapped queue flushes

* cope with no timeout

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-28 20:01:06 -05:00
zeripath
1d7b7504d0
Make CertFile and KeyFile relative to CustomPath (#9868)
* Make CertFile and KeyFile relative to CustomPath

The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath.

Fix #4196

* Improve error reporting when reading certificates

* Apply suggestions from code review

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-01-19 19:07:44 +00:00
zeripath
7cc16740a5
Fix missed change to GetManager() (#9361) 2019-12-15 13:15:17 +00:00
zeripath
e3c3b33ea7
Graceful: Xorm, RepoIndexer, Cron and Others (#9282)
* Change graceful to use a singleton obtained through GetManager instead of a global.
* Graceful: Make TestPullRequests shutdownable
* Graceful: Make the cron tasks graceful
* Graceful: AddTestPullRequest run in graceful ctx
* Graceful: SyncMirrors shutdown
* Graceful: SetDefaultContext for Xorm to be HammerContext
* Avoid starting graceful for migrate commands and checkout
* Graceful: DeliverHooks now can be shutdown
* Fix multiple syncing errors in modules/sync/UniqueQueue &  Make UniqueQueue closable
* Begin the process of making the repo indexer shutdown gracefully
2019-12-15 09:51:28 +00:00
David Svantesson
6460284085 Add comment to exported function WindowsServiceName (make revive) (#9241) 2019-12-03 19:16:29 -05:00
zeripath
60c5339042 Graceful: Cancel Process on monitor pages & HammerTime (#9213)
* Graceful: Create callbacks to with contexts

* Graceful: Say when Gitea is completely finished

* Graceful: Git and Process within HammerTime

Force all git commands to terminate at HammerTime
Force all process commands to terminate at HammerTime
Move almost all git processes to run as git Commands

* Graceful: Always Hammer after Shutdown

* ProcessManager: Add cancel functionality

* Fix tests

* Make sure that process.Manager.Kill() cancels

* Make threadsafe access to Processes and remove own unused Kill

* Remove cmd from the process manager as it is no longer used

* the default context is the correct context

* get rid of double till
2019-11-30 16:40:22 +02:00
zeripath
e3f22ad2cc Graceful: Allow graceful restart for unix sockets (#9113)
Previously we could not handle graceful restarts for http over unix
sockets. These can now be handled.
2019-11-24 03:11:24 +01:00
David Svantesson
a4e398dc45 Run make fmt on master (#9129) 2019-11-22 21:00:01 +00:00