Commit graph

37 commits

Author SHA1 Message Date
Gusted 2d9afd0c21
[CHORE] Remove Microsoft SQL Server Support
- Per https://codeberg.org/forgejo/discussions/issues/122
2024-04-05 23:37:36 +02:00
Michael Kriese 849de07064 feat(xorm): add max idle time setting for db connections (#2418)
Add a new optional `CONN_MAX_IDLETIME`[^1]

This allows to set the `SetConnMaxIdleTime` on `database/sql`.
It's useful to allow to close more idle connections to reduce database connections, especially on postgresql.
For me i would like to use it to set a higher max idle connections but they will all be closed after being idle.
So also the last idle connection will be closed when there is no load on forgejo.
I also use it with max connection lifetime, because currently `database/sql` doesn't detect a postgresql master change[^2] and i'll get `[E] can't update runner status: pq: cannot execute UPDATE in a read-only transaction`[^3] on forgejo until the connection is closed.

[^1]: https://pkg.go.dev/database/sql#DB.SetConnMaxIdleTime
[^2]: https://stackoverflow.com/questions/51858659/how-to-safely-discard-golang-database-sql-pooled-connections-for-example-when-t
[^3]: https://matrix.to/#/!zpNKWqkiEOyljSMQDK:matrix.org/$_AJft_amsGn5hXGOYw75JoBJQnW3aKJEpb-Iw53L_TU?via=schinas.net&via=matrix.org&via=nitro.chat

Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/2418
Reviewed-by: Gusted <gusted@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Michael Kriese <michael.kriese@visualon.de>
Co-committed-by: Michael Kriese <michael.kriese@visualon.de>
2024-02-21 12:17:16 +00:00
Earl Warren 094c84ed6d
Merge branch 'rebase-forgejo-dependency' into wip-forgejo 2024-02-05 18:58:23 +01:00
Gusted 6fb55e9c08
[GITEA] Add slow SQL query warning (squash) Fix setting typo
- Fix typo in the slow query threshold setting, add a deprecation warning.
- Resolves #2203

(cherry picked from commit 02f6608e5f)
(cherry picked from commit 4e8f6b2ffd)
2024-02-05 16:54:44 +01:00
Gusted 664192767c
[GITEA] Add slow SQL query warning
- Databases are one of the most important parts of Forgejo, every
interaction with Forgejo uses the database in one way or another.
Therefore, it is important to maintain the database and recognize when
Forgejo is not doing well with the database. Forgejo already has the
option to log *every* SQL query along with its execution time, but
monitoring becomes impractical for larger instances and takes up
unnecessary storage in the logs.
- Add a QoL enhancement that allows instance administrators to specify a
threshold value beyond which query execution time is logged as a warning
in the xorm logger. The default value is a conservative five seconds to
avoid this becoming a source of spam in the logs.
- The use case for this patch is that with an instance the size of Codeberg, monitoring SQL logs is not very fruitful and most of them are uninteresting. Recently, in the context of persistent deadlock issues (https://codeberg.org/forgejo/forgejo/issues/220), I have noticed that certain queries hold locks on tables like comment and issue for several seconds. This patch helps to identify which queries these are and when they happen.
- Added unit test.

(cherry picked from commit 24bbe7886f)
(cherry picked from commit 6e29145b3c)
(cherry picked from commit 63731e3071)
(cherry picked from commit 3ce1a09736)
(cherry picked from commit a64426907d)
(cherry picked from commit 4b19215691)
(cherry picked from commit e635674435)
(cherry picked from commit 9cf501f1af)
(cherry picked from commit 0d6b934eba)
(cherry picked from commit 4b6c273879)
(cherry picked from commit 89b1315338)
(cherry picked from commit edd8e66ce9)

[GITEA] Add slow SQL query warning (squash) document the setting

(cherry picked from commit ce38599c51)
(cherry picked from commit 794aa67c68)
(cherry picked from commit a4c2c6b004)
(cherry picked from commit 97912752bc)
(cherry picked from commit 00b5327c97)
(cherry picked from commit 1069c860e7)
(cherry picked from commit 84241f42c8)
(cherry picked from commit e4bda0e845)
(cherry picked from commit 7357fb91bf)
(cherry picked from commit a8dd7f6da2)
(cherry picked from commit e636e9f4be)
(cherry picked from commit bf04ae8603)
(cherry picked from commit 93b19e3568)
(cherry picked from commit 83f91363ad)
(cherry picked from commit e34a05bc73)
(cherry picked from commit 68569aeee9)
2024-02-05 16:05:50 +01:00
Caesar Schinas 95c371d1bb
[BRANDING] Rebrand default config settings for new installs (#140)
Replaces `Gitea` with `Forgejo` in the default config settings for new installs.

This will not affect existing installs.

Co-authored-by: Caesar Schinas <caesar@caesarschinas.com>
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/140
Co-authored-by: Caesar Schinas <caesar@noreply.codeberg.org>
Co-committed-by: Caesar Schinas <caesar@noreply.codeberg.org>
(cherry picked from commit ca1319aa16)
(cherry picked from commit 52a4d238a0)
(cherry picked from commit f63536538c)

Conflicts:
	web_src/js/features/install.js
(cherry picked from commit 861cc434e1)
(cherry picked from commit 0e6ea60c80)
(cherry picked from commit 0cbc0ec15d)
(cherry picked from commit 3cc19b0ae2)
(cherry picked from commit 50fcb885fe)
(cherry picked from commit f6039d4df4)
(cherry picked from commit 5ae5c6ba2d)
(cherry picked from commit f0b565e0ed)
(cherry picked from commit adbd4d2015)
(cherry picked from commit d26c540ffd)
(cherry picked from commit 6df6781b42)
(cherry picked from commit b6fb56e1c4)
(cherry picked from commit bb4f98a0ca)
(cherry picked from commit 6779229f27)
(cherry picked from commit c216c85aee)
(cherry picked from commit dff780bced)
(cherry picked from commit 4e036aa3b6)
(cherry picked from commit 8b3bc3e8a6)
(cherry picked from commit 1e4d852332)
(cherry picked from commit 07a15d1844)
(cherry picked from commit fb44b3e10d)
(cherry picked from commit b212d83319)
(cherry picked from commit 5754971be5)
(cherry picked from commit 0c43b4e82c)

Conflicts:
	routers/install/install.go
	https://codeberg.org/forgejo/forgejo/pulls/1351
(cherry picked from commit 2e22a7208a)
(cherry picked from commit 676b0a8a48)
(cherry picked from commit bc4a8bf9bc)
(cherry picked from commit 5e09a4e174)
(cherry picked from commit 712c52a32a)
(cherry picked from commit ba3d93cc4a)
(cherry picked from commit a5a0396abc)
(cherry picked from commit 09b205f30b)
(cherry picked from commit 52b7729e86)
(cherry picked from commit bfe78735e8)
(cherry picked from commit 915c4f7a7b)
(cherry picked from commit 3b1af856bb)
(cherry picked from commit ffe2c0a353)
(cherry picked from commit fd5cc72128)
(cherry picked from commit 95d3555e47)
(cherry picked from commit b9b853f4b0)
(cherry picked from commit 967453eb6f)
(cherry picked from commit 3298f502dd)
(cherry picked from commit 3481d1f816)
(cherry picked from commit 7508c24a0c)
(cherry picked from commit 54e97f7c35)
(cherry picked from commit 492876992f)
2024-02-05 16:02:13 +01:00
sdvcrx 80d48621cd
Fix incorrect PostgreSQL connection string for Unix sockets (#28865)
Fix #28864
2024-01-20 16:04:47 +00:00
wxiaoguang 2df7563f31
Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#28662)
Mainly for MySQL/MSSQL.

It is important for Gitea to use case-sensitive database charset
collation. If the database is using a case-insensitive collation, Gitea
will show startup error/warning messages, and show the errors/warnings
on the admin panel's Self-Check page.

Make `gitea doctor convert` work for MySQL to convert the collations of
database & tables & columns.

* Fix #28131

## ⚠️ BREAKING ⚠️

It is not quite breaking, but it's highly recommended to convert the
database&table&column to a consistent and case-sensitive collation.
2024-01-10 11:03:23 +00:00
wxiaoguang e75e9a0e7e
Fix panic when parsing empty pgsql host (#28708)
Regression of #27723
Fix #28705
2024-01-06 17:30:03 +08:00
wxiaoguang 1e512b800c
Use filepath instead of path to create SQLite3 database file (#28374) 2023-12-06 16:57:52 +00:00
wxiaoguang 17d246cdcc
Fix incorrect pgsql conn builder behavior (#28085)
Fix #28083 and fix the tests
2023-11-17 02:30:57 +00:00
Moritz Poldrack 9b6e77c489
refactor postgres connection string building (#27723)
This patchset changes the connection string builder to use net.URL and
the host/port parser to use the stdlib function for splitting host from
port. It also adds a footnote about a potentially required portnumber
for postgres UNIX sockets.

Fixes: #24552
2023-11-01 18:00:20 +00:00
wxiaoguang ce46834b93
Remove "CHARSET" config option for MySQL, always use "utf8mb4" (#25413)
In modern days, there is no reason to make users set "charset" anymore.

Close #25378

## ⚠️ BREAKING

The key `[database].CHARSET` was removed completely as every newer
(>10years) MySQL database supports `utf8mb4` already.
There is a (deliberately) undocumented new fallback option if anyone
still needs to use it, but we don't recommend using it as it simply
causes problems.
2023-06-21 10:49:25 +00:00
wxiaoguang 4647660776
Rewrite logger system (#24726)
## ⚠️ Breaking

The `log.<mode>.<logger>` style config has been dropped. If you used it,
please check the new config manual & app.example.ini to make your
instance output logs as expected.

Although many legacy options still work, it's encouraged to upgrade to
the new options.

The SMTP logger is deleted because SMTP is not suitable to collect logs.

If you have manually configured Gitea log options, please confirm the
logger system works as expected after upgrading.

## Description

Close #12082 and maybe more log-related issues, resolve some related
FIXMEs in old code (which seems unfixable before)

Just like rewriting queue #24505 : make code maintainable, clear legacy
bugs, and add the ability to support more writers (eg: JSON, structured
log)

There is a new document (with examples): `logging-config.en-us.md`

This PR is safer than the queue rewriting, because it's just for
logging, it won't break other logic.

## The old problems

The logging system is quite old and difficult to maintain:
* Unclear concepts: Logger, NamedLogger, MultiChannelledLogger,
SubLogger, EventLogger, WriterLogger etc
* Some code is diffuclt to konw whether it is right:
`log.DelNamedLogger("console")` vs `log.DelNamedLogger(log.DEFAULT)` vs
`log.DelLogger("console")`
* The old system heavily depends on ini config system, it's difficult to
create new logger for different purpose, and it's very fragile.
* The "color" trick is difficult to use and read, many colors are
unnecessary, and in the future structured log could help
* It's difficult to add other log formats, eg: JSON format
* The log outputer doesn't have full control of its goroutine, it's
difficult to make outputer have advanced behaviors
* The logs could be lost in some cases: eg: no Fatal error when using
CLI.
* Config options are passed by JSON, which is quite fragile.
* INI package makes the KEY in `[log]` section visible in `[log.sub1]`
and `[log.sub1.subA]`, this behavior is quite fragile and would cause
more unclear problems, and there is no strong requirement to support
`log.<mode>.<logger>` syntax.


## The new design

See `logger.go` for documents.


## Screenshot

<details>


![image](https://github.com/go-gitea/gitea/assets/2114189/4462d713-ba39-41f5-bb08-de912e67e1ff)


![image](https://github.com/go-gitea/gitea/assets/2114189/b188035e-f691-428b-8b2d-ff7b2199b2f9)


![image](https://github.com/go-gitea/gitea/assets/2114189/132e9745-1c3b-4e00-9e0d-15eaea495dee)

</details>

## TODO

* [x] add some new tests
* [x] fix some tests
* [x] test some sub-commands (manually ....)

---------

Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-21 22:35:11 +00:00
Lunny Xiao 623a539f23
Move pidfile creation from setting to web cmd package (#23285)
Creating pid file should not belong to setting package and only web
command needs that. So this PR moves pidfile creation from setting
package to web command package to keep setting package more readable.

I marked this as `break` because the PIDFile path moved. For those who
have used the pid build argument, it has to be changed.

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-03-16 15:22:54 +08:00
Jason Song c84238800b
Refactor setting.Database.UseXXX to methods (#23354)
Replace #23350.

Refactor `setting.Database.UseMySQL` to
`setting.Database.Type.IsMySQL()`.

To avoid mismatching between `Type` and `UseXXX`.

This refactor can fix the bug mentioned in #23350, so it should be
backported.
2023-03-07 18:51:06 +08:00
Lunny Xiao c53ad052d8
Refactor the setting to make unit test easier (#22405)
Some bugs caused by less unit tests in fundamental packages. This PR
refactor `setting` package so that create a unit test will be easier
than before.

- All `LoadFromXXX` files has been splited as two functions, one is
`InitProviderFromXXX` and `LoadCommonSettings`. The first functions will
only include the code to create or new a ini file. The second function
will load common settings.
- It also renames all functions in setting from `newXXXService` to
`loadXXXSetting` or `loadXXXFrom` to make the function name less
confusing.
- Move `XORMLog` to `SQLLog` because it's a better name for that.

Maybe we should finally move these `loadXXXSetting` into the `XXXInit`
function? Any idea?

---------

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
2023-02-20 00:12:01 +08:00
Jason Song 0a85537c79
Support disabling database auto migration (#22053)
Gitea will migrate the database model version automatically, but it
should be able to be disabled and keep Gitea shutdown if the version is
not matched.
2022-12-07 09:58:31 -06:00
flynnnnnnnnnn e81ccc406b
Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
delvh 0ebb45cfe7
Replace all instances of fmt.Errorf(%v) with fmt.Errorf(%w) (#21551)
Found using
`find . -type f -name '*.go' -print -exec vim {} -c
':%s/fmt\.Errorf(\(.*\)%v\(.*\)err/fmt.Errorf(\1%w\2err/g' -c ':wq' \;`

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-24 20:29:17 +01:00
Norwin 8a330b6b5b
Add setting SQLITE_JOURNAL_MODE to enable WAL (#20535)
Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-07-30 21:57:41 +02:00
Gusted b7c6ec91ba
Use default values when provided values are empty (#20318)
* Use default values when provided values are empty

- When provided values are empty like `:3000` would imply that host is
empty, use the default value.
- Resolves #20316

* Update database.go

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-07-13 13:33:31 +08:00
Wim cb50375e2b
Add more linters to improve code readability (#19989)
Add nakedret, unconvert, wastedassign, stylecheck and nolintlint linters to improve code readability

- nakedret - https://github.com/alexkohler/nakedret - nakedret is a Go static analysis tool to find naked returns in functions greater than a specified function length.
- unconvert - https://github.com/mdempsky/unconvert - Remove unnecessary type conversions
- wastedassign - https://github.com/sanposhiho/wastedassign -  wastedassign finds wasted assignment statements.
- notlintlint -  Reports ill-formed or insufficient nolint directives
- stylecheck - https://staticcheck.io/docs/checks/#ST - keep style consistent
  - excluded: [ST1003 - Poorly chosen identifier](https://staticcheck.io/docs/checks/#ST1003) and [ST1005 - Incorrectly formatted error string](https://staticcheck.io/docs/checks/#ST1005)
2022-06-20 12:02:49 +02:00
Lunny Xiao 3c6c150740
Add deprecated log when using MySQL with utf8 charset (#19952) 2022-06-13 20:55:08 +08:00
6543 54e9ee37a7
format with gofumpt (#18184)
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
wxiaoguang a6f961fba4
Refactor install page (db type) (#17919)
* Refactor install page (db type)

* set correct default DB HOST for different DB TYPE
* remove legacy TiDB from documents
* unify the usage of DB TYPE, in code we only use "mysql". "MySQL" is only shown to users for friendly name.

* Gitea can use TiDB via MySQL protocol

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-12-07 13:44:08 +08:00
Lunny Xiao 29cc169d20
Fix bug when read mysql database max lifetime (#17682) 2021-11-17 18:59:23 +08:00
zeripath 5e0a3784ea
When reinitialising DBConfig reset the database use flags (#13796)
One perennial issue is users running the install page,
changing the database dialect and then suffering with issues

This PR simply resets all of the database.Use flags on
initDBConfig. This should prevent this issue from occuring.

Fix #13788
Fix #5480

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-12-02 19:39:48 -05:00
Lunny Xiao 80a6b0f5bc
Avatars and Repo avatars support storing in minio (#12516)
* Avatar support minio

* Support repo avatar minio storage

* Add missing migration

* Fix bug

* Fix test

* Add test for minio store type on avatars and repo avatars; Add documents

* Fix bug

* Fix bug

* Add back missed avatar link method

* refactor codes

* Simplify the codes

* Code improvements

* Fix lint

* Fix test mysql

* Fix test mysql

* Fix test mysql

* Fix settings

* Fix test

* fix test

* Fix bug
2020-10-14 21:07:51 +08:00
zeripath fcabbae168
Set utf8mb4 as the default charset on MySQL if CHARSET is unset (#12563)
MySQL in its infinite wisdom determines that UTF8 does not
mean UTF8. Our install scripts know about this and will set
CHARSET to utf8mb4 if we users choose this but... users who
do not explicitly set this variable will default to utf8mb3
without knowing it.

This PR changes the unset CHARSET value to utf8mb4 if users
choose to use mysql.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-08-22 13:56:33 +01:00
Stormxx 26e931ae34
fix a spelling error (#12163)
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-07-06 11:58:31 -04:00
zeripath 9ecf732abc
Prevent panic on empty HOST for mysql (#11850)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-06-11 12:47:55 -04:00
zeripath e8955173a9
Default MSSQL port 0 to allow automatic detection by default (#11642)
Fix #11633

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-05-28 23:59:59 -04:00
guillep2k 4a2d23981d
Fix SQLite concurrency problems by using BEGIN IMMEDIATE (#10368)
* Test locking immediate for SQLite3

* fix url field separator

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-02-27 07:51:37 +08:00
guillep2k ad1b6d439f Add support for database schema in PostgreSQL (#8819)
* Add support for database schema

* Require setting search_path for the db user

* Add schema setting to admin/config.tmpl

* Use a schema different from default for psql tests

* Update postgres scripts to use custom schema

* Update to xorm/core 0.7.3 and xorm/xorm c37aff9b3a

* Fix migration test

Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-01-20 16:45:14 +01:00
zeripath d8161ee3fd
Expose db.SetMaxOpenConns and allow non MySQL dbs to set conn pool params (#8528)
* Expose db.SetMaxOpenConns and allow other dbs to set their connection params
* Add note about port exhaustion

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
2019-10-21 22:20:47 +01:00
Lunny Xiao f83db078f0 Move database settings from models to setting (#7806)
* move database settings from models to setting

* update docs

* fix checkout pr

* fix tests

* fix lint

* remove unsupported tidb options

* correct wrong variable name

* remove tidb totally
2019-08-24 11:24:45 +02:00