Commit graph

8 commits

Author SHA1 Message Date
Loïc Dachary b1c86fb777
[CI] set PASSWORD_HASH_ALGO = argon2 for integration tests
(cherry picked from commit 1d7ce2a39c)
(cherry picked from commit 3af8757583)
(cherry picked from commit 0b81815209)
(cherry picked from commit 7180a3222a)
(cherry picked from commit 069044f4a6)
(cherry picked from commit dd1438f389)
(cherry picked from commit 4ae6b5d64d)
(cherry picked from commit c572666d78)
(cherry picked from commit de0a4480bc)
(cherry picked from commit 60272ef4f7)
(cherry picked from commit a68fdde4be)
(cherry picked from commit ff448cf3a6)
(cherry picked from commit 208498caaa)
(cherry picked from commit 1abd3efad8)
(cherry picked from commit 818c7f4ca9)
(cherry picked from commit 1e552bbe44)
(cherry picked from commit 88dbd6da1d)
2023-06-23 09:27:42 +02:00
FuXiaoHei c757765a9e
Implement actions artifacts (#22738)
Implement action artifacts server api.

This change is used for supporting
https://github.com/actions/upload-artifact and
https://github.com/actions/download-artifact in gitea actions. It can
run sample workflow from doc
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts.
The api design is inspired by
https://github.com/nektos/act/blob/master/pkg/artifacts/server.go and
includes some changes from gitea internal structs and methods.

Actions artifacts contains two parts:

- Gitea server api and storage (this pr implement basic design without
some complex cases supports)
- Runner communicate with gitea server api (in comming)

Old pr https://github.com/go-gitea/gitea/pull/22345 is outdated after
actions merged. I create new pr from main branch.


![897f7694-3e0f-4f7c-bb4b-9936624ead45](https://user-images.githubusercontent.com/2142787/219382371-eb3cf810-e4e0-456b-a8ff-aecc2b1a1032.jpeg)

Add artifacts list in actions workflow page.
2023-05-19 21:37:57 +08:00
wxiaoguang 6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
John Olheiser f92e0a4018
Split CI pipelines (#23385)
- This PR attempts to split our various DB tests into separate
pipelines.
- It splits up some of the extra feature-related tests rather than
having most of them in the MySQL test.
- It disables the race detector for some of the pipelines as well, as it
can cause slower runs and is mostly redundant when the pipelines just
swap DBs.
- It builds without SQLite support for any of the non-SQLite pipelines.
- It moves the e2e test to using SQLite rather than PG (partially
because I moved the minio tests to PG and that mucked up the test
config, and partially because it avoids another running service)
- It splits up the `go mod download` task in the Makefile from the tool
installation, as the tools are only needed in the compliance pipeline.
(Arguably even some of the tools aren't needed there, but that could be
a follow-up PR)
- SQLite is now the only arm64 pipeline, moving PG back to amd64 which
can leverage autoscaler

Should resolve #22010 - one thing that wasn't changed here but is
mentioned in that issue, unit tests are needed in the same pipeline as
an integration test in order to form a complete coverage report (at
least as far as I could tell), so for now it remains in a pipeline with
a DB integration test.

Please let me know if I've inadvertently changed something that was how
it was on purpose.

---

I will say sometimes it's hard to pin down the average time, as a
pipeline could be waiting for a runner for X minutes and that brings the
total up by X minutes as well, but overall this does seem to be faster
on average.

---------

Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2023-03-10 01:13:17 -05:00
wxiaoguang ccb3851281
Add some comments for recent code (#22725)
When using the main branch, I found that some changed code didn't have
comments.

This PR adds some comments.
2023-02-02 11:39:38 -06:00
KN4CK3R b80538f37d
Disable test for incoming email (#22686)
Disable this test for the moment because the used imap container image
seems unstable which results in many failed CI builds.

Co-authored-by: Jason Song <i@wolfogre.com>
2023-01-31 14:56:22 +01:00
KN4CK3R fc037b4b82
Add support for incoming emails (#22056)
closes #13585
fixes #9067
fixes #2386
ref #6226
ref #6219
fixes #745

This PR adds support to process incoming emails to perform actions.
Currently I added handling of replies and unsubscribing from
issues/pulls. In contrast to #13585 the IMAP IDLE command is used
instead of polling which results (in my opinion 😉) in cleaner code.

Procedure:
- When sending an issue/pull reply email, a token is generated which is
present in the Reply-To and References header.
- IMAP IDLE waits until a new email arrives
- The token tells which action should be performed

A possible signature and/or reply gets stripped from the content.

I added a new service to the drone pipeline to test the receiving of
incoming mails. If we keep this in, we may test our outgoing emails too
in future.

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-01-14 23:57:10 +08:00
Kyle D c8ded77680
Kd/ci playwright go test (#20123)
* Add initial playwright config

* Simplify Makefile

* Simplify Makefile

* Use correct config files

* Update playwright settings

* Fix package-lock file

* Don't use test logger for e2e tests

* fix frontend lint

* Allow passing TEST_LOGGER variable

* Init postgres database

* use standard gitea env variables

* Update playwright

* update drone

* Move empty env var to commands

* Cleanup

* Move integrations to subfolder

* tests integrations to tests integraton

* Run e2e tests with go test

* Fix linting

* install CI deps

* Add files to ESlint

* Fix drone typo

* Don't log to console in CI

* Use go test http server

* Add build step before tests

* Move shared init function to common package

* fix drone

* Clean up tests

* Fix linting

* Better mocking for page + version string

* Cleanup test generation

* Remove dependency on gitea binary

* Fix linting

* add initial support for running specific tests

* Add ACCEPT_VISUAL variable

* don't require git-lfs

* Add initial documentation

* Review feedback

* Add logged in session test

* Attempt fixing drone race

* Cleanup and bump version

* Bump deps

* Review feedback

* simplify installation

* Fix ci

* Update install docs
2022-09-02 15:18:23 -04:00
Renamed from integrations/mysql.ini.tmpl (Browse further)