183 lines
17 KiB
Markdown
183 lines
17 KiB
Markdown
# Purpose of this file
|
|
- This shows the Development Processes work.
|
|
- Current tool(s) with LLM Qwen-coder-plus, deepseek, etc.
|
|
|
|
# Development Processes: The following is sorted with the latest items at the top.
|
|
|
|
## a Demo to create a new application base the current architecture design
|
|
|
|
## make a generic rvs-prxy system
|
|
- Implement what we discussed we summary at `Design_Doc/*.md` files. You now implement this script code following all we discussed into `auto-sys-deploy.sh` beside the `.env.development` which as the parameters we aim to passing in.
|
|
- 1) I rename the file name of implmentation of script from `instructions.md` to `implement_script.md`; 2) i have a wrong assumption before as at any computer. It is wrong, end-user runs the script (we are now workinig on it) must be same computer to the mentioned service which opens the port "INTERNAL_SERVICE_PORT", and connect to it by docker network "DOCKER_NETWORK"; 3) i do not download manually, i aim to run the script by this way as download and run: `ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy` or another server as `ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13072 ubuntu@cmty-repos1.ivantito.com "cat /wws/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy`, 4) i mentioned end-user must well prepare the `.env*` (using .env.development as example at this moment); 5) modify the `auto-sys-deploy.sh` and corresponding doc. at `Design_Doc/*.md` files.
|
|
- Reference `Design_Doc\auto-sys-deploy-ref.sh` and `Design_Doc\update-compose-config-ref.sh`, We merged the these two script into ONE script `auto-sys-deploy.sh`. Re-verify the merge of code to form finally `auto-sys-deploy.sh`. After this merge, then we do not need `SCRIPT_PATH` and `DOWNLOAD_SOURCE` any more, remove these two variables from `auto-sys-deploy.sh` and any documents.
|
|
- i read the code `auto-sys-deploy.sh` which has `[[ "$command" == "setup-frpc" ]]`, It is enoough for `[[ "$command" == "setup-rvs-prxy" ]]` at this moment. So remove `[[ "$command" == "setup-frpc" ]]` section and corresponding code. Update corresponding doc. too.
|
|
- I am testing the `auto-sys-deploy.sh` step by step: 1) In `Design_Doc/procedure.md`, i modified the "Run the Deployment Script" as `ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" | bash -s -- setup-rvs-prxy` which i tested as i need to have `|` between the download script and `bash`. 2) i find volume name as `rad-git-sshd-itl-frpc-wacraftlink-frpc`, the docker volume name should be ${THIS_INSTANCE}-frpc, where `INSTANCE_NAME` from `.env.development`, i.e. `wacraftlink-frpc`. 3) Further Proxy settings, eg. "First proxy (rad)" or "Web Proxy Configuration", it is application depending, do not create them always. So, only detect `.env*` has corresponding value, such as: `PROXY_WEB_PORT` which format must be `PROXY_ANY_NAME_PORT`, then dynamically add it, if no mentioned at `.env*`, that means this application does not request proxy port.
|
|
- I am testing the `auto-sys-deploy.sh` step by step: 1) it comes out this issue:
|
|
```
|
|
...
|
|
[INFO] Extracted FRP server address: 199.241.138.188
|
|
[ERROR] Failed to extract remote ports from configuration
|
|
[INFO] Skipping proxy server configuration update
|
|
...
|
|
```
|
|
- From `Design_Doc\procedure.md` -> "Step 7: Configure Proxy Settings", it mentions "Dynamic Proxy Configuration", so modify the code at `auto-sys-deploy.sh` -> setup_rvs_prxy() -> "# Extract remote ports" section -> handles "rad_port ssh_port web_port", but actually, it should not fixed these three ports, it should be dynamically handles those port(s) depends on `PROXY_*_PORT` environment variables from `.env*`. So modify the `auto-sys-deploy.sh` code, and the `Design_Doc\procedure.md` -> "Step 7: Configure Proxy Settings" which should be removed the i) Rad Proxy Configuration, ii) SSH Proxy Configuration, iii) Web Proxy Configuration. You also scan all other documents which should remove these three items too.
|
|
- From `auto-sys-deploy.sh` -> setup_rvs_prxy() -> "# Check if we have at least one port to configure" section -> handles "if [ ${#all_ports[@]} -ge 3 ];" which should not be `-ge 3`, improve the code.
|
|
- I find the code and document are stukk glue to "PROXY_RAD_PORT", "PROXY_SSH_PORT" and "PROXY_WEB_PORT", not using dynamical way, eg. 1) generate_frpc_config() -> "# Check for common proxy variables from .env file", this should be replaced by "# Process PROXY_*_PORT variables if they exist in environment", 2) at `Design_Doc/procedure.md` -> "Step 7: Configure Proxy Settings", still have "Default rad proxy", "Default SSH proxy", "Default web proxy", they should be replaced by `PROXY_*_PORT` environment variables. Improce the code and document.
|
|
- I find `update_proxy_server_config()` twice times at `auto-sys-deploy.sh`, so, creating "/tmp/update-compose-$THIS_INSTANCE.sh" twice too, analysis and improve the code.
|
|
- i find the current format of `cat /from/frpc.toml`
|
|
```
|
|
serverAddr = "199.241.138.188"
|
|
serverPort = 16999
|
|
|
|
[[proxies]]
|
|
name = "web-wacraftlink"
|
|
type = "tcp"
|
|
localPort =
|
|
localIP = "127.0.0.1"
|
|
remotePort = 3000:12077
|
|
```
|
|
- it should be
|
|
```
|
|
serverAddr = "199.241.138.188"
|
|
serverPort = 16999
|
|
|
|
[[proxies]]
|
|
name = "web-wacraftlink"
|
|
type = "tcp"
|
|
localPort = 3000
|
|
localIP = "127.0.0.1"
|
|
remotePort = 12077
|
|
```
|
|
which 3000 be moved to localPort, and remotePort as a poprt number only, no `:`. Improve the code.
|
|
- Remind to keep the `.env` file at last, since the `.env` is this setup all parameters.
|
|
- i have following issue of "Invalid proxy configuration format"
|
|
```
|
|
`ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" | bash -s -- setup-rvs-prxy`
|
|
...
|
|
[INFO] Using FRP server port: 16999
|
|
[INFO] Adding web proxy: local port 3000 to remote port 12077
|
|
[ERROR] Invalid proxy configuration format: web-:3000:12077
|
|
[INFO] Creating/updating frpc.toml configuration
|
|
[INFO] FRP client configuration created successfully
|
|
...
|
|
```
|
|
- When Error occur, should stop any further processes.
|
|
- Inside `auto-sys-deploy.sh`, it has update_ports_config() and update_ports_config(), i find it seems duplicate code, these codes should be run at FRP_SERVER_ADDR, so these functions should be located inside "$temp_script", but i find another code as one of general function of `auto-sys-deploy.sh`. Analysis the code, and remove the duplicating and dead code.
|
|
- For check_ports_config_status(), check_labels_config_status(), and find_section_insertion_point() are same situation as duplicate code. Analysis the code, and remove the duplicating and dead code.
|
|
- From `.env*` -> `PROXY_*_PORT`, it creates this currently:
|
|
```
|
|
# XXXXX Self-hosting ports ====
|
|
- "12077:12077/tcp"
|
|
- "t1.wacraftlink.com.hk:t1.wacraftlink.com.hk/tcp"
|
|
# =========================
|
|
```
|
|
In current case `.env.development` as `PROXY_WEB_PORT=12077`, so comes out about, however, it has a issue of `- "t1.wacraftlink.com.hk:t1.wacraftlink.com.hk/tcp"` which should not exist, only `- "12077:12077/tcp"` is enough due to `.env.development` as `PROXY_WEB_PORT=12077` only.
|
|
- When i `cat frpc.toml`
|
|
```
|
|
serverAddr = "199.241.138.188"
|
|
serverPort = 16999
|
|
|
|
[[proxies]]
|
|
name = "web-wacraftlink"
|
|
type = "tcp"
|
|
localPort = 3000
|
|
localIP = "127.0.0.1"
|
|
remotePort = 12077
|
|
```
|
|
- for localIP, it should not use IP, it should use docker container name, since we are use docker network. So the container should follow this rule as `${INSTANCE_NAME}-app` where `${INSTANCE_NAME}` from `.env*`. Modify the code to prepare a right `frpc.toml`.
|
|
- From `.env*`, it has `${DOCKER_NETWORK}` variable, however, i do not fin it to apply into `docker run ...` cmd applying into the frpc container. Add it back.
|
|
- From `auto-sys-deploy.sh`, it has "Proceed with deployment? (OK/Cancel) [Cancel]: OK", disable the part of code, do not remove it, only disable it, do not need to ask any more for operation smoothly;
|
|
- i find the content of "$temp_script" is duplicated, 1st item as "746 to 1126" and 2nd item as "1127 to 1507". Verify these two items, then merge the content, finally remove the last one.
|
|
- i find seems duplicated function `restart_docker_compose()` exists twice, verify do we need to duplicate it?
|
|
- ~~`update_ports_config()` function seems exist twice times inside the "$temp_script", verify it which one should be right, then remove the duplicated one.~~
|
|
- From `.env*`, we have `FRP_SERVER_ADDR` variable, i aim to add a `FRP_SERVER_USER` to instead the currently hardcode of `teenfuture` user.
|
|
- For `generate_frpc_config()` -> handle `frpc.toml` content -> `config_content+="localIP = \"${THIS_INSTANCE}-app\"\n"`, i aim to change from `${THIS_INSTANCE}-app` to ${SERVICE_DOCKER_CONTAINER_NAME} from `.env*`. I have already updated the `.env.development` and `.env.example`. You update the `auto-sys-deploy.sh` code.
|
|
- From `.env*` -> `PROXY_*_PORT`, it creates this currently:
|
|
```
|
|
# XXXXX Self-hosting ports ====
|
|
- "12077:12077/tcp"
|
|
# =========================
|
|
```
|
|
- if it is `PROXY_WEB_PORT`, it mentions as `WEB`, we do not care the port number even other than 80 and 443, we still skip the section "# XXXXX Self-hosting ports ====", that means no this section present at `apps/zzthis/docker-compose.app.yml`, otherwise other `PROXY_XXXX_PORT` which is `XXXX` is not `WEB`, then keep this section.
|
|
- I am in the case of `PROXY_WEB_PORT=12077` from `.env.development` now, but this section "# XXXXX Self-hosting ports ====" is still exist.
|
|
- i review the `auto-sys-deploy.sh` code -> `update_ports_config()` -> handle `local INSTANCE_MARKER ...`, it does not add any logic matching the above i just mentioned. Modify the code.
|
|
- ~~if it is `PROXY_WEB_PORT`, since it is web, so should be shared common pport 443 and 80, so, the above section "# XXXXX Self-hosting ports ====" can skip it. or other `PROXY_*_PORT` except `PROXY_WEB_PORT`, should still keep this section.~~
|
|
- ~~Currently it shows this command `cd /data/ssd/rvs-prxy/frps/ && ./stop.sh && ./start.sh` for manually to execute.~~ For manually testing as better, since this is production proxy, cannot always restart.
|
|
|
|
|
|
|
|
|
|
## make a generic doc. for this arch.
|
|
- Reference `Design_Doc\auto-sys-deploy-ref.sh` and `Design_Doc\update-compose-config-ref.sh`, the current procedure as below:
|
|
- Download from cmty-repos2.ivantito.com: `ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy`
|
|
|
|
- Download from cmty-repos1.ivantito.com: `ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13072 ubuntu@cmty-repos1.ivantito.com "cat /wws/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy`
|
|
- Windows -- not yet implement
|
|
- You see the `DOWNLOAD_SOURCE` and `SCRIPT_PATH` are same url and path to how to download `auto-sys-deploy.sh`. Give me better method as shorter of whole command. Otherwise, see is it good to merge `auto-sys-deploy.sh` and `update-compose-config.sh` together, then the length of command can be shorter.
|
|
- For `.env` file, it must be created by end-user at their any local computer with same path of `auto-sys-deploy.sh`.
|
|
- Result -- shorter_cmd.md
|
|
|
|
|
|
- explain to me the purpose of "DOWNLOAD_SOURCE" from our previous discussion.
|
|
- Based on our previous discussion and the architecture documentation, the purpose of the DOWNLOAD_SOURCE environment variable is:
|
|
|
|
- The DOWNLOAD_SOURCE is used to specify the source location from which the update-compose-config.sh script can download the necessary configuration update script to run on the proxy server.
|
|
|
|
- Looking at the architecture flow in action section of the documentation, when the auto-sys-deploy.sh script executes, it eventually calls the update_proxy_server_config function which:
|
|
|
|
1. Creates a temporary script to run on the proxy server
|
|
2. This script needs to download and execute the update-compose-config.sh script on the proxy server
|
|
3. The DOWNLOAD_SOURCE variable defines where this script should be downloaded from
|
|
|
|
- The download mechanism supports different source types:
|
|
- HTTP/HTTPS URLs (e.g., https://raw.githubusercontent.com/username/repository/main)
|
|
- SSH commands or local paths
|
|
- The script can download itself using tools like curl or wget from the specified source
|
|
|
|
- This design allows the system to be flexible about where the configuration update script is stored, enabling scenarios where:
|
|
- The script is stored in a remote repository (like GitHub/GitLab)
|
|
- The script is accessed via SSH from another server
|
|
- The script is available from a local file path
|
|
|
|
- The DOWNLOAD_SOURCE enables the deployment system to remotely update the proxy server's configuration without requiring the script to be pre-staged on the proxy server, making the architecture more flexible and deployable from various environments.
|
|
|
|
- For connecting the internal service, i aim to narrow down interface as: i assume the service is using docker container with expose port, eg. 3000, so this "Generic Reverse Proxy" is connecting into another service docker container by expose port. This expose port number as one of parameter at `.env*`. As well, connect into the service docker container by docker network which mentioned before. Add this concept into `Design/architecture.md` and `Design/procedure.md`.
|
|
|
|
- From `Design/procedure.md`, i aim to add `.env` or `.env.development` or `.env.production` as default values instead of the questions during the script to run. If some parameter(s) is / are missed at `.env*`, then will still ask question(s). Every answered question(s) will auto. add into `.env*` as default value for next time. Add this concept into `Design/architecture.md` and `Design/procedure.md`.
|
|
- create a demo `.env.example` as reference to let end-user easy to follow.
|
|
|
|
- I need you to guide me step by step to make my internal service connecting to my public domain.
|
|
- Reference from following doc.:
|
|
- `Design/architecture.md`
|
|
- `Design/auto-sys-deploy.sh.md`
|
|
- `Design/update-compose-config.sh.md`
|
|
- `Design/auto-sys-deploy-ref.sh`
|
|
- `Design/update-compose-config-ref.sh`
|
|
- `Design/DevProc.md`
|
|
- Finally, i need a worked setting following above guideline, and a step by step guideline (`Design/procedure.md`) for next time, next application.
|
|
|
|
- Add more detail into `Design_Doc/architecture.md`: 1) Add block diagram(s) as visualization of the description of `Design_Doc/architecture.md`, 2) attached `auto-sys-deploy-ref.sh` and `update-compose-config-ref.sh` with corresponding `auto-sys-deploy.sh.md` and `update-compose-config.sh.md` as an example, using real example parameters, outputs, etc. during the explanation.
|
|
|
|
- Base your understand of `auto-sys-deploy-ref.sh` and `update-compose-config-ref.sh` with your mardown documents at `Design_Doc/`. I aim to have a summary document at `Design_Doc/architecture.md`, it talks about the generic concept of this architecture which can apply to any applications with clearly listing the parameters and corresponding outputs. For example, currently `auto-sys-deploy-ref.sh` and `update-compose-config-ref.sh`:
|
|
- Prerequisites
|
|
- Using FRP (fast reverse proxy) service with frpc and frps
|
|
- using Traefik
|
|
- Docker and Docker Compose installed
|
|
- SSH key at `~/.ssh/itl-devs1` for proxy server access
|
|
- `DOWNLOAD_SOURCE` environment variable set
|
|
- User permissions for Docker commands
|
|
- SSH keys for authentication (`~/.ssh/authorized_keys` and `~/.ssh/devs1`)
|
|
- parameters
|
|
- **Instance Name**: Alphanumeric, hyphens, and underscores only
|
|
- **Docker Network**: Existing Docker network or custom network name
|
|
- **FRP Server Address**: IP address of the FRP server
|
|
- **FRP Server Port**: Default is 16999
|
|
- **Proxy Ports**: Configurable remote ports for rad (default 8776), ssh (default 22), and web (default 80) proxies
|
|
- application port(s)
|
|
- Domain name for Traefik routing rules
|
|
- outcomes
|
|
- public domain connected to a internal service.
|
|
- etc.
|
|
|
|
## Understand how previous to do
|
|
- Analysis and explain these two scripts to me: `auto-sys-deploy-ref.sh` and `update-compose-config-ref.sh`; Write your explaination into `Design_Doc/auto-sys-deploy.sh.md` and `Design_Doc/update-compose-config.sh.md` corresponding. |