Purpose of this file

This shows the Development Processes work.
Current tool(s) with LLM Qwen-coder-plus, deepseek, etc.

Development Processes: The following is sorted with the latest items at the top.

a Demo to create a new application base the current architecture design

make a generic rvs-prxy system

Implement what we discussed we summary at Design_Doc/*.md files. You now implement this script code following all we discussed into auto-sys-deploy.sh beside the .env.development which as the parameters we aim to passing in.
- 1. I rename the file name of implmentation of script from instructions.md to implement_script.md; 2) i have a wrong assumption before as at any computer. It is wrong, end-user runs the script (we are now workinig on it) must be same computer to the mentioned service which opens the port "INTERNAL_SERVICE_PORT", and connect to it by docker network "DOCKER_NETWORK"; 3) i do not download manually, i aim to run the script by this way as download and run: ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy or another server as ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13072 ubuntu@cmty-repos1.ivantito.com "cat /wws/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy, 4) i mentioned end-user must well prepare the .env* (using .env.development as example at this moment); 5) modify the auto-sys-deploy.sh and corresponding doc. at Design_Doc/*.md files.
- Reference Design_Doc\auto-sys-deploy-ref.sh and Design_Doc\update-compose-config-ref.sh, We merged the these two script into ONE script auto-sys-deploy.sh. Re-verify the merge of code to form finally auto-sys-deploy.sh. After this merge, then we do not need SCRIPT_PATH and DOWNLOAD_SOURCE any more, remove these two variables from auto-sys-deploy.sh and any documents.
- i read the code auto-sys-deploy.sh which has [[ "$command" == "setup-frpc" ]], It is enoough for [[ "$command" == "setup-rvs-prxy" ]] at this moment. So remove [[ "$command" == "setup-frpc" ]] section and corresponding code. Update corresponding doc. too.
- I am testing the auto-sys-deploy.sh step by step: 1) In Design_Doc/procedure.md, i modified the "Run the Deployment Script" as ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" | bash -s -- setup-rvs-prxy which i tested as i need to have | between the download script and bash. 2) i find volume name as rad-git-sshd-itl-frpc-wacraftlink-frpc, the docker volume name should be ${THIS_INSTANCE}-frpc, where INSTANCE_NAME from .env.development, i.e. wacraftlink-frpc. 3) Further Proxy settings, eg. "First proxy (rad)" or "Web Proxy Configuration", it is application depending, do not create them always. So, only detect .env* has corresponding value, such as: PROXY_WEB_PORT which format must be PROXY_ANY_NAME_PORT, then dynamically add it, if no mentioned at .env*, that means this application does not request proxy port.
- I am testing the auto-sys-deploy.sh step by step: 1) it comes out this issue:
```
...
[INFO] Extracted FRP server address: 199.241.138.188
[ERROR] Failed to extract remote ports from configuration
[INFO] Skipping proxy server configuration update
...
```
  - From Design_Doc\procedure.md -> "Step 7: Configure Proxy Settings", it mentions "Dynamic Proxy Configuration", so modify the code at auto-sys-deploy.sh -> setup_rvs_prxy() -> "# Extract remote ports" section -> handles "rad_port ssh_port web_port", but actually, it should not fixed these three ports, it should be dynamically handles those port(s) depends on PROXY_*_PORT environment variables from .env*. So modify the auto-sys-deploy.sh code, and the Design_Doc\procedure.md -> "Step 7: Configure Proxy Settings" which should be removed the i) Rad Proxy Configuration, ii) SSH Proxy Configuration, iii) Web Proxy Configuration. You also scan all other documents which should remove these three items too.
  - From auto-sys-deploy.sh -> setup_rvs_prxy() -> "# Check if we have at least one port to configure" section -> handles "if [ ${#all_ports[@]} -ge 3 ];" which should not be -ge 3, improve the code.
  - I find the code and document are stukk glue to "PROXY_RAD_PORT", "PROXY_SSH_PORT" and "PROXY_WEB_PORT", not using dynamical way, eg. 1) generate_frpc_config() -> "# Check for common proxy variables from .env file", this should be replaced by "# Process PROXY_*_PORT variables if they exist in environment", 2) at Design_Doc/procedure.md -> "Step 7: Configure Proxy Settings", still have "Default rad proxy", "Default SSH proxy", "Default web proxy", they should be replaced by PROXY_*_PORT environment variables. Improce the code and document.
  - I find update_proxy_server_config() twice times at auto-sys-deploy.sh, so, creating "/tmp/update-compose-$THIS_INSTANCE.sh" twice too, analysis and improve the code.
  - i find the current format of cat /from/frpc.toml
```
serverAddr = "199.241.138.188"
serverPort = 16999

[[proxies]]
name = "web-wacraftlink"
type = "tcp"
localPort =
localIP = "127.0.0.1"
remotePort = 3000:12077
```
    - it should be
```
serverAddr = "199.241.138.188"
serverPort = 16999

[[proxies]]
name = "web-wacraftlink"
type = "tcp"
localPort = 3000
localIP = "127.0.0.1"
remotePort = 12077
```
      which 3000 be moved to localPort, and remotePort as a poprt number only, no :. Improve the code.
  - Remind to keep the .env file at last, since the .env is this setup all parameters.
- i have following issue of "Invalid proxy configuration format"
```
`ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" | bash -s -- setup-rvs-prxy`
...
[INFO] Using FRP server port: 16999
[INFO] Adding web proxy: local port 3000 to remote port 12077
[ERROR] Invalid proxy configuration format: web-:3000:12077
[INFO] Creating/updating frpc.toml configuration
[INFO] FRP client configuration created successfully
...
```
  - When Error occur, should stop any further processes.
- Inside auto-sys-deploy.sh, it has update_ports_config() and update_ports_config(), i find it seems duplicate code, these codes should be run at FRP_SERVER_ADDR, so these functions should be located inside "$temp_script", but i find another code as one of general function of auto-sys-deploy.sh. Analysis the code, and remove the duplicating and dead code.
  - For check_ports_config_status(), check_labels_config_status(), and find_section_insertion_point() are same situation as duplicate code. Analysis the code, and remove the duplicating and dead code.
- From .env* -> PROXY_*_PORT, it creates this currently:
```
# XXXXX Self-hosting ports ====
- "12077:12077/tcp"
- "t1.wacraftlink.com.hk:t1.wacraftlink.com.hk/tcp"
# =========================
```
  In current case .env.development as PROXY_WEB_PORT=12077, so comes out about, however, it has a issue of - "t1.wacraftlink.com.hk:t1.wacraftlink.com.hk/tcp" which should not exist, only - "12077:12077/tcp" is enough due to .env.development as PROXY_WEB_PORT=12077 only.
- When i cat frpc.toml
```
serverAddr = "199.241.138.188"
serverPort = 16999

[[proxies]]
name = "web-wacraftlink"
type = "tcp"
localPort = 3000
localIP = "127.0.0.1"
remotePort = 12077
```
  - for localIP, it should not use IP, it should use docker container name, since we are use docker network. So the container should follow this rule as ${INSTANCE_NAME}-app where ${INSTANCE_NAME} from .env*. Modify the code to prepare a right frpc.toml.
- From .env*, it has ${DOCKER_NETWORK} variable, however, i do not fin it to apply into docker run ... cmd applying into the frpc container. Add it back.
- From auto-sys-deploy.sh, it has "Proceed with deployment? (OK/Cancel) [Cancel]: OK", disable the part of code, do not remove it, only disable it, do not need to ask any more for operation smoothly;
- i find the content of "$temp_script" is duplicated, 1st item as "746 to 1126" and 2nd item as "1127 to 1507". Verify these two items, then merge the content, finally remove the last one.
  - i find seems duplicated function restart_docker_compose() exists twice, verify do we need to duplicate it?
  - ~~update_ports_config() function seems exist twice times inside the "$temp_script", verify it which one should be right, then remove the duplicated one.~~
- From .env*, we have FRP_SERVER_ADDR variable, i aim to add a FRP_SERVER_USER to instead the currently hardcode of teenfuture user.
- For generate_frpc_config() -> handle frpc.toml content -> config_content+="localIP = \"${THIS_INSTANCE}-app\"\n", i aim to change from ${THIS_INSTANCE}-app to ${SERVICE_DOCKER_CONTAINER_NAME} from .env*. I have already updated the .env.development and .env.example. You update the auto-sys-deploy.sh code.
- From .env* -> PROXY_*_PORT, it creates this currently:
```
# XXXXX Self-hosting ports ====
- "12077:12077/tcp"
# =========================
```
  - if it is PROXY_WEB_PORT, it mentions as WEB, we do not care the port number even other than 80 and 443, we still skip the section "# XXXXX Self-hosting ports ====", that means no this section present at apps/zzthis/docker-compose.app.yml, otherwise other PROXY_XXXX_PORT which is XXXX is not WEB, then keep this section.
  - I am in the case of PROXY_WEB_PORT=12077 from .env.development now, but this section "# XXXXX Self-hosting ports ====" is still exist.
  - i review the auto-sys-deploy.sh code -> update_ports_config() -> handle local INSTANCE_MARKER ..., it does not add any logic matching the above i just mentioned. Modify the code.
  - if it is PROXY_WEB_PORT, since it is web, so should be shared common pport 443 and 80, so, the above section "# XXXXX Self-hosting ports ====" can skip it. or other PROXY_*_PORT except PROXY_WEB_PORT, should still keep this section.
- ~~Currently it shows this command cd /data/ssd/rvs-prxy/frps/ && ./stop.sh && ./start.sh for manually to execute.~~ For manually testing as better, since this is production proxy, cannot always restart.

make a generic doc. for this arch.

Reference Design_Doc\auto-sys-deploy-ref.sh and Design_Doc\update-compose-config-ref.sh, the current procedure as below:
- Download from cmty-repos2.ivantito.com: ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13076 ubuntu@cmty-repos2.ivantito.com "cat /repos/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy
Download from cmty-repos1.ivantito.com: ssh -i ~/.ssh/itl-devs1 -o StrictHostKeyChecking=accept-new -p 13072 ubuntu@cmty-repos1.ivantito.com "cat /wws/repos/cmptr-infra/rvs-prxy/auto-sys-deploy.sh" bash -s -- setup-rvs-prxy
- Windows -- not yet implement
- You see the DOWNLOAD_SOURCE and SCRIPT_PATH are same url and path to how to download auto-sys-deploy.sh. Give me better method as shorter of whole command. Otherwise, see is it good to merge auto-sys-deploy.sh and update-compose-config.sh together, then the length of command can be shorter.
- For .env file, it must be created by end-user at their any local computer with same path of auto-sys-deploy.sh.
- Result -- shorter_cmd.md
explain to me the purpose of "DOWNLOAD_SOURCE" from our previous discussion.
- Based on our previous discussion and the architecture documentation, the purpose of the DOWNLOAD_SOURCE environment variable is:
- The DOWNLOAD_SOURCE is used to specify the source location from which the update-compose-config.sh script can download the necessary configuration update script to run on the proxy server.
- Looking at the architecture flow in action section of the documentation, when the auto-sys-deploy.sh script executes, it eventually calls the update_proxy_server_config function which:
1. Creates a temporary script to run on the proxy server
2. This script needs to download and execute the update-compose-config.sh script on the proxy server
3. The DOWNLOAD_SOURCE variable defines where this script should be downloaded from
- The download mechanism supports different source types:
- HTTP/HTTPS URLs (e.g., https://raw.githubusercontent.com/username/repository/main)
- SSH commands or local paths
- The script can download itself using tools like curl or wget from the specified source
- This design allows the system to be flexible about where the configuration update script is stored, enabling scenarios where:
- The script is stored in a remote repository (like GitHub/GitLab)
- The script is accessed via SSH from another server
- The script is available from a local file path
- The DOWNLOAD_SOURCE enables the deployment system to remotely update the proxy server's configuration without requiring the script to be pre-staged on the proxy server, making the architecture more flexible and deployable from various environments.
For connecting the internal service, i aim to narrow down interface as: i assume the service is using docker container with expose port, eg. 3000, so this "Generic Reverse Proxy" is connecting into another service docker container by expose port. This expose port number as one of parameter at .env*. As well, connect into the service docker container by docker network which mentioned before. Add this concept into Design/architecture.md and Design/procedure.md.
From Design/procedure.md, i aim to add .env or .env.development or .env.production as default values instead of the questions during the script to run. If some parameter(s) is / are missed at .env*, then will still ask question(s). Every answered question(s) will auto. add into .env* as default value for next time. Add this concept into Design/architecture.md and Design/procedure.md.
- create a demo .env.example as reference to let end-user easy to follow.
I need you to guide me step by step to make my internal service connecting to my public domain.
- Reference from following doc.:
  - Design/architecture.md
  - Design/auto-sys-deploy.sh.md
  - Design/update-compose-config.sh.md
  - Design/auto-sys-deploy-ref.sh
  - Design/update-compose-config-ref.sh
  - Design/DevProc.md
- Finally, i need a worked setting following above guideline, and a step by step guideline (Design/procedure.md) for next time, next application.
Add more detail into Design_Doc/architecture.md: 1) Add block diagram(s) as visualization of the description of Design_Doc/architecture.md, 2) attached auto-sys-deploy-ref.sh and update-compose-config-ref.sh with corresponding auto-sys-deploy.sh.md and update-compose-config.sh.md as an example, using real example parameters, outputs, etc. during the explanation.
Base your understand of auto-sys-deploy-ref.sh and update-compose-config-ref.sh with your mardown documents at Design_Doc/. I aim to have a summary document at Design_Doc/architecture.md, it talks about the generic concept of this architecture which can apply to any applications with clearly listing the parameters and corresponding outputs. For example, currently auto-sys-deploy-ref.sh and update-compose-config-ref.sh:
- Prerequisites
  - Using FRP (fast reverse proxy) service with frpc and frps
  - using Traefik
  - Docker and Docker Compose installed
  - SSH key at ~/.ssh/itl-devs1 for proxy server access
  - DOWNLOAD_SOURCE environment variable set
  - User permissions for Docker commands
  - SSH keys for authentication (~/.ssh/authorized_keys and ~/.ssh/devs1)
- parameters
  - Instance Name: Alphanumeric, hyphens, and underscores only
  - Docker Network: Existing Docker network or custom network name
  - FRP Server Address: IP address of the FRP server
  - FRP Server Port: Default is 16999
  - Proxy Ports: Configurable remote ports for rad (default 8776), ssh (default 22), and web (default 80) proxies
  - application port(s)
  - Domain name for Traefik routing rules
- outcomes
  - public domain connected to a internal service.
- etc.

Understand how previous to do

Analysis and explain these two scripts to me: auto-sys-deploy-ref.sh and update-compose-config-ref.sh; Write your explaination into Design_Doc/auto-sys-deploy.sh.md and Design_Doc/update-compose-config.sh.md corresponding.

17 KiB Raw Blame History

Purpose of this file

Development Processes: The following is sorted with the latest items at the top.

a Demo to create a new application base the current architecture design

make a generic rvs-prxy system

make a generic doc. for this arch.

Understand how previous to do

17 KiB

Raw Blame History