1
0
mirror of https://github.com/sasjs/server.git synced 2026-07-23 21:25:29 +00:00

Compare commits

..

34 Commits

Author SHA1 Message Date
semantic-release-bot 2fe733d02f chore(release): 0.39.5 [skip ci]
## [0.39.5](https://github.com/sasjs/server/compare/v0.39.4...v0.39.5) (2026-07-14)

### Bug Fixes

* **api:** harden mock SAS executable and timeouts for CI ([67fce47](https://github.com/sasjs/server/commit/67fce475a3c53b3c20919902edb45dce809dffb9))
* **api:** return prompt error response instead of hanging when a SAS session fails ([f84af4a](https://github.com/sasjs/server/commit/f84af4ac0639625fee4508e764f0eed07d7b4e8a))
* fix code.spec.ts failing on a fresh checkout (CI) ([0af6b63](https://github.com/sasjs/server/commit/0af6b63ffa770bfa2b77505d74778be517f8d631))
2026-07-14 06:38:40 +00:00
Allan Bowe 63086a8a0b Merge pull request #389 from sasjs/issue-388
fix(api): return prompt error response instead of hanging when a SAS …
2026-07-14 07:35:39 +01:00
YuryShkoda 0af6b63ffa fix: fix code.spec.ts failing on a fresh checkout (CI) 2026-07-14 09:20:35 +03:00
YuryShkoda 67fce475a3 fix(api): harden mock SAS executable and timeouts for CI
- wrap the file read/write in a small retry() helper instead of
  letting an uncaught exception (e.g. a rename racing an existsSync
  check) crash the process with a non-zero exit indistinguishable
  from a genuine SAS failure
- bump the internal give-up deadline from 1500ms to 8000ms
- bump the corresponding Jest test timeouts from 15000ms to 30000ms to match
2026-07-14 08:41:35 +03:00
YuryShkoda f84af4ac06 fix(api): return prompt error response instead of hanging when a SAS session fails
The SAS-runtime poll loop in processProgram only checked for
SessionState.completed, so a session that failed (e.g. via %abort;)
left the loop spinning on delay(50) forever - the request never
resolved, even though the SAS log had already been written. Split a single completed-on-either-outcome
flag into separate completed/failed states without updating this loop.

- processProgram now throws when session.state becomes 'failed'
- Execution.ts catches that, reads the complete log (guaranteed
  complete since the session's process has already exited by then),
  and throws a SessionExecutionError carrying it
- stp.ts/code.ts surface { ..., log } in the HTTP error response
- add unit tests for the poll loop and the log-enrichment logic, plus
  a mock SAS executable + end-to-end test exercising the real
  session/process pipeline without needing a SAS install
- add docs/diagrams covering the session lifecycle and the SAS
  execution handshake mechanism, for future context
2026-07-13 13:46:36 +03:00
semantic-release-bot 8b2a6155a9 chore(release): 0.39.4 [skip ci]
## [0.39.4](https://github.com/sasjs/server/compare/v0.39.3...v0.39.4) (2025-12-21)

### Bug Fixes

* **deps:** bump dependencies and resolve audit issues ([4f78202](https://github.com/sasjs/server/commit/4f782025dbcdfcbae6ca1fabb42ce1bc385e0162))
2025-12-21 01:27:37 +00:00
Trevor Moody a56c0b0340 Merge pull request #386 from sasjs/depBumps_20251221
fix(deps): bump dependencies and resolve audit issues
2025-12-21 01:24:31 +00:00
Trevor Moody 02fe79d4d7 chore(deps): full package-lock rebuild 2025-12-21 01:22:35 +00:00
Trevor Moody 00a107babd chore(deps): rebuilt api/package-json to sync 2025-12-21 01:15:05 +00:00
Trevor Moody 4f782025db fix(deps): bump dependencies and resolve audit issues 2025-12-21 00:27:24 +00:00
semantic-release-bot 8b5abcd661 chore(release): 0.39.3 [skip ci]
## [0.39.3](https://github.com/sasjs/server/compare/v0.39.2...v0.39.3) (2025-11-25)

### Bug Fixes

* (deps) bump @sasjs/core to 4.59.7 ([ab96653](https://github.com/sasjs/server/commit/ab966535642d08d4e8e984007b98c8fdffbe30f7))
* (deps) rerun npm i to sync ([225f381](https://github.com/sasjs/server/commit/225f381bdf8ad5aa2af8d75648df1dd5175e12e0))
2025-11-25 12:39:15 +00:00
Allan Bowe 48e8cb7b2d Merge pull request #385 from sasjs/bumpCore_20251125
fix: (deps) rerun npm i to sync
2025-11-25 12:36:18 +00:00
Trevor Moody 225f381bdf fix: (deps) rerun npm i to sync 2025-11-25 12:27:14 +00:00
Allan Bowe 3f49186e3b Merge pull request #384 from sasjs/bumpCore_20251125
fix: (deps) bump @sasjs/core to 4.59.7
2025-11-25 12:24:27 +00:00
Trevor Moody ab96653564 fix: (deps) bump @sasjs/core to 4.59.7 2025-11-25 12:18:50 +00:00
semantic-release-bot 471c28eaa2 chore(release): 0.39.2 [skip ci]
## [0.39.2](https://github.com/sasjs/server/compare/v0.39.1...v0.39.2) (2025-09-25)

### Bug Fixes

* addressing test fail ([e51b204](https://github.com/sasjs/server/commit/e51b20421adc1598ea267c79b1fb4dbc085f97b9))
* packages missmatch ([379ea60](https://github.com/sasjs/server/commit/379ea604bcb5686b5299fae6a32f759c45b275ea))
* type libs ([6d123c3](https://github.com/sasjs/server/commit/6d123c3e23628c1d703eaa13142c77f0da970a55))
* typescript errors ([631e956](https://github.com/sasjs/server/commit/631e95604b64b1a96f2abade659348618f3b00b2))
* typescript errors ([198cd79](https://github.com/sasjs/server/commit/198cd79354254511c21ac1acfbf7b6bcfdab2af7))
2025-09-25 16:57:01 +00:00
Allan Bowe 584ffe9e0e Merge pull request #383 from sasjs/npm_update_20250919
Npm update 20250919
2025-09-25 17:53:46 +01:00
M e51b20421a fix: addressing test fail 2025-09-25 13:49:32 +02:00
M 631e95604b fix: typescript errors 2025-09-25 13:40:10 +02:00
M 198cd79354 fix: typescript errors 2025-09-25 13:34:55 +02:00
M 379ea604bc fix: packages missmatch 2025-09-25 13:12:23 +02:00
M 9ffa403bcb chore: package-lock 2025-09-25 13:06:06 +02:00
M 6d123c3e23 fix: type libs 2025-09-25 13:03:47 +02:00
M dda1aadc67 chore(git): Merge branch 'main' into npm_update_20250919 2025-09-25 12:48:10 +02:00
M d47cf15cdb ci: ubuntu 22 2025-09-25 12:46:19 +02:00
Trevor Moody d0c7968d66 build: updated package dependencies for /web 2025-09-19 18:24:58 +01:00
Trevor Moody a5c99971cc build: server/api dependency update 2025-09-19 14:06:50 +01:00
semantic-release-bot c422e7f02e chore(release): 0.39.1 [skip ci]
## [0.39.1](https://github.com/sasjs/server/compare/v0.39.0...v0.39.1) (2025-03-13)

### Bug Fixes

* extra bit of sleep for file recognition ([f4768bf](https://github.com/sasjs/server/commit/f4768bffd3dbb2fe243966572ba74002024d96e1)), closes [#381](https://github.com/sasjs/server/issues/381)
2025-03-13 10:59:10 +00:00
Allan Bowe 02a993611c Merge pull request #382 from sasjs/381-add-slight-delay-to-enable-file-detection
fix: extra bit of sleep for file recognition
2025-03-13 10:56:13 +00:00
sabhas aca2fff4ac chore(workflow): run the build workflow on ubuntu 20.04 2025-03-13 15:50:23 +05:00
sabhas af1a386b13 chore(workflow): install openssl 1.1 in actions 2025-03-13 15:43:20 +05:00
Allan Bowe f5018ce1df chore: prettier fix 2025-03-12 17:33:02 +00:00
Allan Bowe 3529232f1f chore: whitespace removal 2025-03-12 17:29:02 +00:00
Allan Bowe f4768bffd3 fix: extra bit of sleep for file recognition
closes #381
2025-03-12 17:27:57 +00:00
31 changed files with 13118 additions and 24577 deletions
+1
View File
@@ -0,0 +1 @@
* text=auto eol=lf
+3 -3
View File
@@ -5,7 +5,7 @@ on:
jobs:
lint:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
strategy:
matrix:
@@ -28,7 +28,7 @@ jobs:
run: npm run lint-web
build-api:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
strategy:
matrix:
@@ -66,7 +66,7 @@ jobs:
CI: true
build-web:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
strategy:
matrix:
+1 -1
View File
@@ -7,7 +7,7 @@ on:
jobs:
release:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
strategy:
matrix:
+1
View File
@@ -0,0 +1 @@
ignore-scripts=true
+42
View File
@@ -1,3 +1,45 @@
## [0.39.5](https://github.com/sasjs/server/compare/v0.39.4...v0.39.5) (2026-07-14)
### Bug Fixes
* **api:** harden mock SAS executable and timeouts for CI ([67fce47](https://github.com/sasjs/server/commit/67fce475a3c53b3c20919902edb45dce809dffb9))
* **api:** return prompt error response instead of hanging when a SAS session fails ([f84af4a](https://github.com/sasjs/server/commit/f84af4ac0639625fee4508e764f0eed07d7b4e8a))
* fix code.spec.ts failing on a fresh checkout (CI) ([0af6b63](https://github.com/sasjs/server/commit/0af6b63ffa770bfa2b77505d74778be517f8d631))
## [0.39.4](https://github.com/sasjs/server/compare/v0.39.3...v0.39.4) (2025-12-21)
### Bug Fixes
* **deps:** bump dependencies and resolve audit issues ([4f78202](https://github.com/sasjs/server/commit/4f782025dbcdfcbae6ca1fabb42ce1bc385e0162))
## [0.39.3](https://github.com/sasjs/server/compare/v0.39.2...v0.39.3) (2025-11-25)
### Bug Fixes
* (deps) bump @sasjs/core to 4.59.7 ([ab96653](https://github.com/sasjs/server/commit/ab966535642d08d4e8e984007b98c8fdffbe30f7))
* (deps) rerun npm i to sync ([225f381](https://github.com/sasjs/server/commit/225f381bdf8ad5aa2af8d75648df1dd5175e12e0))
## [0.39.2](https://github.com/sasjs/server/compare/v0.39.1...v0.39.2) (2025-09-25)
### Bug Fixes
* addressing test fail ([e51b204](https://github.com/sasjs/server/commit/e51b20421adc1598ea267c79b1fb4dbc085f97b9))
* packages missmatch ([379ea60](https://github.com/sasjs/server/commit/379ea604bcb5686b5299fae6a32f759c45b275ea))
* type libs ([6d123c3](https://github.com/sasjs/server/commit/6d123c3e23628c1d703eaa13142c77f0da970a55))
* typescript errors ([631e956](https://github.com/sasjs/server/commit/631e95604b64b1a96f2abade659348618f3b00b2))
* typescript errors ([198cd79](https://github.com/sasjs/server/commit/198cd79354254511c21ac1acfbf7b6bcfdab2af7))
## [0.39.1](https://github.com/sasjs/server/compare/v0.39.0...v0.39.1) (2025-03-13)
### Bug Fixes
* extra bit of sleep for file recognition ([f4768bf](https://github.com/sasjs/server/commit/f4768bffd3dbb2fe243966572ba74002024d96e1)), closes [#381](https://github.com/sasjs/server/issues/381)
# [0.39.0](https://github.com/sasjs/server/compare/v0.38.0...v0.39.0) (2024-10-31)
@@ -0,0 +1,38 @@
# Code execution & session diagrams
Mermaid diagrams describing how `@sasjs/server` executes submitted code
(SAS/JS/PY/R) against pooled "sessions". Written for fast context-loading by
an AI coding agent: each diagram is self-contained, node/edge labels carry
the actual file:line references, and prose is kept to the minimum needed to
disambiguate the diagram.
| File | Covers |
|---|---|
| [session-lifecycle.md](session-lifecycle.md) | `SessionState` state machine; how it differs between the SAS runtime and the JS/PY/R runtimes |
| [sas-execution-handshake.md](sas-execution-handshake.md) | The SYSIN/AUTOEXEC file-swap mechanism a SAS session uses to turn one long-lived `sas` process into a single-use execution slot |
| [request-execution-flow.md](request-execution-flow.md) | End-to-end flowchart from HTTP request to response, covering session-pool acquisition and both runtime branches (SAS vs JS/PY/R) |
## Core concept
A "session" is not a request-scoped object. It is a pooled, reusable
execution slot with a filesystem folder (`session.path`) and a state
(`SessionState`). For the SAS runtime specifically, a session also owns one
real OS process, spawned at session-*creation* time and consumed by exactly
one code submission (see `sas-execution-handshake.md`). For JS/PY/R, a
session is just an ID + folder; the actual interpreter process is spawned
fresh per request inside `processProgram`.
## Key source files
- `api/src/controllers/internal/Session.ts` — session pool + lifecycle
(`SessionController`, `SASSessionController`), `SessionState` transitions.
- `api/src/controllers/internal/processProgram.ts` — writes the submitted
code into the session and drives it to completion/failure, per runtime.
- `api/src/controllers/internal/Execution.ts``ExecutionController`,
the entry point controllers call; acquires a session, calls
`processProgram`, reads back `log.log`/`webout.txt`/headers, builds the
HTTP response (or a `SessionExecutionError` on failure).
- `api/src/controllers/internal/create{SAS,JS,Python,R}Program.ts`
per-runtime code templating (wraps the user's submitted code with
boilerplate: variable injection, `_webout` redirection, etc).
- `api/src/types/Session.ts``SessionState` enum and `Session` interface.
@@ -0,0 +1,64 @@
# Request execution flow (end-to-end)
Full path from an inbound HTTP request to the HTTP response, covering
session-pool acquisition and the two structurally different runtime
branches: SAS (reuses a parked long-lived process) vs JS/PY/R (spawns a
fresh interpreter process per request).
```mermaid
flowchart TD
A["HTTP request<br/>POST /SASjsApi/code/execute<br/>or /SASjsApi/stp/execute"] --> B["Controller<br/>code.ts / stp.ts<br/>try/catch wrapper"]
B --> C["ExecutionController.executeProgram<br/>or .executeFile<br/>Execution.ts:58-91"]
C --> D{"session already provided?<br/>(e.g. file-upload flow)"}
D -- no --> E["getSessionController(runTime).getSession()<br/>Session.ts:59-69"]
D -- yes --> F["use provided session"]
E --> G{"pending session<br/>available in pool?"}
G -- yes --> H["reuse it"]
G -- no --> I["createSession()<br/>SAS: spawns a real sas process, see<br/>sas-execution-handshake.md<br/>JS/PY/R: cheap - folder + id only"]
H --> J["pre-warm: if pool has fewer<br/>than 3 pending, fire off more<br/>createSession() calls (not awaited)<br/>Session.ts:66"]
I --> J
F --> K
J --> K["session.state = running<br/>Execution.ts:96"]
K --> L["write webout.txt (empty),<br/>reqHeaders.txt<br/>Execution.ts:103-107"]
L --> M["processProgram(...)<br/>processProgram.ts:16-27"]
M --> N{"runTime?"}
N -- SAS --> O["createSASProgram():<br/>wrap user code with _webout<br/>filename, macro vars, autoexec"]
O --> P["write code.sas via .bkp + rename<br/>processProgram.ts:48-49"]
P --> Q["poll: while session.state !== completed<br/>processProgram.ts:52-58"]
Q -- "state becomes completed" --> R["resolve"]
Q -- "state becomes failed" --> S["throw Error(session.failureReason)"]
N -- "JS / PY / R" --> T["createJSProgram / createPythonProgram /<br/>createRProgram: wrap user code similarly"]
T --> U["write code file; spawn interpreter<br/>fresh via execFile; pipe stdout/stderr<br/>into a log.log write stream<br/>processProgram.ts:108-134"]
U -- "exit 0" --> R
U -- "exit non-zero" --> S
R --> V["read log.log, webout.txt,<br/>stpsrv_header.txt<br/>Execution.ts:128-131"]
V --> W["build httpHeaders + result<br/>return ExecuteReturnRaw<br/>Execution.ts:128-151"]
W --> X["Controller: res.send(result)<br/>HTTP 200"]
S --> Y["catch in Execution.ts:109-126:<br/>read whatever log exists,<br/>throw SessionExecutionError(message, log)"]
Y --> Z["Controller catch block<br/>rethrow { code: 400, status: 'failure',<br/>message, error, log }"]
Z --> AA["res.status(err.code).send(err)<br/>HTTP 400 with complete log"]
```
## Notes
- **Session pool is per-runtime.** SAS sessions and JS/PY/R sessions live in
separate controllers/pools (`process.sasSessionController` vs
`process.sessionController`, `Session.ts:239-253`) - a pending SAS session
is never handed out for a JS request or vice versa.
- **The SAS branch and the JS/PY/R branch converge** on the same
success/failure signal shape: `session.state` becomes `completed` or
`failed` either way, and both paths flow through the same log/webout
reading logic in `Execution.ts` afterward. The mechanics of *how* that
state gets set differ substantially (see `session-lifecycle.md`).
- **`includePrintOutput`** (SAS only) additionally appends `output.lst`
content to the result when debug mode is on - omitted above for brevity;
see `Execution.ts:135-142`.
- **`triggerProgram`/`triggerCode`** (fire-and-forget variants, not shown)
call the same `ExecutionController` methods without awaiting them and
immediately return `{ sessionId }`; the client polls
`GET /SASjsApi/session/{sessionId}/state` separately.
@@ -0,0 +1,81 @@
# SAS session execution handshake
How one long-lived `sas` process is turned into a single-use execution slot
via a SYSIN file-swap trick. This is the mechanism specific to
`RunTimeType.SAS`; JS/PY/R sessions spawn a fresh interpreter process per
request instead (see `request-execution-flow.md`).
```mermaid
sequenceDiagram
participant Client as HTTP client
participant Ctrl as Controller<br/>(code.ts / stp.ts)
participant Exec as ExecutionController<br/>(Execution.ts)
participant Pool as SASSessionController<br/>(Session.ts)
participant FS as Filesystem<br/>(session.path)
participant SAS as spawned "sas" process
Note over Pool,SAS: SESSION CREATION - happens ahead of any request,<br/>pool pre-warms up to 3 sessions (Session.ts:59-69)
Pool->>FS: createFile(code.sas, "") - empty dummy SYSIN (Session.ts:117-118)
Pool->>FS: readFile(sysInitCompiledPath) - compiled system-init<br/>macros, produced by `npm run compileSysInit` (Session.ts:105)
Note over Pool: throws if this file doesn't exist yet -<br/>session creation never reaches execFile below
Pool->>FS: createFile(autoexec.sas, autoExecContent) (Session.ts:108-114)
Pool->>SAS: execFile(sasLoc, -SYSIN code.sas -AUTOEXEC autoexec.sas -LOG log.log ...) (Session.ts:127-147)
activate SAS
Note right of SAS: process stays active (this box)<br/>from spawn until it finally exits below
SAS->>FS: autoexec step 1: delete code.sas, the dummy SYSIN (Session.ts:257-261)
Pool->>FS: waitForSession() polls fileExists(code.sas) (Session.ts:175-192)
FS-->>Pool: code.sas no longer exists
Pool->>Pool: session.state = pending (Session.ts:190)
SAS->>SAS: autoexec step 2: busy-wait for code.sas to reappear,<br/>up to 15 minutes (Session.ts:263-274)
Note over SAS: process is now idle, parked mid-autoexec,<br/>waiting on the filesystem.<br/>Counts as "pending" in the pool.
Note over Client,SAS: REQUEST ARRIVES - reuses the parked process above
Client->>Ctrl: POST .../execute { code, runTime: sas }
Ctrl->>Exec: executeProgram({ program, runTime: SAS, ... })
Exec->>Pool: getSession() returns the pending session (Session.ts:59-69)
Exec->>Exec: session.state = running (Execution.ts:96)
Exec->>FS: createFile(webout.txt, "") and createFile(reqHeaders.txt, ...) (Execution.ts:103-107)
Exec->>Pool: processProgram(program, session, ...) (Execution.ts:110-121)
Pool->>FS: createSASProgram() wraps user code with macro vars,<br/>_webout filename, autoexec injection (createSASProgram.ts)
Pool->>FS: write code.sas.bkp, then rename to code.sas (processProgram.ts:48-49)
Note over FS: write-then-rename, not a direct write,<br/>so SAS never reads a partial file
FS-->>SAS: code.sas exists again
SAS->>SAS: autoexec step 2 loop exits, sleeps 0.01s, autoexec ends (Session.ts:267-271)
Note over SAS: -SYSIN has pointed at code.sas from the start,<br/>so SAS now executes ITS CONTENT as the main job
SAS->>FS: writes log.log, output.lst, webout.txt while running
Pool->>Pool: processProgram poll loop:<br/>while session.state !== completed (processProgram.ts:52-58)
alt program reaches EOF normally
SAS->>SAS: exits with code 0
SAS-->>Pool: execFilePromise resolves - .then() (Session.ts:148-152)
Pool->>Pool: session.state = completed
Pool-->>Exec: processProgram() resolves
else program aborts (fatal error, license failure, hard STOP)
SAS->>SAS: exits with non-zero code
SAS-->>Pool: execFilePromise rejects - .catch() (Session.ts:153-163)
Pool->>Pool: session.state = failed<br/>session.failureReason = err.toString()
Pool-->>Exec: processProgram() throws (processProgram.ts:53-55)
end
deactivate SAS
Exec->>FS: read log.log, webout.txt, stpsrv_header.txt (Execution.ts:123, 128-131)
Exec-->>Ctrl: { httpHeaders, result }<br/>or throws SessionExecutionError{ message, log } (Execution.ts:109-126)
Ctrl-->>Client: 200 + result<br/>or 400 { status, message, error, log }
```
## Why this design
- SAS has meaningful startup cost (loading the engine, running system init
macros). Spawning a fresh `sas` process per request would pay that cost
every time. Instead, a process is spawned once and parked, ready to run
exactly one job the moment code shows up at a path it's already watching.
- The pool (`SessionController.getSession`, `Session.ts:59-69`) keeps up to
3 such parked processes ready, so most requests get an instant handoff
instead of waiting through the ~15-minute autoexec spin-wait or a cold
start.
- A session is single-use: once its process consumes the real SYSIN content
and exits (success or failure), that process is gone. The session object
itself is deleted later by `scheduleSessionDestroy` (`Session.ts:204-236`),
not reused for a second job.
+70
View File
@@ -0,0 +1,70 @@
# Session lifecycle (`SessionState`)
Enum defined in `api/src/types/Session.ts`. A `Session` is `{ id, state,
path, creationTimeStamp, deathTimeStamp, expiresAfterMins?, failureReason? }`.
Sessions live in an in-memory array on a per-runtime singleton controller
(`process.sasSessionController` for SAS, `process.sessionController` shared
by JS/PY/R — see `getSessionController()` in `Session.ts:239-253`).
```mermaid
stateDiagram-v2
[*] --> initialising: SAS runtime<br/>SASSessionController.createSession()<br/>Session.ts:77-95
[*] --> pending: JS/PY/R runtime, no process yet<br/>SessionController.createSession()<br/>Session.ts:30-57
initialising --> pending: dummy SYSIN file deleted<br/>by SAS's autoexec<br/>waitForSession(), Session.ts:175-192
initialising --> failed: spawned SAS process exits<br/>before handshake completes<br/>Session.ts:153-163, 184-188
pending --> running: session picked for a request<br/>executeProgram(), Execution.ts:94-96
running --> completed: process exits 0<br/>SAS: Session.ts:148-152 (original process)<br/>JS/PY/R: processProgram.ts:116-120 (fresh process)
running --> failed: process exits non-zero<br/>SAS: Session.ts:153-163<br/>JS/PY/R: processProgram.ts:121-131<br/>failureReason = err.toString()
completed --> [*]: deleteSession()<br/>scheduleSessionDestroy(), Session.ts:194-236
failed --> [*]: deleteSession()<br/>scheduleSessionDestroy()
note right of initialising
SAS only. The real sas
executable is already
running, spin-waiting
inside its AUTOEXEC for
real code to arrive.
See sas-execution-handshake.md
end note
note right of pending
Session sits in the pool,
reusable. Pre-warmed up to
3 pending sessions per
runtime (getSession(),
Session.ts:59-69)
end note
note right of running
SAS: no new process spawned
here, request just feeds
code to the already-running
process.
JS/PY/R: a brand new
interpreter process is
spawned right now.
end note
```
## Consumers of `state`
| Reader | Location | Watches for |
|---|---|---|
| `waitForSession` | `Session.ts:175-192` | `failed` (breaks early) or the dummy SYSIN file disappearing (implies session survived init) |
| `processProgram` (SAS branch poll loop) | `processProgram.ts:52-58` | `completed` (success exit) or `failed` (throws, carrying `session.failureReason`) |
| `scheduleSessionDestroy` | `Session.ts:204-236` | `running` (extends death timer instead of destroying) |
## Asymmetry between runtimes
- **SAS**: one OS process per session, spawned once at `createSession()`
time and reused for exactly one job (see `sas-execution-handshake.md`).
`running`/`completed`/`failed` are all driven by that *same* process's
eventual exit.
- **JS/PY/R**: a session is cheap (folder + id, no process). The interpreter
process is spawned fresh, per request, inside `processProgram.ts:115` and
its exit drives `completed`/`failed` directly in the same function - there
is no separate poll loop for these runtimes.
+6090 -14458
View File
File diff suppressed because it is too large Load Diff
+19 -21
View File
@@ -6,11 +6,9 @@
"scripts": {
"initial": "npm run swagger && npm run compileSysInit && npm run copySASjsCore && npm run downloadMacros",
"prestart": "npm run initial",
"prebuild": "npm run initial",
"start": "NODE_ENV=development nodemon ./src/server.ts",
"start:prod": "node ./build/src/server.js",
"build": "rimraf build && tsc",
"postbuild": "npm run copy:files",
"build": "npm run initial && rimraf build && tsc && npm run copy:files",
"swagger": "tsoa spec",
"prepare": "[ -d .git ] && git config core.hooksPath ./.git-hooks || true",
"test": "mkdir -p tmp && mkdir -p ../web/build && jest --silent --coverage",
@@ -48,25 +46,25 @@
},
"author": "4GL Ltd",
"dependencies": {
"@sasjs/core": "^4.40.1",
"@sasjs/utils": "3.2.0",
"@sasjs/core": "^4.59.10",
"@sasjs/utils": "^3.5.6",
"bcryptjs": "^2.4.3",
"connect-mongo": "^4.6.0",
"cookie-parser": "^1.4.6",
"connect-mongo": "^5.1.0",
"cookie-parser": "^1.4.7",
"cors": "^2.8.5",
"express": "^4.17.1",
"express-session": "^1.17.2",
"express": "^4.21.2",
"express-session": "^1.18.2",
"helmet": "^5.0.2",
"joi": "^17.4.2",
"jsonwebtoken": "^8.5.1",
"jsonwebtoken": "^9.0.3",
"ldapjs": "2.3.3",
"mongoose": "^6.0.12",
"morgan": "^1.10.0",
"mongoose": "^6.13.8",
"morgan": "^1.10.1",
"multer": "^1.4.5-lts.1",
"rate-limiter-flexible": "2.4.1",
"rotating-file-stream": "^3.0.4",
"swagger-ui-express": "4.3.0",
"unzipper": "^0.10.11",
"unzipper": "^0.12.3",
"url": "^0.10.3"
},
"devDependencies": {
@@ -76,32 +74,32 @@
"@types/cors": "^2.8.12",
"@types/express": "^4.17.12",
"@types/express-session": "^1.17.4",
"@types/jest": "^26.0.24",
"@types/jest": "^29.5.0",
"@types/jsonwebtoken": "^8.5.5",
"@types/ldapjs": "^2.2.4",
"@types/morgan": "^1.9.3",
"@types/multer": "^1.4.7",
"@types/node": "^15.12.2",
"@types/node": "^20.0.0",
"@types/supertest": "^2.0.11",
"@types/swagger-ui-express": "^4.1.3",
"@types/unzipper": "^0.10.5",
"adm-zip": "^0.5.9",
"axios": "0.27.2",
"axios": "^1.12.2",
"csrf": "^3.1.0",
"dotenv": "^16.0.1",
"http-headers-validation": "^0.0.1",
"jest": "^27.0.6",
"jest": "^29.7.0",
"mongodb-memory-server": "8.11.4",
"nodejs-file-downloader": "4.10.2",
"nodemon": "^2.0.7",
"nodemon": "^3.0.0",
"pkg": "5.6.0",
"prettier": "^2.3.1",
"prettier": "^3.0.0",
"rimraf": "^3.0.2",
"supertest": "^6.1.3",
"ts-jest": "^27.0.3",
"ts-jest": "^29.1.0",
"ts-node": "^10.0.0",
"tsoa": "3.14.1",
"typescript": "^4.3.2"
"typescript": "^5.0.0"
},
"nodemonConfig": {
"ignore": [
+3 -2
View File
@@ -234,9 +234,10 @@ const verifyAuthCode = async (
jwt.verify(code, process.secrets.AUTH_CODE_SECRET, (err, data) => {
if (err) return resolve(undefined)
const payload = data as InfoJWT
const clientInfo: InfoJWT = {
clientId: data?.clientId,
userId: data?.userId
clientId: payload?.clientId,
userId: payload?.userId
}
if (clientInfo.clientId === clientId) {
return resolve(clientInfo)
+2 -1
View File
@@ -114,7 +114,8 @@ const executeCode = async (
code: 400,
status: 'failure',
message: 'Job execution failed.',
error: typeof err === 'object' ? err.toString() : err
error: typeof err === 'object' ? err.toString() : err,
log: err?.log
}
}
}
+36 -12
View File
@@ -15,6 +15,24 @@ export interface ExecutionVars {
[key: string]: string | number | undefined
}
// Thrown when the session itself fails (e.g. SAS exits abnormally via
// %abort;). Carries the complete log - by the time this is thrown, the
// session's process has already exited, so the log file it wrote is final,
// not a partial/truncated snapshot.
export class SessionExecutionError extends Error {
constructor(
message: string,
public log?: string
) {
super(message)
// required for `instanceof` to work when compiling to ES5, since the
// default __extends helper does not preserve the prototype chain for
// classes extending built-ins like Error
Object.setPrototypeOf(this, SessionExecutionError.prototype)
}
}
export interface ExecuteReturnRaw {
httpHeaders: HTTPHeaders
result: string | Buffer
@@ -88,18 +106,24 @@ export class ExecutionController {
preProgramVariables?.httpHeaders.join('\n') ?? ''
)
await processProgram(
program,
preProgramVariables,
vars,
session,
weboutPath,
headersPath,
tokenFile,
runTime,
logPath,
otherArgs
)
try {
await processProgram(
program,
preProgramVariables,
vars,
session,
weboutPath,
headersPath,
tokenFile,
runTime,
logPath,
otherArgs
)
} catch (err: any) {
const log = (await fileExists(logPath)) ? await readFile(logPath) : ''
throw new SessionExecutionError(err.message, log)
}
const log = (await fileExists(logPath)) ? await readFile(logPath) : ''
const headersContent = (await fileExists(headersPath))
+10 -3
View File
@@ -260,9 +260,16 @@ data _null_;
rc=filename(fname,getoption('SYSIN') );
if rc = 0 and fexist(fname) then rc=fdelete(fname);
rc=filename(fname);
/* now wait for the real SYSIN */
slept=0;
do until ( fileexist(getoption('SYSIN')) or slept>(60*15) );
/* now wait for the real SYSIN (location of code.sas) */
slept=0;fname='';
do until (slept>(60*15));
rc=filename(fname,getoption('SYSIN'));
if rc = 0 and fexist(fname) then do;
putlog fname=;
rc=filename(fname);
rc=sleep(0.01,1); /* wait just a little more */
stop;
end;
slept=slept+sleep(0.01,1);
end;
stop;
@@ -50,6 +50,10 @@ export const processProgram = async (
// we now need to poll the session status
while (session.state !== SessionState.completed) {
if (session.state === SessionState.failed) {
throw new Error(session.failureReason || 'SAS session failed')
}
await delay(50)
}
} else {
@@ -0,0 +1,65 @@
import path from 'path'
import os from 'os'
import { createFile, deleteFolder, generateTimestamp } from '@sasjs/utils'
import * as ProcessProgramModule from '../processProgram'
import { ExecutionController, SessionExecutionError } from '../Execution'
import { Session, SessionState, PreProgramVars } from '../../../types'
import { RunTimeType } from '../../../utils'
const preProgramVariables: PreProgramVars = {
username: 'testUser',
userId: 1,
displayName: 'Test User',
serverUrl: 'http://localhost:5000',
httpHeaders: []
}
describe('ExecutionController.executeProgram (SAS failure path)', () => {
let session: Session
beforeEach(() => {
const sessionId = `test-session-${generateTimestamp()}`
session = {
id: sessionId,
state: SessionState.pending,
creationTimeStamp: '0',
deathTimeStamp: '0',
path: path.join(os.tmpdir(), sessionId)
}
})
afterEach(async () => {
jest.restoreAllMocks()
await deleteFolder(session.path)
})
it('throws a SessionExecutionError carrying the complete log when the session fails', async () => {
const logPath = path.join(session.path, 'log.log')
const logContent =
'NOTE: SAS session\nERROR: SAS session terminated. See log for details.\n'
await createFile(logPath, logContent)
jest
.spyOn(ProcessProgramModule, 'processProgram')
.mockImplementation(async () => {
throw new Error('ERROR: SAS session terminated. See log for details.')
})
const controller = new ExecutionController()
const resultPromise = controller.executeProgram({
program: '%abort;',
preProgramVariables,
vars: {},
session,
runTime: RunTimeType.SAS
})
await expect(resultPromise).rejects.toBeInstanceOf(SessionExecutionError)
await expect(resultPromise).rejects.toMatchObject({
log: logContent,
message: expect.stringContaining('SAS session terminated')
})
})
})
@@ -0,0 +1,109 @@
import path from 'path'
import os from 'os'
import { createFile, deleteFolder, generateTimestamp } from '@sasjs/utils'
import { processProgram } from '../processProgram'
import { Session, SessionState, PreProgramVars } from '../../../types'
import {
RunTimeType,
getSessionsFolder,
generateUniqueFileName
} from '../../../utils'
const preProgramVariables: PreProgramVars = {
username: 'testUser',
userId: 1,
displayName: 'Test User',
serverUrl: 'http://localhost:5000',
httpHeaders: []
}
const makeSession = (): Session => {
const sessionId = generateUniqueFileName(generateTimestamp())
const sessionFolder = path.join(getSessionsFolder(), sessionId)
const creationTimeStamp = sessionId.split('-').pop() as string
const deathTimeStamp = (
parseInt(creationTimeStamp) +
15 * 60 * 1000 -
1000
).toString()
return {
id: sessionId,
state: SessionState.running,
creationTimeStamp,
deathTimeStamp,
path: sessionFolder
}
}
describe('processProgram (SAS runtime)', () => {
let session: Session
let logPath: string
let weboutPath: string
let headersPath: string
let tokenFile: string
beforeAll(() => {
const root = path.join(
os.tmpdir(),
`sasjs-processProgram-spec-${generateTimestamp()}`
)
process.sasjsRoot = root
process.driveLoc = path.join(root, 'drive')
})
beforeEach(async () => {
session = makeSession()
logPath = path.join(session.path, 'log.log')
weboutPath = path.join(session.path, 'webout.txt')
headersPath = path.join(session.path, 'stpsrv_header.txt')
tokenFile = path.join(session.path, 'reqHeaders.txt')
await createFile(weboutPath, '')
})
afterEach(async () => {
await deleteFolder(session.path)
})
it('rejects instead of hanging when the session fails (e.g. %abort;)', async () => {
setTimeout(() => {
session.state = SessionState.failed
session.failureReason =
'ERROR: SAS session terminated. See log for details.'
}, 100)
await expect(
processProgram(
'%abort;',
preProgramVariables,
{},
session,
weboutPath,
headersPath,
tokenFile,
RunTimeType.SAS,
logPath
)
).rejects.toThrow(/SAS session terminated/)
}, 3000)
it('resolves without throwing when the session completes normally', async () => {
setTimeout(() => {
session.state = SessionState.completed
}, 100)
await expect(
processProgram(
'%put hello world;',
preProgramVariables,
{},
session,
weboutPath,
headersPath,
tokenFile,
RunTimeType.SAS,
logPath
)
).resolves.toBeUndefined()
}, 3000)
})
+2 -1
View File
@@ -166,7 +166,8 @@ const execute = async (
code: 400,
status: 'failure',
message: 'Job execution failed.',
error: typeof err === 'object' ? err.toString() : err
error: typeof err === 'object' ? err.toString() : err,
log: err?.log
}
}
}
+5 -2
View File
@@ -106,7 +106,10 @@ const login = async (
const rateLimiter = RateLimiter.getInstance()
if (!validPass) {
const retrySecs = await rateLimiter.consume(req.ip, user?.username)
const retrySecs = await rateLimiter.consume(
req.ip || 'unknown',
user?.username
)
if (retrySecs > 0) throw errors.tooManyRequests(retrySecs)
}
@@ -114,7 +117,7 @@ const login = async (
if (!validPass) throw errors.invalidPassword
// Reset on successful authorization
rateLimiter.resetOnSuccess(req.ip, user.username)
rateLimiter.resetOnSuccess(req.ip || 'unknown', user.username)
req.session.loggedIn = true
req.session.user = {
+3 -3
View File
@@ -37,10 +37,10 @@ export const authenticateAccessToken: RequestHandler = async (
if (user.isActive) {
req.user = user
return csrfProtection(req, res, nextFunction)
} else return res.sendStatus(401)
} else return res.status(401).send('Unauthorized')
}
}
return res.sendStatus(401)
return res.status(401).send('Unauthorized')
}
await authenticateToken(
@@ -118,6 +118,6 @@ const authenticateToken = async (
return next()
}
res.sendStatus(401)
res.status(401).send('Unauthorized')
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ import { convertSecondsToHms } from '@sasjs/utils'
import { RateLimiter } from '../utils'
export const bruteForceProtection: RequestHandler = async (req, res, next) => {
const ip = req.ip
const ip = req.ip || 'unknown'
const username = req.body.username
const rateLimiter = RateLimiter.getInstance()
+1 -1
View File
@@ -76,7 +76,7 @@ groupSchema.post('save', function (group: IGroup, next: Function) {
})
// pre remove hook to remove all references of group from users
groupSchema.pre('remove', async function () {
groupSchema.pre('remove', async function (this: IGroupDocument) {
const userIds = this.users
await Promise.all(
userIds.map(async (userId) => {
+132
View File
@@ -0,0 +1,132 @@
import path from 'path'
import { createFile, fileExists } from '@sasjs/utils'
import { Express } from 'express'
import mongoose, { Mongoose } from 'mongoose'
import { MongoMemoryServer } from 'mongodb-memory-server'
import request from 'supertest'
import {
UserController,
PermissionController,
PermissionType,
PermissionSettingForRoute,
PrincipalType
} from '../../../controllers/'
import {
generateAccessToken,
saveTokensInDB,
RunTimeType,
sysInitCompiledPath
} from '../../../utils'
// Real, unmocked end-to-end test of the SAS execution pipeline (session
// spawn -> autoexec handshake -> exit code -> HTTP response), using a fake
// "SAS executable" (mockSas.js) in place of a real SAS install. This is the
// regression test for issue #388: SAS code that aborts (%abort;) used to
// hang the request forever instead of returning an error response.
const mockSasPath = path.join(__dirname, 'files', 'mockSas.js')
const clientId = 'codeSpecClientID'
const user = {
displayName: 'Code Spec User',
username: 'codeSpecUsername',
password: '87654321',
isAdmin: false,
isActive: true
}
let app: Express
let accessToken: string
describe('code', () => {
let con: Mongoose
let mongoServer: MongoMemoryServer
let userController: UserController
let permissionController: PermissionController
beforeAll(async () => {
// SASSessionController.createSession() unconditionally reads this file
// (Session.ts:105) before it ever spawns the SAS process. It's normally
// produced by `npm run compileSysInit` (part of the `initial`/`build`
// scripts), which `npm test` does not run - so on a fresh checkout
// (e.g. CI, where "Run Unit Tests" happens before "Build Package") it
// doesn't exist yet. Content is irrelevant here since mockSas.js never
// interprets it; it just needs to exist and be readable.
if (!(await fileExists(sysInitCompiledPath))) {
await createFile(sysInitCompiledPath, '')
}
mongoServer = await MongoMemoryServer.create()
process.env.DB_CONNECT = mongoServer.getUri()
process.env.MODE = 'server'
process.env.RUN_TIMES = 'sas'
process.env.SAS_PATH = mockSasPath
const appPromise = (await import('../../../app')).default
app = await appPromise
con = await mongoose.connect(mongoServer.getUri())
userController = new UserController()
permissionController = new PermissionController()
const dbUser = await userController.createUser(user)
accessToken = await generateAndSaveToken(dbUser.id)
await permissionController.createPermission({
path: '/SASjsApi/code/execute',
type: PermissionType.route,
principalType: PrincipalType.user,
principalId: dbUser.id,
setting: PermissionSettingForRoute.grant
})
process.runTimes = [RunTimeType.SAS]
process.sasLoc = mockSasPath
}, 30000)
afterAll(async () => {
await con.connection.dropDatabase()
await con.connection.close()
await mongoServer.stop()
})
describe('execute', () => {
it('returns 200 with the mock log when the SAS program completes normally', async () => {
const response = await request(app)
.post('/SASjsApi/code/execute')
.auth(accessToken, { type: 'bearer' })
.send({ code: '%put hello world;', runTime: 'sas' })
.expect(200)
expect(response.text).toEqual(
expect.stringContaining('mock SAS execution')
)
}, 30000)
it('returns a prompt 400 (not a hang) with the complete log when the SAS session fails (%abort;)', async () => {
const response = await request(app)
.post('/SASjsApi/code/execute')
.auth(accessToken, { type: 'bearer' })
.send({ code: '%abort;', runTime: 'sas' })
.expect(400)
expect(response.body).toMatchObject({
status: 'failure',
message: 'Job execution failed.'
})
expect(response.body.log).toEqual(
expect.stringContaining('mock SAS execution')
)
}, 30000)
})
})
const generateAndSaveToken = async (userId: number) => {
const accessToken = generateAccessToken({
clientId,
userId
})
await saveTokensInDB(userId, clientId, accessToken, 'refreshToken')
return accessToken
}
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env node
// Minimal fake "SAS executable" used by tests in place of a real SAS
// install. It only fulfils the CLI/filesystem handshake contract that
// Session.ts / processProgram.ts rely on:
//
// -SYSIN <path> the file used as a signal channel: it starts out
// containing a dummy (empty) placeholder, we delete it
// to signal "session ready", then wait for it to be
// rewritten with the real submitted program.
// -LOG <path> where we write a fake log so downstream fileExists()/
// readFile() calls have something to find.
//
// It does NOT interpret real SAS syntax. Exit code is the only thing that
// matters to the Node side: 0 mimics a normal SAS termination, non-zero
// mimics an abnormal one (e.g. %abort;).
const fs = require('fs')
const arg = (flag) => {
const idx = process.argv.indexOf(flag)
return idx === -1 ? undefined : process.argv[idx + 1]
}
const sysin = arg('-SYSIN')
const logPath = arg('-LOG')
// Give up waiting for the real program after this long, so an unused
// pre-warmed session (sasjs pools up to 3 ready sessions) doesn't linger
// forever and keep a test process alive. Generous rather than tight: CI
// runners are frequently slower/more contended than a local dev machine
// (shared CPU, coverage instrumentation on the Node side slowing down the
// round trip this process is waiting on), and this cost is only ever paid
// by sessions nothing is actually waiting on - the session actually used
// by a request gets its real code written far sooner than this in practice.
const GIVE_UP_AFTER_MS = 8000
const sleepSync = (ms) => {
const until = Date.now() + ms
while (Date.now() < until) {
/* busy-wait, mirroring the real autoexec's SAS-side sleep() loop */
}
}
// Any of the filesystem calls below can legitimately race against
// processProgram's own write-then-rename (e.g. existsSync sees the file,
// then a rename mid-flight makes the following read miss). Treat that as
// "not ready yet" and retry a few times, rather than letting an uncaught
// exception crash this process with a non-zero exit - which would be
// indistinguishable, to the Node side, from a genuine SAS failure.
const retry = (fn, attempts = 5, delayMs = 20) => {
for (let i = 0; i < attempts; i++) {
try {
return { ok: true, value: fn() }
} catch (err) {
if (i === attempts - 1) return { ok: false, error: err }
sleepSync(delayMs)
}
}
}
// 1. remove the dummy SYSIN, signalling "session ready" to waitForSession()
if (sysin) {
retry(() => {
if (fs.existsSync(sysin)) fs.unlinkSync(sysin)
})
}
// 2. wait for the real program to be written back to the same path
const deadline = Date.now() + GIVE_UP_AFTER_MS
while (!sysin || !fs.existsSync(sysin)) {
if (Date.now() > deadline) process.exit(0)
sleepSync(10)
}
// small settle delay, mirroring the real autoexec's sleep(0.01,1) after
// detecting the file, so a fast-moving rename isn't read mid-write
sleepSync(50)
const readResult = retry(() => fs.readFileSync(sysin, 'utf-8'))
if (!readResult.ok) {
process.stderr.write(
`mockSas.js: failed to read ${sysin}: ${readResult.error}\n`
)
process.exit(1)
}
const code = readResult.value
if (logPath) {
retry(() => fs.writeFileSync(logPath, `NOTE: mock SAS execution\n${code}\n`))
}
if (code.includes('%abort;')) {
process.stderr.write('ERROR: SAS session terminated. See log for details.\n')
process.exit(1)
}
process.exit(0)
+4 -1
View File
@@ -277,7 +277,10 @@ const performLogin = async (
.set('x-xsrf-token', csrfToken)
.send(credentials)
return { authCookies: header['set-cookie'].join() }
return {
authCookies:
(header['set-cookie'] as unknown as string[] | undefined)?.join() || ''
}
}
const extractCSRF = (text: string) =>
+3 -1
View File
@@ -1,5 +1,6 @@
import jwt from 'jsonwebtoken'
import User from '../model/User'
import { InfoJWT } from '../types/InfoJWT'
const isValidToken = async (
token: string,
@@ -11,7 +12,8 @@ const isValidToken = async (
jwt.verify(token, key, (err, decoded) => {
if (err) return reject(false)
if (decoded?.userId === userId && decoded?.clientId === clientId) {
const payload = decoded as InfoJWT
if (payload?.userId === userId && payload?.clientId === clientId) {
return resolve(true)
}
+4233 -8631
View File
File diff suppressed because it is too large Load Diff
+1983 -1428
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -19,9 +19,8 @@
"@types/jest": "^26.0.24",
"@types/node": "^12.20.28",
"@types/react": "^17.0.27",
"axios": "^0.24.0",
"axios": "^1.12.2",
"monaco-editor": "^0.33.0",
"monaco-editor-webpack-plugin": "^7.0.1",
"react": "^17.0.2",
"react-copy-to-clipboard": "^5.1.0",
"react-dom": "^17.0.2",
@@ -54,6 +53,7 @@
"eslint-webpack-plugin": "^3.1.1",
"file-loader": "^6.2.0",
"html-webpack-plugin": "5.5.0",
"monaco-editor-webpack-plugin": "^7.0.1",
"path": "0.12.7",
"prettier": "^2.4.1",
"sass": "^1.44.0",
+5 -5
View File
@@ -1,15 +1,15 @@
body {
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
sans-serif;
font-family:
-apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu',
'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
code {
font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
monospace;
font-family:
source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace;
}
.container {