feat: adding gitpod.yml with sasjs code extension

initial commit

.gitignore

@@ -1,2 +1,3 @@
 node_modules
-.DS_Store
+.DS_Store
+sasjsbuild/

.gitpod.yml

@@ -0,0 +1,3 @@
+vscode:
+  extensions:
+    - sasjs.sasjs-for-vscode@1.2.6:AJmar85B1uSEapxRaRQGrQ==

CONTRIBUTING.md

@@ -1,83 +1,32 @@
 # Contributing
 
-Contributions to the macrocore library are warmly welcomed!  To avoid any
-misunderstandings, do please first discuss the change you wish to make via issue,
-email, or any other method with the owners of this repository before submitting
-a PR.
+Contributions are warmly welcomed!  To avoid any misunderstandings, do please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before submitting a PR.
 
-Please note we have a code of conduct, please follow it in all your interactions
-with the project.
+Please note we have a [code of conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/), please follow it in all your interactions with the project.
+
+# Environment Setup
+
+This repository makes use of the [SASjs](https://sasjs.io) framework for code organisation, compilation, documentation, and deployment.  The following tools are highly recommended:
+
+* [NPM](https://sasjs.io/windows/#npm) - the runtime and dependency manager for [SASjs CLI](https://cli.sasjs.io) (batteries included)
+* [VSCode](https://sasjs.io/windows/#vscode) - feature packed IDE for code editing (warning - highly effective!)
+* [GIT](https://sasjs.io/windows/#git) - a safety net you cannot (and should not) do without.
+
+For generating the documentation (`sasjs doc`) it is also necessary to install [doxygen](https://www.doxygen.nl/manual/install.html).
 
 
-## Code of Conduct
+To get configured:
 
-### Our Pledge
+1.  Clone the repository
+2.  Install local dependencies (`npm install`)
+3.  Install the SASjs CLI globally (`npm install -g @sasjs/cli`)
+4.  Add a target, and authentication (`npm add`).  See [docs](https://cli.sasjs.io/add/).
 
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, gender identity and expression, level of experience,
-nationality, personal appearance, race, religion, or sexual identity and
-orientation.
+To contribute:
 
-### Our Standards
+1.  Create your feature branch (`git checkout -b myfeature`)
+2.  Make your change
+3.  Update the `all.sas` file (`python3 build.py`)
+4.  Commit the change, using the [conventional commit](https://www.conventionalcommits.org/en/v1.0.0) standard
+5.  Push and make a PR
 
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
-  address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-### Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-### Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-### Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at support@macropeople.com. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-### Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at [http://contributor-covenant.org/version/1/4][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/

LICENSE

@@ -1,4 +1,4 @@
-Copyright 2018 (Allan Bowe)
+Copyright 2020 (Allan Bowe)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

README.md

@@ -40,6 +40,27 @@ Documentation: https://sasjs.github.io/core.github.io/files.html
 - X command enabled
 - Prefixes: _mmw_,_mmu_,_mmx_
 
+**lua** library 
+
+Wait - this is a macro library - what is LUA doing here?  Well, it is a little known fact that you CAN run LUA within a SAS Macro.  It has to be written to a text file with a `.lua` extension, from where you can `%include` it.  So, without using the `proc lua` wrapper.
+
+To contribute, simply write your freeform LUA in the LUA folder.  Then run the `build.py`, which will convert your LUA into a data step with put statements, and create the macro wrapper with a `ml_` prefix.  You can then use your module in any program by running:
+
+```
+/* compile the lua module */
+%ml_yourmodule()
+
+/* Execute.  Do not use the restart keyword! */
+proc lua; 
+submit;
+  print(yourStuff);
+endsubmit;
+run;
+```
+
+- X command enabled
+- Prefixes: _mmw_,_mmu_,_mmx_
+
 # Installation
 
 First, download the repo to a location your SAS system can access. Then update your sasautos path to include the components you wish to have available,eg:
@@ -72,6 +93,7 @@ filename mc url "https://raw.githubusercontent.com/sasjs/core/main/all.sas";
   - _mm_ for metadata macros (interface with the metadata server).
   - _mmx_ for macros that use metadata and are XCMD enabled
   - _mx_ for macros that are XCMD enabled
+  - _ml_ for macros that are used to compile LUA modules
   - _mv_ for macros that will only work in Viya
 - follow verb-noun convention
 - unix style line endings (lf)
@@ -91,7 +113,25 @@ The **Macro Core** documentation is created using [doxygen](http://www.doxygen.n
 - version. The EARLIEST SAS version in which this macro is known to work.
 - author. Author name, contact details optional
 
-All macros must be commented in the doxygen format, to enable the [online documentation](https://sasjs.github.io/core.github.io/).
+All macros must be commented in the doxygen format, to enable the [online documentation](https://core.sasjs.io).
+
+### Dependencies
+SAS code can contain one of two types of dependency - SAS Macros, and SAS Programs.  When compiling projects using the [SASjs CLI](https://cli.sasjs.io) the doxygen header is scanned for `  @li` items under the following headers:
+
+```
+  <h4> SAS Macros </h4>
+  @li mf_nobs.sas
+  @li mm_assignlib.sas
+
+  <h4> SAS Programs </h4>
+  @li somefile.ddl SOMEFREF
+  @li someprogram.sas FREFTWO
+```
+
+The CLI can then extract all the dependencies and insert as precode (SAS Macros) or in a temp engine fileref (SAS Programs) when creating SAS Jobs and Services.
+
+When contributing to this library, it is therefore important to ensure that all dependencies are listed in the header in this format.
+
 
 ## Coding Standards
 
@@ -102,6 +142,7 @@ All macros must be commented in the doxygen format, to enable the [online docume
 - Mandatory parameters should be positional, all optional parameters should be keyword (var=) style.
 - All dataset references must be 2 level (eg `work.blah`, not `blah`). This is to avoid contention when options [DATASTMTCHK](https://support.sas.com/documentation/cdl/en/lrdict/64316/HTML/default/viewer.htm#a000279064.htm)=ALLKEYWORDS is in effect.
 - Avoid naming collisions! All macro variables should be local scope. Use system generated work tables where possible - eg `data ; set sashelp.class; run; data &output; set &syslast; run;`
+- If you have a long-running SQL query, the use of a `quit;` statement is recommended in order to benefit from the timing statistics.
 
 # General Notes
 

base/mf_abort.sas

@@ -21,6 +21,7 @@
 
   @version 9.2
   @author Allan Bowe
+  @cond
 **/
 
 %macro mf_abort(mac=mf_abort.sas, type=, msg=, iftrue=%str(1=1)
@@ -139,3 +140,5 @@
     %abort cancel;
   %end;
 %mend;
+
+/** @endcond */

base/mf_existfeature.sas

@@ -1,5 +1,5 @@
 /**
-  @file mf_existfeature.sas
+  @file
   @brief Checks whether a feature exists
   @details Check to see if a feature is supported in your environment.
     Run without arguments to see a list of detectable features.
@@ -7,19 +7,20 @@
     actual feature detection, as that is tricky / impossible to do
     without generating errors in most cases.
 
-      %put %mf_existfeature(PROCLUA);
+        %put %mf_existfeature(PROCLUA);
 
   @param feature the feature to detect.  Leave blank to list all in log.
-  
+
   @return output returns 1 or 0 (or -1 if not found)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getplatform.sas
 
 
   @version 8
   @author Allan Bowe
 **/
+/** @cond */
 
 %macro mf_existfeature(feature
 )/*/STORE SOURCE*/;
@@ -31,12 +32,16 @@
     %put Supported features:  PROCLUA;
   %end;
   %else %if &feature=PROCLUA %then %do;
+    /* https://blogs.sas.com/content/sasdummy/2015/08/03/using-lua-within-your-sas-programs */
     %if &platform=SASVIYA %then 1;
-    %else %if "&sysver"="9.3" or "&sysver"="9.4" %then 1;
-    %else 0;
+    %else %if "&sysver"="9.2" or "&sysver"="9.3" %then 0;
+    %else %if "&SYSVLONG" < "9.04.01M3" %then 0;
+    %else 1;
   %end;
   %else %do;
     -1
     %put &sysmacroname: &feature not found;
   %end;
-%mend;
+%mend;
+
+/** @endcond */

base/mf_existfileref.sas

@@ -0,0 +1,27 @@
+/**
+  @file
+  @brief Checks whether a fileref exists
+  @details You can probably do without this macro as it is just a one liner.
+  Mainly it is here as a convenient way to remember the syntax!
+
+  @param fref the fileref to detect
+
+  @return output Returns 1 if found and 0 if not found.  Note - it is possible
+  that the fileref is found, but the file does not (yet) exist. If you need
+  to test for this, you may as well use the fileref function directly.
+
+  @version 8
+  @author [Allan Bowe](https://www.linkedin.com/in/allanbowe/)
+**/
+
+%macro mf_existfileref(fref
+)/*/STORE SOURCE*/;
+
+  %if %sysfunc(fileref(&fref))=0 %then %do;
+    1
+  %end;
+  %else %do;
+    0
+  %end;
+
+%mend;

base/mf_existvar.sas

@@ -12,6 +12,7 @@
   @version 9.2
   @author Allan Bowe
 **/
+/** @cond */
 
 %macro mf_existvar(libds /* 2 part dataset name */
       , var /* variable name */
@@ -29,4 +30,6 @@
       %let rc=%sysfunc(close(&dsid));
   %end;
 
-%mend;
+%mend;
+
+/** @endcond */

base/mf_existvarlist.sas

@@ -6,7 +6,7 @@
 
         %put %mf_existVarList(sashelp.class, age sex name dummyvar)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_abort.sas
 
   @param libds 2 part dataset or view reference
@@ -14,6 +14,7 @@
 
   @version 9.2
   @author Allan Bowe
+  @cond
 **/
 
 %macro mf_existvarlist(libds, varlist
@@ -53,4 +54,6 @@
     0
     %put Vars not found: &found;
   %end;
-%mend;
+%mend;
+
+/** @endcond */

base/mf_getattrn.sas

@@ -23,7 +23,7 @@
   %local dsid rc;
   %let dsid=%sysfunc(open(&libds,is));
   %if &dsid = 0 %then %do;
-    %put WARNING: Cannot open %trim(&libds), system message below;
+    %put %str(WARN)ING: Cannot open %trim(&libds), system message below;
     %put %sysfunc(sysmsg());
     -1
   %end;

base/mf_getengine.sas

@@ -3,7 +3,7 @@
   @brief Returns the engine type of a SAS library
   @details Usage:
 
-      %put %mf_getEngine(SASHELP);
+      %put %mf_getengine(SASHELP);
 
   returns:
   > V9
@@ -21,9 +21,11 @@
 
   @version 9.2
   @author Allan Bowe
-**/
 
-%macro mf_getEngine(libref
+**/
+/** @cond */
+
+%macro mf_getengine(libref
 )/*/STORE SOURCE*/;
   %local dsid engnum rc engine;
 
@@ -42,3 +44,5 @@
  &engine
 
 %mend;
+
+/** @endcond */

base/mf_getkeyvalue.sas

@@ -3,8 +3,8 @@
   @brief retrieves a key value pair from a control dataset
   @details By default, control dataset is work.mp_setkeyvalue.  Usage:
 
-    %mp_setkeyvalue(someindex,22,type=N)
-    %put %mf_getkeyvalue(someindex)
+      %mp_setkeyvalue(someindex,22,type=N)
+      %put %mf_getkeyvalue(someindex)
 
 
   @param key Provide a key on which to perform the lookup

base/mf_getplatform.sas

@@ -10,7 +10,7 @@
 
   @param switch the param for which to return a platform specific variable
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_mval.sas
   @li mf_trimstr.sas
 

base/mf_getquotedstr.sas

@@ -3,11 +3,18 @@
   @brief Adds custom quotes / delimiters to a  delimited string
   @details Can be used in open code, eg as follows:
 
-    %put %mf_getquotedstr(blah   blah  blah);
+      %put %mf_getquotedstr(blah   blah  blah);
 
   which returns:
 > 'blah','blah','blah'
 
+  Alternatively:
+
+      %put %mf_getquotedstr(these words are double quoted,quote=D)
+
+  for:
+> "these","words","are","double","quoted"
+
   @param in_str the unquoted, spaced delimited string to transform
   @param dlm= the delimeter to be applied to the output (default comma)
   @param indlm= the delimeter used for the input (default is space)

base/mf_getschema.sas

@@ -17,6 +17,7 @@
 
   @version 9.2
   @author Allan Bowe
+  @cond
 **/
 
 %macro mf_getschema(libref
@@ -38,3 +39,5 @@
   &schema
 
 %mend;
+
+/** @endcond */

base/mf_getuniquefileref.sas

@@ -1,17 +1,18 @@
 /**
   @file
   @brief Assigns and returns an unused fileref
-  @details Use as follows:
+  @details
+  Use as follows:
 
-    %let fileref1=%mf_getuniquefileref();
-    %let fileref2=%mf_getuniquefileref();
-    %put &fileref1 &fileref2;
+      %let fileref1=%mf_getuniquefileref();
+      %let fileref2=%mf_getuniquefileref();
+      %put &fileref1 &fileref2;
 
   which returns:
 
 > mcref0 mcref1
 
-  @prefix= first part of fileref. Remember that filerefs can only be 8
+  @param prefix= first part of fileref. Remember that filerefs can only be 8
     characters, so a 7 letter prefix would mean that `maxtries` should be 10.
   @param maxtries= the last part of the libref.  Provide an integer value.
 

base/mf_getuniquelibref.sas

@@ -14,7 +14,7 @@
 
 > mclib3
 
-  @prefix= first part of libref.  Remember that librefs can only be 8 characters,
+  @param prefix= first part of libref.  Remember that librefs can only be 8 characters,
     so a 7 letter prefix would mean that maxtries should be 10.
   @param maxtries= the last part of the libref.  Provide an integer value.
 

base/mf_getvalue.sas

@@ -7,7 +7,7 @@
       %put %mf_getvalue(sashelp.class,name,filter=%quote(age=15));
       %put %mf_getvalue(sashelp.class,name);
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getattrn.sas
 
   @param libds dataset to query

base/mf_isdir.sas

@@ -0,0 +1,33 @@
+/**
+  @file
+  @brief Checks whether a path is a valid directory
+  @details
+  Usage:
+
+      %let isdir=%mf_isdir(/tmp);
+
+  With thanks and full credit to Andrea Defronzo - https://www.linkedin.com/in/andrea-defronzo-b1a47460/
+
+  @param path full path of the file/directory to be checked
+
+  @return output returns 1 if path is a directory, 0 if it is not
+
+  @version 9.2
+**/
+
+%macro mf_isdir(path
+)/*/STORE SOURCE*/;
+	%local rc did is_directory fref_t;
+
+	%let is_directory = 0;
+	%let rc = %sysfunc(filename(fref_t, %superq(path)));
+	%let did = %sysfunc(dopen(&fref_t.));
+	%if &did. ^= 0 %then %do;
+	   %let is_directory = 1;
+	   %let rc = %sysfunc(dclose(&did.));
+	%end;
+	%let rc = %sysfunc(filename(fref_t));
+
+	&is_directory
+
+%mend;

base/mf_nobs.sas

@@ -6,7 +6,7 @@
 
         %put Number of observations=%mf_nobs(sashelp.class);
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getattrn.sas
 
   @param libds library.dataset

base/mf_trimstr.sas

@@ -8,7 +8,7 @@
         %put %mf_trimstr(/blah/,h); * /blah/;
         %put %mf_trimstr(/blah/,h/);* /bla;
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
 
 
   @param basestr The string to be modified
@@ -23,13 +23,23 @@
 **/
 
 %macro mf_trimstr(basestr,trimstr);
-%local trimlen trimval;
+%local baselen trimlen trimval;
+
+/* return if basestr is shorter than trimstr (or 0) */
+%let baselen=%length(%superq(basestr));
 %let trimlen=%length(%superq(trimstr));
+%if &baselen < &trimlen or &baselen=0 %then %return;
+
+/* obtain the characters from the end of basestr */
 %let trimval=%qsubstr(%superq(basestr)
   ,%length(%superq(basestr))-&trimlen+1
   ,&trimlen);
 
-%if %superq(trimval)=%superq(trimstr) %then %do;
+/* compare and if matching, chop it off! */
+%if %superq(basestr)=%superq(trimstr) %then %do;
+  %return;
+%end;
+%else %if %superq(trimval)=%superq(trimstr) %then %do;
   %qsubstr(%superq(basestr),1,%length(%superq(basestr))-&trimlen)
 %end;
 %else %do;

base/mf_verifymacvars.sas

@@ -11,7 +11,7 @@
   Returns:
   > 1
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_abort.sas
 
   @param verifyvars space separated list of macro variable names

base/mp_abort.sas

@@ -6,12 +6,19 @@
     results back to the client in an STP Web App context, or completely stop
     in the case of a batch run.
 
+  Using SAS Abort Cancel mechanisms can cause hung sessions in some Stored Process
+  environments.  This macro takes a unique approach - we set the SAS syscc to 0,
+  run `stpsrvset('program error', 0)` (if SAS 9) and then - we open a macro
+  but don't close it!  This provides a graceful abort for SAS web services in all
+  web enabled environments.
+
   @param mac= to contain the name of the calling macro
   @param msg= message to be returned
   @param iftrue= supply a condition under which the macro should be executed.
 
   @version 9.4M3
   @author Allan Bowe
+  @cond
 **/
 
 %macro mp_abort(mac=mp_abort.sas, type=, msg=, iftrue=%str(1=1)
@@ -24,7 +31,7 @@
   %put NOTE - &msg;
 
   /* Stored Process Server web app context */
-  %if %symexist(_metaperson) 
+  %if %symexist(_metaperson)
   or (%symexist(SYSPROCESSNAME) and "&SYSPROCESSNAME"="Compute Server" )
   %then %do;
     options obs=max replace nosyntaxcheck mprint;
@@ -65,8 +72,15 @@
     %end;
 
     %if %symexist(SYS_JES_JOB_URI) %then %do;
-      /* refer web service output to file service in one hit */
-      filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name="_webout.json";
+      /* setup webout */
+      OPTIONS NOBOMFILE;
+      %if "X&SYS_JES_JOB_URI.X"="XX" %then %do;
+          filename _webout temp lrecl=999999 mod;
+      %end;
+      %else %do;
+        filename _webout filesrvc parenturi="&SYS_JES_JOB_URI"
+          name="_webout.json" lrecl=999999 mod;
+      %end;
     %end;
 
     /* send response in SASjs JSON format */
@@ -138,3 +152,4 @@
   %end;
 %mend;
 
+/** @endcond */

base/mp_cleancsv.sas

@@ -8,6 +8,7 @@
     applying CRLF line endings and converting embedded cr and crlf to lf.
 
   usage:
+
       fileref mycsv "/path/your/csv";
       %mp_cleancsv(in=mycsv,out=/path/new.csv)
 
@@ -17,6 +18,7 @@
 
   @version 9.2
   @author Allan Bowe
+  @cond
 **/
 
 %macro mp_cleancsv(in=NOTPROVIDED,out=NOTPROVIDED,qchar='22'x);
@@ -65,4 +67,5 @@
       else put inchar $char1.;
     end;
   run;
-%mend;
+%mend;
+/** @endcond */

base/mp_createconstraints.sas

@@ -22,7 +22,7 @@
   @param outds= a table containing the create statements (create_statement column)
   @param execute= `YES|NO` - default is NO. To actually create, use YES.
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
 
   @version 9.2
   @author Allan Bowe

base/mp_createwebservice.sas

@@ -26,7 +26,7 @@ Usage:
     ;;;;
     %mp_createwebservice(path=/Public/app/common,name=appInit,code=ft15f001,replace=YES)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getplatform.sas
   @li mm_createwebservice.sas
   @li mv_createwebservice.sas

base/mp_csv2ds.sas

@@ -0,0 +1,144 @@
+/**
+  @file mp_csv2ds.sas
+  @brief Efficient import of arbitrary CSV using a dataset as template
+  @details Used to import relevant columns from a large CSV using
+  a dataset to provide the types and lengths.  Assumes that a header
+  row is provided, and datarows start on line 2.  Extra columns in
+  both the CSV and base dataset are ignored.
+
+  Usage:
+
+      filename mycsv temp;
+      data _null_;
+        file mycsv;
+        put 'name,age,nickname';
+        put 'John,48,Jonny';
+        put 'Jennifer,23,Jen';
+      run;
+
+      %mp_csv2ds(inref=mycsv,outds=myds,baseds=sashelp.class)
+
+
+  @param inref= fileref to the CSV
+  @param outds= output ds (lib.ds format)
+  @param view= Set to YES or NO to determine whether the output should be
+    a view or not.  Default is NO (not a view).
+  @param baseds= Template dataset on which to create the input statement.
+    Is used to determine types, lengths, and any informats.
+
+  @version 9.2
+  @author Allan Bowe
+
+  <h4> SAS Macros </h4>
+  @li mp_abort.sas
+  @li mf_existds.sas
+
+**/
+
+%macro mp_csv2ds(inref=0,outds=0,baseds=0,view=NO);
+
+%mp_abort(iftrue=( &inref=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(the INREF variable must be provided)
+)
+%mp_abort(iftrue=( %superq(outds)=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(the OUTDS variable must be provided)
+)
+%mp_abort(iftrue=( &baseds=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(the BASEDS variable must be provided)
+)
+%mp_abort(iftrue=( &baseds=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(the BASEDS variable must be provided)
+)
+%mp_abort(iftrue=( %mf_existds(&baseds)=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(the BASEDS dataset (&baseds) needs to be assigned, and to exist)
+)
+
+/* count rows */
+%local hasheader; %let hasheader=0;
+data _null_;
+  if _N_ > 1 then do;
+     call symputx('hasheader',1,'l');
+     stop;
+  end;
+  infile &inref;
+  input;
+run;
+%mp_abort(iftrue=( &hasheader=0 )
+  ,mac=&sysmacroname
+  ,msg=%str(No header row in &inref)
+)
+
+/* get the variables in the CSV */
+data _data_;
+  infile &inref;
+  input;
+  length name $32;
+  do i=1 to countc(_infile_,',')+1;
+    name=upcase(scan(_infile_,i,','));
+    output;
+  end;
+  stop;
+run;
+%local csv_vars;%let csv_vars=&syslast;
+
+/* get the variables in the dataset */
+proc contents noprint data=&baseds
+  out=_data_ (keep=name type length format: informat);
+run;
+%local base_vars; %let base_vars=&syslast;
+
+proc sql undo_policy=none;
+create table &csv_vars as
+  select a.*
+    ,b.type
+    ,b.length
+    ,b.format
+    ,b.formatd
+    ,b.formatl
+    ,b.informat
+  from &csv_vars a
+  left join &base_vars b
+  on a.name=upcase(b.name)
+  order by i;
+
+/* prepare the input statement */
+%local instat dropvars;
+data _null_;
+  set &syslast end=last;
+  length in dropvars $32767;
+  retain in dropvars;
+  if missing(type) then do;
+    informat='$1.';
+    dropvars=catx(' ',dropvars,name);
+  end;
+  else if missing(informat) then do;
+    if type=1 then informat='best.';
+    else informat=cats('$',length,'.');
+  end;
+  else informat=cats(informat,'.');
+  in=catx(' ',in,name,':',informat);
+  if last then do;
+    call symputx('instat',in,'l');
+    call symputx('dropvars',dropvars,'l');
+  end;
+run;
+
+/* import the CSV */
+data &outds
+  %if %upcase(&view)=YES %then %do;
+   /view=&outds
+  %end;
+  ;
+  infile &inref dsd firstobs=2;
+  input &instat;
+  %if %length(&dropvars)>0 %then %do;
+    drop &dropvars;
+  %end;
+run;
+
+%mend;

base/mp_dirlist.sas

@@ -1,17 +1,17 @@
 /**
   @file
   @brief Returns all files and subdirectories within a specified parent
-  @details When used with getattrs=NO, is not OS specific (uses dopen / dread). 
+  @details When used with getattrs=NO, is not OS specific (uses dopen / dread).
 
   If getattrs=YES then the doptname / foptname functions are used to scan all
-  properties - any characters that are not valid in a SAS name (v7) are simply 
+  properties - any characters that are not valid in a SAS name (v7) are simply
   stripped, and the table is transposed so theat each property is a column
-  and there is one file per row.  An attempt is made to get all properties 
+  and there is one file per row.  An attempt is made to get all properties
   whether a file or folder, but some files/folders cannot be accessed, and so
   not all properties can / will be populated.
 
   Credit for the rename approach:
-  https://communities.sas.com/t5/SAS-Programming/SAS-Function-to-convert-string-to-Legal-SAS-Name/m-p/27375/highlight/true#M5003 
+  https://communities.sas.com/t5/SAS-Programming/SAS-Function-to-convert-string-to-Legal-SAS-Name/m-p/27375/highlight/true#M5003
 
 
   usage:
@@ -20,21 +20,25 @@
 
       %mp_dirlist(outds=cwdfileprops, getattrs=YES)
 
-  @warning In a Unix environment, the existence of a named pipe will cause this 
+      %mp_dirlist(fref=MYFREF)
+
+  @warning In a Unix environment, the existence of a named pipe will cause this
   macro to hang.  Therefore this tool should be used with caution in a SAS 9 web
   application, as it can use up all available multibridge sessions if requests
   are resubmitted.
-  If anyone finds a way to positively identify a named pipe using SAS (without 
+  If anyone finds a way to positively identify a named pipe using SAS (without
   X CMD) do please raise an issue!
 
 
   @param path= for which to return contents
+  @param fref= Provide a DISK engine fileref as an alternative to PATH
   @param outds= the output dataset to create
-  @param getattrs= YES/NO (default=NO).  Uses doptname and foptname to return 
-  all attributes for each file / folder.  
+  @param getattrs= YES/NO (default=NO).  Uses doptname and foptname to return
+  all attributes for each file / folder.
 
 
   @returns outds contains the following variables:
+   - directory (containing folder)
    - file_or_folder (file / folder)
    - filepath (path/to/file.name)
    - filename (just the file name)
@@ -47,18 +51,26 @@
 **/
 
 %macro mp_dirlist(path=%sysfunc(pathname(work))
+    , fref=0
     , outds=work.mp_dirlist
     , getattrs=NO
 )/*/STORE SOURCE*/;
 %let getattrs=%upcase(&getattrs)XX;
 
-data &outds (compress=no keep=file_or_folder filepath filename ext msg);
-  length filepath $500 fref fref2 $8 file_or_folder $6 filename $80 ext $20 msg $200;
-  rc = filename(fref, "&path");
+data &outds (compress=no keep=file_or_folder filepath filename ext msg directory);
+  length directory filepath $500 fref fref2 $8 file_or_folder $6 filename $80 ext $20 msg $200;
+  %if &fref=0 %then %do;
+    rc = filename(fref, "&path");
+  %end;
+  %else %do;
+    fref="&fref";
+    rc=0;
+  %end;
   if rc = 0 then do;
      did = dopen(fref);
+     directory=dinfo(did,'Directory');
      if did=0 then do;
-        putlog "NOTE: This directory is empty - &path";
+        putlog "NOTE: This directory is empty - " directory;
         msg=sysmsg();
         put _all_;
         stop;
@@ -73,7 +85,8 @@ data &outds (compress=no keep=file_or_folder filepath filename ext msg);
   dnum = dnum(did);
   do i = 1 to dnum;
     filename = dread(did, i);
-    rc = filename(fref2, "&path/"!!filename);
+    filepath=cats(directory,'/',filename);
+    rc = filename(fref2,filepath);
     midd=dopen(fref2);
     dmsg=sysmsg();
     if did > 0 then file_or_folder='folder';
@@ -82,12 +95,12 @@ data &outds (compress=no keep=file_or_folder filepath filename ext msg);
     fmsg=sysmsg();
     if midf > 0 then file_or_folder='file';
     rc=fclose(midf);
-    
-    if index(fmsg,'File is in use') or index(dmsg,'is not a directory') 
+
+    if index(fmsg,'File is in use') or index(dmsg,'is not a directory')
       then file_or_folder='file';
     else if index(fmsg, 'Insufficient authorization') then file_or_folder='file';
     else if file_or_folder='' then file_or_folder='locked';
-      
+
     if file_or_folder='file' then do;
       ext = prxchange('s/.*\.{1,1}(.*)/$1/', 1, filename);
       if filename = ext then ext = ' ';
@@ -96,7 +109,6 @@ data &outds (compress=no keep=file_or_folder filepath filename ext msg);
       ext='';
       file_or_folder='folder';
     end;
-    filepath="&path/"!!filename;
     output;
   end;
   rc = dclose(did);
@@ -120,7 +132,7 @@ run;
       else do i=1 to foptnum(fid);
         infoname=foptname(fid,i);
         infoval=finfo(fid,infoname);
-        sasname=compress(infoname, '_', 'adik');	
+        sasname=compress(infoname, '_', 'adik');
         if anydigit(sasname)=1 then sasname=substr(sasname,anyalpha(sasname));
         if upcase(sasname) ne 'FILENAME' then output;
       end;
@@ -137,7 +149,7 @@ run;
       else do i=1 to doptnum(fid);
         infoname=doptname(fid,i);
         infoval=dinfo(fid,infoname);
-        sasname=compress(infoname, '_', 'adik');	
+        sasname=compress(infoname, '_', 'adik');
         if anydigit(sasname)=1 then sasname=substr(sasname,anyalpha(sasname));
         if upcase(sasname) ne 'FILENAME' then output;
       end;

base/mp_dropmembers.sas

@@ -9,7 +9,7 @@
       create view view2 as select * from sashelp.class;
       %mp_dropmembers(list=data1 view2)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_isblank.sas
 
 

base/mp_ds2cards.sas

@@ -3,25 +3,27 @@
   @brief Create a CARDS file from a SAS dataset.
   @details Uses dataset attributes to convert all data into datalines.
     Running the generated file will rebuild the original dataset.
-  usage:
+  Usage:
 
       %mp_ds2cards(base_ds=sashelp.class
         , cards_file= "C:\temp\class.sas"
         , maxobs=5)
 
-    stuff to add
+  TODO:
      - labelling the dataset
      - explicity setting a unix LF
      - constraints / indexes etc
 
-  @param base_ds= Should be two level - eg work.blah.  This is the table that
+  @param [in] base_ds= Should be two level - eg work.blah.  This is the table that
                    is converted to a cards file.
-  @param tgt_ds= Table that the generated cards file would create. Optional -
+  @param [in] tgt_ds= Table that the generated cards file would create. Optional -
                   if omitted, will be same as BASE_DS.
-  @param cards_file= Location in which to write the (.sas) cards file
-  @param maxobs= to limit output to the first <code>maxobs</code> observations
-  @param showlog= whether to show generated cards file in the SAS log (YES/NO)
-  @param outencoding= provide encoding value for file statement (eg utf-8)
+  @param [out] cards_file= Location in which to write the (.sas) cards file
+  @param [in] maxobs= to limit output to the first <code>maxobs</code> observations
+  @param [in] showlog= whether to show generated cards file in the SAS log (YES/NO)
+  @param [in] outencoding= provide encoding value for file statement (eg utf-8)
+  @param [in] append= If NO then will rebuild the cards file if it already exists,
+  otherwise will append to it.  Used by the mp_lib2cards.sas macro.
 
 
   @version 9.2
@@ -34,6 +36,7 @@
     ,random_sample=NO
     ,showlog=YES
     ,outencoding=
+    ,append=NO
 )/*/STORE SOURCE*/;
 %local i setds nvars;
 
@@ -46,6 +49,8 @@
 %if (&tgt_ds = ) %then %let tgt_ds=&base_ds;
 %if %index(&tgt_ds,.)=0 %then %let tgt_ds=WORK.%scan(&base_ds,2,.);
 %if ("&outencoding" ne "") %then %let outencoding=encoding="&outencoding";
+%if ("&append" = "") %then %let append=;
+%else %let append=mod;
 
 /* get varcount */
 %let nvars=0;
@@ -103,8 +108,7 @@ proc sql
   order by ranuni(42)
 %end;
   ;
-
-
+reset outobs=max;
 create table datalines1 as
    select name,type,length,varnum,format,label from dictionary.columns
    where libname="%upcase(%scan(&base_ds,1))"
@@ -173,7 +177,7 @@ data _null_;
 run;
 
 data _null_;
-  file &cards_file. &outencoding lrecl=32767 termstr=nl;
+  file &cards_file. &outencoding lrecl=32767 termstr=nl &append;
   length __attrib $32767;
   if _n_=1 then do;
     put '/*******************************************************************';

base/mp_ds2csv.sas

@@ -0,0 +1,58 @@
+/**
+  @file
+  @brief Export a dataset to a CSV file
+  @details Export to a file or a fileref
+  Usage:
+
+      %mp_ds2csv(sashelp.class,outref="%sysfunc(pathname(work))/file.csv")
+
+  @param ds The dataset to be exported
+  @param outfile= The output filename - should be quoted.
+  @param outref= The output fileref (takes precedence if provided)
+  @param outencoding= The output encoding to use (unquoted)
+
+  @version 9.2
+  @author Allan Bowe (credit mjsq)
+**/
+
+%macro mp_ds2csv(ds, outref=0, outfile=, outencoding=0
+)/*/STORE SOURCE*/;
+
+%if not %sysfunc(exist(&ds)) %then %do;
+   %put WARNING:  &ds does not exist;
+   %return;
+%end;
+
+%if %index(&ds,.)=0 %then %let ds=WORK.&ds;
+
+%if &outencoding=0 %then %let outencoding=;
+%else %let outencoding=encoding="&outencoding";
+
+%local outloc;
+%if &outref=0 %then %let outloc=&outfile;
+%else %let outloc=&outref;
+
+/* credit to mjsq - https://stackoverflow.com/a/55642267 */
+
+/* first get headers */
+data _null_;
+   file &outloc dlm=',' dsd &outencoding lrecl=32767;
+   length header $ 2000;
+   dsid=open("&ds.","i");
+   num=attrn(dsid,"nvars");
+   do i=1 to num;
+      header = trim(left(coalescec(varlabel(dsid,i),varname(dsid,i))));
+      put header @;
+   end;
+   rc=close(dsid);
+run;
+
+/* next, export data */
+data _null_;
+   set &ds.;
+   file &outloc mod dlm=',' dsd &outencoding lrecl=32767;
+   put (_all_) (+0);
+run;
+
+
+%mend;

base/mp_getconstraints.sas

@@ -21,7 +21,7 @@
   @param ds= The target dataset.  Leave blank (default) for all datasets.
   @param outds the output dataset
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
 
   @version 9.2
   @author Allan Bowe

base/mp_getdbml.sas

@@ -0,0 +1,326 @@
+/**
+  @file
+  @brief Extract DBML from SAS Libraries
+  @details DBML is an open source markup format to represent databases.
+  More details: https://www.dbml.org/home/
+
+  Usage:
+
+
+      %mp_getdbml(liblist=SASHELP WORK,outref=mydbml,showlog=YES)
+
+  Take the log output and paste it into the renderer at https://dbdiagram.io
+  to view your data model diagram.  The code takes a "best guess" at
+  the one to one and one to many relationships (based on constraints
+  and indexes, and assuming that the column names would match).
+
+  You may need to adjust the rendered DBML to suit your needs.
+
+  ![dbml for sas](https://i.imgur.com/8T1tIZp.gif)
+
+
+  <h4> SAS Macros </h4>
+  @li mf_getquotedstr.sas
+  @li mp_getconstraints.sas
+
+  @param liblist= Space seperated list of librefs to take as
+    input (Default=SASHELP)
+  @param outref= Fileref to contain the DBML (Default=getdbml)
+  @param showlog= set to YES to show the DBML in the log (Default is NO)
+
+  @version 9.3
+  @author Allan Bowe
+**/
+
+%macro mp_getdbml(liblist=SASHELP,outref=getdbml,showlog=NO
+)/*/STORE SOURCE*/;
+
+/* check fileref is assigned */
+%if %sysfunc(fileref(&outref)) > 0 %then %do;
+  filename &outref temp;
+%end;
+
+%let liblist=%upcase(&liblist);
+
+proc sql noprint;
+create table _data_ as
+  select * from dictionary.tables
+  where upcase(libname) in (%mf_getquotedstr(&liblist))
+  order by libname,memname;
+%local tabinfo; %let tabinfo=&syslast;
+
+create table _data_ as
+  select * from dictionary.columns
+  where upcase(libname) in (%mf_getquotedstr(&liblist))
+  order by libname,memname,varnum;
+%local colinfo; %let colinfo=&syslast;
+
+%local dsnlist;
+  select distinct upcase(cats(libname,'.',memname)) into: dsnlist
+  separated by ' '
+  from &syslast
+;
+
+create table _data_ as
+  select * from dictionary.indexes
+  where upcase(libname) in (%mf_getquotedstr(&liblist))
+  order by idxusage, indxname, indxpos;
+%local idxinfo; %let idxinfo=&syslast;
+
+/* Extract all Primary Key and Unique data constraints */
+%mp_getconstraints(lib=%scan(&liblist,1),outds=_data_)
+%local colconst; %let colconst=&syslast;
+
+%do x=2 %to %sysfunc(countw(&liblist));
+  %mp_getconstraints(lib=%scan(&liblist,&x),outds=_data_)
+  proc append base=&colconst data=&syslast;
+  run;
+%end;
+
+
+
+
+/* header info */
+data _null_;
+  file &outref;
+  put "// DBML generated by &sysuserid on %sysfunc(datetime(),datetime19.) ";
+  put "Project sasdbml {";
+  put "  database_type: 'SAS'";
+  put "  Note: 'Generated by the mp_getdbml() macro'";
+  put "}";
+run;
+
+/* create table groups */
+data _null_;
+  file &outref mod;
+  set &tabinfo;
+  by libname;
+  if first.libname then put "TableGroup " libname "{";
+  ds=quote(cats(libname,'.',memname));
+  put '   ' ds;
+  if last.libname then put "}";
+run;
+
+/* table for pks */
+data _data_;
+  length curds const col $39;
+  call missing (of _all_);
+  stop;
+run;
+%let pkds=&syslast;
+
+%local x curds constraints_used constcheck;
+%do x=1 %to %sysfunc(countw(&dsnlist,%str( )));
+  %let curds=%scan(&dsnlist,&x,%str( ));
+  %let constraints_used=;
+  %let constcheck=0;
+  data _null_;
+    file &outref mod;
+    length lab $1024 typ $20;
+    set &colinfo (where=(
+        libname="%scan(&curds,1,.)" and upcase(memname)="%scan(&curds,2,.)"
+    )) end=last;
+
+    if _n_=1 then do;
+      table='Table "'!!"&curds"!!'"{';
+      put table;
+    end;
+    name=upcase(name);
+    lab=" note:"!!quote(trim(tranwrd(label,'"',"'")));
+    if upcase(format)=:'DATETIME' then typ='datetime';
+    else if type='char' then typ=cats('char(',length,')');
+    else typ='num';
+
+    if notnull='yes' then notnul=' not null';
+    if notnull='no' and missing(label) then put '  ' name typ;
+    else if notnull='yes' and missing(label) then put '  ' name typ '[' notnul ']';
+    else if notnull='no' then put '  ' name typ '[' lab ']';
+    else put '  ' name typ '[' notnul ',' lab ']';
+
+  run;
+
+  data _data_(keep=curds const col);
+    length ctype $11 cols constraints_used $5000;
+    set &colconst (where=(
+      upcase(libref)="%scan(&curds,1,.)"
+      and upcase(table_name)="%scan(&curds,2,.)"
+      and constraint_type in ('PRIMARY','UNIQUE')
+    )) end=last;
+    file &outref mod;
+    by constraint_type constraint_name;
+    retain cols;
+    column_name=upcase(column_name);
+
+    if _n_=1 then put / '  indexes {';
+
+    if upcase(strip(constraint_type)) = 'PRIMARY' then ctype='[pk]';
+    else ctype='[unique]';
+
+    if first.constraint_name then cols = cats('(',column_name);
+    else cols=cats(cols,',',column_name);
+
+    if last.constraint_name then do;
+      cols=cats(cols,')',ctype)!!' //'!!constraint_name;
+      put '    ' cols;
+      constraints_used=catx(' ',constraints_used, constraint_name);
+      call symputx('constcheck',1);
+    end;
+
+    if last then call symputx('constraints_used',cats(upcase(constraints_used)));
+
+    length curds const col $39;
+    curds="&curds";
+    const=constraint_name;
+    col=column_name;
+  run;
+
+  proc append base=&pkds data=&syslast;run;
+
+  /* Create Unique Indexes, but only if they were not already defined within the Constraints section. */
+  data _data_(keep=curds const col);
+    set &idxinfo (where=(
+      libname="%scan(&curds,1,.)"
+      and upcase(memname)="%scan(&curds,2,.)"
+      and unique='yes'
+      and upcase(indxname) not in (%mf_getquotedstr(&constraints_used))
+    ));
+    file &outref mod;
+    by idxusage indxname;
+    name=upcase(name);
+    if &constcheck=1 then stop; /* in fact we only care about PKs so stop if we have */
+    if _n_=1 and &constcheck=0 then put / '  indexes {';
+
+    length cols $5000;
+    retain cols;
+    if first.indxname then cols = cats('(',name);
+    else cols=cats(cols,',',name);
+
+    if last.indxname then do;
+      cols=cats(cols,')[unique]')!!' //'!!indxname;
+      put '    ' cols;
+      call symputx('constcheck',1);
+    end;
+
+    length curds const col $39;
+    curds="&curds";
+    const=indxname;
+    col=name;
+  run;
+  proc append base=&pkds data=&syslast;run;
+
+  data _null_;
+    file &outref mod;
+    if &constcheck =1 then put '  }';
+    put '}';
+  run;
+
+%end;
+
+/**
+ * now we need to figure out the relationships
+ */
+
+/* sort alphabetically so we can have one set of unique cols per table */
+proc sort data=&pkds nodupkey;
+  by curds const col;
+run;
+
+data &pkds.1 (keep=curds col)
+     &pkds.2 (keep=curds cols);
+  set &pkds;
+  by curds const;
+  length retconst $39 cols $5000;
+  retain retconst cols;
+  if first.curds then do;
+    retconst=const;
+    cols=upcase(col);
+  end;
+  else cols=catx(' ',cols,upcase(col));
+  if retconst=const then do;
+    output &pkds.1;
+    if last.const then output &pkds.2;
+  end;
+run;
+
+%let curdslist="0";
+%do x=1 %to %sysfunc(countw(&dsnlist,%str( )));
+  %let curds=%scan(&dsnlist,&x,%str( ));
+
+  %let pkcols=0;
+  data _null_;
+    set &pkds.2(where=(curds="&curds"));
+    call symputx('pkcols',cols);
+  run;
+  %if &pkcols ne 0 %then %do;
+    %let curdslist=&curdslist,"&curds";
+
+    /* start with one2one */
+    data &pkds.4;
+      file &outref mod;
+      set &pkds.2(where=(cols="&pkcols" and curds not in (&curdslist)));
+      line='Ref: "'!!"&curds"
+        !!cats('".(',"%mf_getquotedstr(&pkcols,dlm=%str(,),quote=%str( ))",')')
+        !!' - '
+        !!cats(quote(trim(curds)),'.(',"%mf_getquotedstr(&pkcols,dlm=%str(,),quote=%str( ))",')');
+      put line;
+    run;
+
+    /* now many2one */
+    /* get table with one row per col */
+    data &pkds.5;
+      set &pkds.1(where=(curds="&curds"));
+    run;
+    /* get tables which contain the PK columns */
+    proc sql;
+    create table &pkds.5a as
+      select upcase(cats(b.libname,'.',b.memname)) as curds
+        ,b.name
+      from &pkds.5 a
+      inner join &colinfo b
+      on a.col=upcase(b.name);
+    /* count to make sure those tables contain ALL the columns */
+    create table &pkds.5b as
+      select curds,count(*) as cnt
+      from &pkds.5a
+      where curds not in (select curds from &pkds.2 where cols="&pkcols") /* not a one to one match */
+        and curds ne "&curds" /* exclude self */
+      group by 1;
+    create table &pkds.6 as
+      select a.*
+        ,b.cols
+      from &pkds.5b a
+      left join &pkds.4 b
+      on a.curds=b.curds;
+
+    data _null_;
+      set &pkds.6;
+      file &outref mod;
+      colcnt=%sysfunc(countw(&pkcols));
+      if cnt=colcnt then do;
+        /* table contains all the PK cols, and was not a direct / 121 match */
+        line='Ref: "'!!"&curds"
+          !!'".('
+          !!"%mf_getquotedstr(&pkcols,dlm=%str(,),quote=%str( ))"
+          !!') > '
+          !!cats(quote(trim(curds))
+              ,'.('
+              ,"%mf_getquotedstr(&pkcols,dlm=%str(,),quote=%str( ))"
+              ,')'
+          );
+        put line;
+      end;
+    run;
+  %end;
+%end;
+
+
+%if %upcase(&showlog)=YES %then %do;
+  options ps=max;
+  data _null_;
+    infile &outref;
+    input;
+    putlog _infile_;
+  run;
+%end;
+
+%mend;

base/mp_getddl.sas

@@ -15,7 +15,7 @@
       proc sql; describe table &syslast;
       %mp_getddl(work,test,flavour=tsql,showlog=YES)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_getconstraints.sas
 
   @param lib libref of the library to create DDL for.  Should be assigned.
@@ -30,7 +30,6 @@
    datetime2 format or regular decimal type
   @version 9.3
   @author Allan Bowe
-  @source https://github.com/sasjs/core
 **/
 
 %macro mp_getddl(libref,ds,fref=getddl,flavour=SAS,showlog=NO,schema=
@@ -132,7 +131,7 @@ run;
     %let curds=%scan(&dsnlist,&x);
     data _null_;
       file &fref mod;
-      length nm lab $1024;
+      length nm lab $1024 typ $20;
       set &colinfo (where=(upcase(memname)="&curds")) end=last;
 
       if _n_=1 then do;
@@ -146,10 +145,12 @@ run;
       end;
       else put "   ,"@@;
       if length(format)>1 then fmt=" format="!!cats(format);
-      len=" length="!!cats(length);
-      lab=" label="!!quote(trim(label));
+      if length(label)>1 then lab=" label="!!quote(trim(label));
       if notnull='yes' then notnul=' not null';
-      put name type len fmt notnul lab;
+      if type='char' then typ=cats('char(',length,')');
+      else if length ne 8 then typ='num length='!!left(length);
+      else typ='num';
+      put name typ fmt notnul lab;
     run;
 
     /* Extra step for data constraints */
@@ -159,7 +160,7 @@ run;
       file &fref mod;
       put ');';
     run;
-  
+
     /* Create Unique Indexes, but only if they were not already defined within the Constraints section. */
     data _null_;
       *length ds $128;
@@ -339,7 +340,7 @@ run;
 
   %end;
 %end;
-%if &showlog=YES %then %do;
+%if %upcase(&showlog)=YES %then %do;
   options ps=max;
   data _null_;
     infile &fref;

base/mp_getmaxvarlengths.sas

@@ -21,7 +21,7 @@
   @param libds Two part dataset (or view) reference.
   @param outds= The output dataset to create
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getvarlist.sas
   @li mf_getvartype.sas
   @li mf_getvarformat.sas

base/mp_guesspk.sas

@@ -3,9 +3,9 @@
   @brief Guess the primary key of a table
   @details Tries to guess the primary key of a table based on the following logic:
 
-  * Columns with nulls are ignored
-  * Return only column combinations that provide unique results
-  * Start from one column, then move out to include composite keys of 2 to 6 columns
+      * Columns with nulls are ignored
+      * Return only column combinations that provide unique results
+      * Start from one column, then move out to include composite keys of 2 to 6 columns
 
   The library of the target should be assigned before using this macro.
 
@@ -22,7 +22,7 @@
   @param min_rows= The minimum number of rows a table should have in order to try
     and guess the PK.  Default=5.
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getvarlist.sas
   @li mf_getuniquename.sas
   @li mf_nobs.sas
@@ -51,7 +51,7 @@
   /* get null count and row count */
   %let tmpvar=%mf_getuniquename();
   proc sql noprint;
-  create table _data_ as select 
+  create table _data_ as select
     count(*) as &tmpvar
   %do i=1 %to &vcnt;
     %let var=%scan(&vars,&i);
@@ -85,10 +85,10 @@
     %put &sysmacroname: &baseds has no combination of unique records! Exiting.;
     %return;
   %end;
-  
+
   /* now check cardinality */
   proc sql noprint;
-  create table _data_ as select 
+  create table _data_ as select
   %do i=1 %to &ppkcnt;
     %let var=%scan(&posspks,&i);
     count(distinct &var) as &var
@@ -212,7 +212,7 @@
       %end;
     %end;
   %end;
- 
+
   %if &ppkcnt=4 %then %do;
     %put &sysmacroname: No more PK guess possible;
     %return;
@@ -228,7 +228,7 @@
         %let lev3=%scan(&posspks,&k);
         %if &lev1 ne &lev3 and &lev2 ne &lev3 %then %do l=4 %to &ppkcnt;
           %let lev4=%scan(&posspks,&l);
-          %if &lev1 ne &lev4 and &lev2 ne &lev4 and &lev3 ne &lev4 %then 
+          %if &lev1 ne &lev4 and &lev2 ne &lev4 and &lev3 ne &lev4 %then
           %do m=5 %to &ppkcnt;
             %let lev5=%scan(&posspks,&m);
             %if &lev1 ne &lev5 & &lev2 ne &lev5 & &lev3 ne &lev5 & &lev4 ne &lev5 %then %do;
@@ -250,7 +250,7 @@
       %end;
     %end;
   %end;
- 
+
   %if &ppkcnt=5 %then %do;
     %put &sysmacroname: No more PK guess possible;
     %return;
@@ -266,17 +266,17 @@
         %let lev3=%scan(&posspks,&k);
         %if &lev1 ne &lev3 and &lev2 ne &lev3 %then %do l=4 %to &ppkcnt;
           %let lev4=%scan(&posspks,&l);
-          %if &lev1 ne &lev4 and &lev2 ne &lev4 and &lev3 ne &lev4 %then 
+          %if &lev1 ne &lev4 and &lev2 ne &lev4 and &lev3 ne &lev4 %then
           %do m=5 %to &ppkcnt;
             %let lev5=%scan(&posspks,&m);
-            %if &lev1 ne &lev5 & &lev2 ne &lev5 & &lev3 ne &lev5 & &lev4 ne &lev5 %then 
+            %if &lev1 ne &lev5 & &lev2 ne &lev5 & &lev3 ne &lev5 & &lev4 ne &lev5 %then
             %do n=6 %to &ppkcnt;
               %let lev6=%scan(&posspks,&n);
-              %if &lev1 ne &lev6 & &lev2 ne &lev6 & &lev3 ne &lev6 
-              & &lev4 ne &lev6 & &lev5 ne &lev6 %then 
+              %if &lev1 ne &lev6 & &lev2 ne &lev6 & &lev3 ne &lev6
+              & &lev4 ne &lev6 & &lev5 ne &lev6 %then
               %do;
                 /* check for four level uniqueness */
-                proc sort data=&pkds(keep=&lev1 &lev2 &lev3 &lev4 &lev5 &lev6) 
+                proc sort data=&pkds(keep=&lev1 &lev2 &lev3 &lev4 &lev5 &lev6)
                   out=&tmpds noduprec;
                   by _all_;
                 run;
@@ -295,7 +295,7 @@
       %end;
     %end;
   %end;
- 
+
   %if &ppkcnt=6 %then %do;
     %put &sysmacroname: No more PK guess possible;
     %return;

base/mp_lib2cards.sas

@@ -1,21 +1,35 @@
 /**
   @file
   @brief Convert all library members to CARDS files
-  @details Gets list of members then calls the <code>%mp_ds2cards()</code>
-            macro
-    usage:
+  @details Gets list of members then calls the <code>%mp_ds2cards()</code> macro.
+  Usage:
 
-    %mp_lib2cards(lib=sashelp
-        , outloc= C:\temp )
+      %mp_lib2cards(lib=sashelp
+          , outloc= C:\temp )
 
-  <h4> Dependencies </h4>
+  The output will be one cards file in the `outloc` directory per dataset in the
+  input `lib` library.  If the `outloc` directory does not exist, it is created.
+
+  To create a single SAS file with the first 1000 records of each table in a
+  library you could use this syntax:
+
+      %mp_lib2cards(lib=sashelp
+          , outloc= /tmp
+          , outfile= myfile.sas
+          , maxobs= 1000
+      )
+
+  <h4> SAS Macros </h4>
   @li mf_mkdir.sas
+  @li mf_trimstr.sas
   @li mp_ds2cards.sas
 
-  @param lib= Library in which to convert all datasets
-  @param outloc= Location in which to store output.  Defaults to WORK library.
-    Do not use a trailing slash (my/path not my/path/).  No quotes.
-  @param maxobs= limit output to the first <code>maxobs</code> observations
+  @param [in] lib= Library in which to convert all datasets
+  @param [out] outloc= Location in which to store output.  Defaults to WORK
+    library. No quotes.
+  @param [out] outfile= Optional output file NAME - if provided, then will create
+  a single output file instead of one file per input table.
+  @param [in] maxobs= limit output to the first <code>maxobs</code> observations
 
   @version 9.2
   @author Allan Bowe
@@ -25,6 +39,7 @@
     ,outloc=%sysfunc(pathname(work)) /* without trailing slash */
     ,maxobs=max
     ,random_sample=NO
+    ,outfile=0
 )/*/STORE SOURCE*/;
 
 /* Find the tables */
@@ -36,16 +51,28 @@ select distinct lowcase(memname)
   from dictionary.tables
   where upcase(libname)="%upcase(&lib)";
 
+/* trim trailing slash, if provided */
+%let outloc=%mf_trimstr(&outloc,/);
+%let outloc=%mf_trimstr(&outloc,\);
+
 /* create the output directory */
 %mf_mkdir(&outloc)
 
 /* create the cards files */
 %do x=1 %to %sysfunc(countw(&memlist));
-   %let ds=%scan(&memlist,&x);
-   %mp_ds2cards(base_ds=&lib..&ds
-      ,cards_file="&outloc/&ds..sas"
-      ,maxobs=&maxobs
-      ,random_sample=&random_sample)
+  %let ds=%scan(&memlist,&x);
+  %mp_ds2cards(base_ds=&lib..&ds
+    ,maxobs=&maxobs
+    ,random_sample=&random_sample
+  %if "&outfile" ne "0" %then %do;
+    ,append=YES
+    ,cards_file="&outloc/&outfile"
+  %end;
+  %else %do;
+    ,append=NO
+    ,cards_file="&outloc/&ds..sas"
+  %end;
+  )
 %end;
 
 %mend;

base/mp_prevobs.sas

@@ -0,0 +1,88 @@
+/**
+  @file
+  @brief Enables previous observations to be re-instated
+  @details Remembers the last X observations by storing them in a hash table.
+  Is a convenience over the use of lag() or retain, when an entire observation
+  needs to be restored.
+
+  This macro will also restore automatic variables (such as _n_ and _error_).
+
+  Example Usage:
+
+      data example;
+        set sashelp.class;
+        calc_var=_n_*3;
+        %* initialise hash and save from PDV ;
+        %mp_prevobs(INIT,history=2)
+        if _n_ =10 then do;
+          %* fetch previous but 1 record;
+          %mp_prevobs(FETCH,-2) 
+          put _n_= name= age= calc_var=; 
+          %* fetch previous record;
+          %mp_prevobs(FETCH,-1) 
+          put _n_= name= age= calc_var=; 
+          %* reinstate current record ;
+          %mp_prevobs(FETCH,0) 
+          put _n_= name= age= calc_var=;
+        end;
+      run;
+
+  Result:
+
+  <img src="https://imgur.com/PSjHoET.png" alt="mp_prevobs sas" width="400"/>
+
+  Credit is made to `data _null_` for authoring this very helpful paper:
+  https://www.lexjansen.com/pharmasug/2008/cc/CC08.pdf
+
+  @param action Either FETCH a current or previous record, or INITialise.
+  @param record The relative (to current) position of the previous observation 
+   to return.  
+  @param history= The number of records to retain in the hash table. Default=5
+  @param prefix= the prefix to give to the variables used to store the hash name
+   and index. Default=mp_prevobs
+
+  @version 9.2
+  @author Allan Bowe
+
+**/
+
+%macro mp_prevobs(action,record,history=5,prefix=mp_prevobs
+)/*/STORE SOURCE*/;
+%let action=%upcase(&action);
+%let prefix=%upcase(&prefix);
+%let record=%eval((&record+0) * -1);
+
+%if &action=INIT %then %do;
+    
+  if _n_ eq 1 then do; 
+    attrib &prefix._VAR length=$64; 
+    dcl hash &prefix._HASH(ordered:'Y');
+    &prefix._KEY=0;
+    &prefix._HASH.defineKey("&prefix._KEY"); 
+    do while(1); 
+      call vnext(&prefix._VAR); 
+      if &prefix._VAR='' then leave;
+      if &prefix._VAR eq "&prefix._VAR" then continue; 
+      else if &prefix._VAR eq "&prefix._KEY" then continue; 
+      &prefix._HASH.defineData(&prefix._VAR);
+    end; 
+    &prefix._HASH.defineDone(); 
+  end;
+  /* this part has to happen before FETCHing */
+  &prefix._KEY+1;
+  &prefix._rc=&prefix._HASH.add();
+  if &prefix._rc then putlog 'adding' &prefix._rc=;
+  %if &history>0 %then %do;
+    if &prefix._key>&history+1 then 
+      &prefix._HASH.remove(key: &prefix._KEY - &history - 1);
+    if &prefix._rc then putlog 'removing' &prefix._rc=;
+  %end;
+%end;
+%else %if &action=FETCH %then %do;
+  if &record > &prefix._key then putlog "Not enough records in &Prefix._hash yet";
+  else &prefix._rc=&prefix._HASH.find(key: &prefix._KEY - &record);
+  if &prefix._rc then putlog &prefix._rc= " when fetching " &prefix._KEY=
+    "with record &record and " _n_=;
+%end;
+
+%mend;

base/mp_searchdata.sas

@@ -3,7 +3,7 @@
   @brief Searches all data in a library
   @details
   Scans an entire library and creates a copy of any table
-    containing a specific string or numeric value.  Only 
+    containing a specific string OR numeric value.  Only 
     matching records are written out.
     If both a string and numval are provided, the string
     will take precedence.
@@ -25,7 +25,7 @@
   @param outobs= set to a positive integer to restrict the number of observations
   @param filter_text= add a (valid) filter clause to further filter the results
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getvarlist.sas
   @li mf_getvartype.sas
   @li mf_mkdir.sas
@@ -44,9 +44,13 @@
   ,filter_text=%str(1=1)
 )/*/STORE SOURCE*/;
 
-%local table_list table table_num table colnum col start_tm vars type coltype;
+%local table_list table table_num table colnum col start_tm check_tm vars type coltype;
 %put process began at %sysfunc(datetime(),datetime19.);
 
+%if &syscc ge 4 %then %do;
+  %put %str(WAR)NING: SYSCC=&syscc on macro entry;
+  %return;
+%end;
 
 %if &string = %then %let type=N;
 %else %let type=C;
@@ -78,6 +82,7 @@ proc sql
     %put NO COLUMNS IN &lib..&table!  This will be skipped.;
   %end;
   %else %do;
+    %let check_tm=%sysfunc(datetime());
     /* build sql statement */
     create table mpsearch.&table as select * from &lib..&table
       where %unquote(&filter_text) and 
@@ -88,14 +93,19 @@ proc sql
       %let coltype=%mf_getvartype(&lib..&table,&col);
       %if &type=C and &coltype=C %then %do;
         /* if a char column, see if it contains the string */
-        or (&col ? "&string")
+        or ("&col"n ? "&string")
       %end;
       %else %if &type=N and &coltype=N %then %do;
         /* if numeric match exactly */
-        or (&col = &numval)
+        or ("&col"n = &numval)
       %end;
     %end;
     );
+    %put Search query for &table took %sysevalf(%sysfunc(datetime())-&check_tm) seconds;
+    %if &sqlrc ne 0 %then %do;
+      %put %str(WAR)NING: SQLRC=&sqlrc when processing &table;
+      %return;
+    %end;
     %if %mf_nobs(mpsearch.&table)=0 %then %do;
       drop table mpsearch.&table;
     %end;

base/mp_setkeyvalue.sas

@@ -3,10 +3,10 @@
   @brief Logs a key value pair a control dataset
   @details If the dataset does not exist, it is created.  Usage:
 
-    %mp_setkeyvalue(someindex,22,type=N)
-    %mp_setkeyvalue(somenewindex,somevalue)
+      %mp_setkeyvalue(someindex,22,type=N)
+      %mp_setkeyvalue(somenewindex,somevalue)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_existds.sas
 
   @param key Provide a key on which to perform the lookup
@@ -26,7 +26,7 @@
 
   %if not (%mf_existds(&libds)) %then %do;
     data &libds (index=(key/unique));
-      length key $32 valc $256 valn 8 type $1;
+      length key $64 valc $2048 valn 8 type $1;
       call missing(of _all_);
       stop;
     run;

base/mp_streamfile.sas

@@ -1,5 +1,5 @@
 /**
-  @file mp_streamfile.sas
+  @file
   @brief Streams a file to _webout according to content type
   @details Will set headers using appropriate functions (SAS 9 vs Viya) and send
   content as a binary stream.
@@ -11,12 +11,13 @@
 
       %mp_streamfile(contenttype=csv,inloc=/some/where.txt,outname=myfile.txt)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getplatform.sas
   @li mp_binarycopy.sas
 
   @param contenttype= Either TEXT, ZIP, CSV, EXCEL (default TEXT)
   @param inloc= /path/to/file.ext to be sent
+  @param inref= fileref of file to be sent (if provided, overrides `inloc`)
   @param outname= the name of the file, as downloaded by the browser
 
   @author Allan Bowe
@@ -27,6 +28,7 @@
 %macro mp_streamfile(
   contenttype=TEXT
   ,inloc=
+  ,inref=0
   ,outname=
 )/*/STORE SOURCE*/;
 
@@ -42,7 +44,7 @@
   %end;
   %else %if &platform=SASVIYA %then %do;
     filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name='_webout.zip'
-      contenttype='application/zip' 
+      contenttype='application/zip'
       contentdisp="attachment; filename=&outname";
   %end;
 %end;
@@ -56,7 +58,7 @@
   %end;
   %else %if &platform=SASVIYA %then %do;
     filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name='_webout.xls'
-      contenttype='application/vnd.ms-excel' 
+      contenttype='application/vnd.ms-excel'
       contentdisp="attachment; filename=&outname";
   %end;
 %end;
@@ -69,7 +71,7 @@
   %end;
   %else %if &platform=SASVIYA %then %do;
     filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name='_webout.xls'
-      contenttype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' 
+      contenttype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
       contentdisp="attachment; filename=&outname";
   %end;
 %end;
@@ -102,7 +104,7 @@
 %else %if &contentype=HTML %then %do;
   %if &platform=SASVIYA %then %do;
     filename _webout filesrvc parenturi="&SYS_JES_JOB_URI" name="_webout.json"
-      contenttype="text/html"; 
+      contenttype="text/html";
   %end;
 %end;
 %else %do;
@@ -110,6 +112,11 @@
   %return;
 %end;
 
-%mp_binarycopy(inloc="&inloc",outref=_webout)
+%if &inref ne 0 %then %do;
+ %mp_binarycopy(inref=&inref,outref=_webout)
+%end;
+%else %do;
+ %mp_binarycopy(inloc="&inloc",outref=_webout)
+%end;
 
 %mend;

base/mp_testjob.sas

@@ -0,0 +1,92 @@
+/**
+  @file
+  @brief Runs arbitrary code for a specified amount of time
+  @details Executes a series of procs and data steps to enable performance
+  testing of arbitrary jobs.
+
+      %mp_testjob(
+         duration=60*5
+      )
+
+  @param [in] duration= the time in seconds which the job should run for. Actual
+  time may vary, as the check is done in between steps.  Default = 30 (seconds).
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquelibref.sas
+  @li mf_getuniquename.sas
+  @li mf_mkdir.sas
+
+  @version 9.4
+  @author Allan Bowe
+
+**/
+
+%macro mp_testjob(duration=30
+)/*/STORE SOURCE*/;
+%local lib dir ds1 ds2 ds3 start_tm i;
+
+%let start_tm=%sysfunc(datetime());
+%let duration=%sysevalf(&duration);
+
+/* create a temporary library in WORK */
+%let lib=%mf_getuniquelibref();
+%let dir=%mf_getuniquename();
+%mf_mkdir(%sysfunc(pathname(work))/&dir)
+libname &lib "%sysfunc(pathname(work))/&dir";
+
+/* loop through until time expires */
+%let ds1=%mf_getuniquename();
+%let ds2=%mf_getuniquename();
+%let ds3=%mf_getuniquename();
+%do i=0 %to 1;
+
+  /* create big dataset */
+  data &lib..&ds1(compress=no );
+    do x=1 to 1000000;
+      randnum0=ranuni(0)*3;
+      randnum1=ranuni(0)*2;
+      bigchar=repeat('A',300);
+      output;
+    end;
+  run;
+  %if %sysevalf( (%sysfunc(datetime())-&start_tm)>&duration ) %then %goto gate;
+
+  proc summary ;
+    class randnum0 randnum1;
+    output out=&lib..&ds2;
+  run;quit;
+  %if %sysevalf( (%sysfunc(datetime())-&start_tm)>&duration ) %then %goto gate;
+
+  /* add more data */
+  proc sql;
+  create table &lib..&ds3 as
+    select *, ranuni(0)*10 as randnum2
+  from &lib..&ds1
+  order by randnum1;
+  quit;
+  %if %sysevalf( (%sysfunc(datetime())-&start_tm)>&duration ) %then %goto gate;
+
+  proc sort data=&lib..&ds3;
+    by descending x;
+  run;
+  %if %sysevalf( (%sysfunc(datetime())-&start_tm)>&duration ) %then %goto gate;
+
+  /* wait 5 seconds */
+  data _null_;
+    call sleep(5,1);
+  run;
+  %if %sysevalf( (%sysfunc(datetime())-&start_tm)>&duration ) %then %goto gate;
+
+  %let i=0;
+
+%end;
+
+%gate:
+%put time is up!;
+proc datasets lib=&lib kill;
+run;
+quit;
+libname &lib clear;
+
+
+%mend;

base/mp_testwritespeedlibrary.sas

@@ -0,0 +1,59 @@
+/**
+  @file mp_testwritespeedlibrary.sas
+  @brief Tests the write speed of a new table in a SAS library
+  @details Will create a new table of a certain size in an 
+  existing SAS library.  The table will have one column,
+  and will be subsequently deleted.
+      
+      %mp_testwritespeedlibrary(
+        lib=work
+        ,size=0.5
+        ,outds=work.results
+      )
+
+  @param lib= (WORK) The library in which to create the table
+  @param size= (0.1) The size in GB of the table to create
+  @param outds= (WORK.RESULTS) The output dataset to be created.
+
+  <h4> SAS Macros </h4>
+  @li mf_getuniquename.sas
+  @li mf_existds.sas
+
+  @version 9.4
+  @author Allan Bowe
+
+**/
+
+%macro mp_testwritespeedlibrary(lib=WORK
+  ,outds=work.results
+  ,size=0.1
+)/*/STORE SOURCE*/;
+%local ds start;
+
+/* find an unused, unique name for the new table */
+%let ds=%mf_getuniquename();
+%do %until(%mf_existds(&lib..&ds)=0);
+  %let ds=%mf_getuniquename();
+%end;
+
+%let start=%sysfunc(datetime());
+
+data &lib..&ds(compress=no keep=x);
+  header=128*1024;
+  size=(1073741824/8 * &size) - header;
+  do x=1 to size;
+    output;
+  end;
+run;
+
+proc sql;
+drop table &lib..&ds;
+
+data &outds;
+  lib="&lib";
+  start_dttm=put(&start,datetime19.);
+  end_dttm=put(datetime(),datetime19.);
+  duration_seconds=end_dttm-start_dttm;
+run;
+
+%mend;

base/mp_tree.sas

@@ -0,0 +1,68 @@
+/**
+  @file
+  @brief Recursively scans a directory tree to get all subfolders and content
+  @details
+  Usage:
+
+      %mp_tree(dir=/tmp, outds=work.tree)
+
+  Credits:
+
+  * Roger Deangelis, https://communities.sas.com/t5/SAS-Programming/listing-all-files-within-a-directory-and-subdirectories/m-p/332616/highlight/true#M74887
+  * Tom, https://communities.sas.com/t5/SAS-Programming/listing-all-files-of-all-types-from-all-subdirectories/m-p/334113/highlight/true#M75419
+
+
+  @param dir= Directory to be scanned (default=/tmp)
+  @param outds= Dataset to create (default=work.mp_tree)
+
+  @returns outds contains the following variables:
+
+   - `dir`: a flag (1/0) to say whether it is a directory or not.  This is not
+     reliable - folders that you do not have permission to open will be flagged
+     as directories.
+   - `ext`: file extension
+   - `filename`: file name
+   - `dirname`: directory name
+   - `fullpath`: directory + file name
+
+  @version 9.2
+**/
+
+%macro mp_tree(dir=/tmp
+  ,outds=work.mp_tree
+)/*/STORE SOURCE*/;
+
+data &outds ;
+  length dir 8 ext filename dirname $256 fullpath $512 ;
+  call missing(of _all_);
+  fullpath = "&dir";
+run;
+
+%local sep;
+%if &sysscp=WIN or &SYSSCP eq DNTHOST %then %let sep=\;
+%else %let sep=/;
+
+data &outds ;
+  modify &outds ;
+  retain sep "&sep";
+  rc=filename('tmp',fullpath);
+  dir_id=dopen('tmp');
+  dir = (dir_id ne 0) ;
+  if dir then dirname=fullpath;
+  else do;
+    filename=scan(fullpath,-1,sep) ;
+    dirname =substrn(fullpath,1,length(fullpath)-length(filename));
+    if index(filename,'.')>1 then ext=scan(filename,-1,'.');
+  end;
+  replace;
+  if dir then do;
+    do i=1 to dnum(dir_id);
+      fullpath=cats(dirname,sep,dread(dir_id,i));
+      output;
+    end;
+    rc=dclose(dir_id);
+  end;
+  rc=filename('tmp');
+run;
+
+%mend;

base/mp_unzip.sas

@@ -12,7 +12,7 @@
 
       %mp_unzip(ziploc="/some/file.zip",outdir=/some/folder)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_mkdir.sas
   @li mf_getuniquefileref.sas
 

base/mp_updatevarlength.sas

@@ -18,7 +18,7 @@
   @param var The variable to modify
   @param len The new length to apply
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_existds.sas
   @li mp_abort.sas
   @li mf_existvar.sas

base/mp_zip.sas

@@ -12,7 +12,7 @@
   be sure that _debug is not set (else the SPWA will send non zipped content
   as well).
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_dirlist.sas
 
   @param in= unquoted filepath, dataset of files or directory to zip

build.py

@@ -9,19 +9,22 @@ for file in files:
     ml = open('lua/' + name + '.sas', "w")
     ml.write("/**\n")
     ml.write("  @file " + name + '.sas\n')
-    ml.write("  @brief Creates the " + basename + " file\n")
+    ml.write("  @brief Compiles the " + basename + " lua file\n")
     ml.write("  @details Writes " + basename + " to the work directory\n")
+    ml.write("  and then includes it.\n")
     ml.write("  Usage:\n\n")
     ml.write("      %" + name + "()\n\n")
     ml.write("**/\n\n")
     ml.write("%macro " + name + "();\n")
     ml.write("data _null_;\n")
-    ml.write("  file \"%sysfunc(pathname(work))/" + basename + "\";\n")
+    ml.write("  file \"%sysfunc(pathname(work))/" + name + ".lua\";\n")
     with open(file) as infile:
         for line in infile:
           ml.write("  put '" + line.rstrip().replace("'","''") + " ';\n")
-    ml.write("run;\n")
+    ml.write("run;\n\n")
+    ml.write("%inc \"%sysfunc(pathname(work))/" + name + ".lua\";\n\n")
     ml.write("%mend;\n")
+
 ml.close()
 
 # prepare web files

doxy/build.sh

@@ -1,40 +0,0 @@
-#!/bin/bash
-####################################################################
-# PROJECT: Macro Core Docs Build                                   #
-####################################################################
-
-BUILD_FOLDER="/tmp/macrocore_docs"
-
-# move to project root
-cd ..
-
-# create build directory
-rm -rf $BUILD_FOLDER
-mkdir $BUILD_FOLDER
-
-# copy relevant files
-cp -r base $BUILD_FOLDER
-cp -r meta $BUILD_FOLDER
-cp -r metax $BUILD_FOLDER
-cp -r viya $BUILD_FOLDER
-cp -r doxy $BUILD_FOLDER
-cp main.dox $BUILD_FOLDER
-cp doxy/Doxyfile $BUILD_FOLDER
-
-# update Doxyfile and generate
-cd $BUILD_FOLDER
-echo "OUTPUT_DIRECTORY=$BUILD_FOLDER/out" >> $BUILD_FOLDER/Doxyfile
-echo "INPUT+=main.dox" >> $BUILD_FOLDER/Doxyfile
-doxygen Doxyfile
-
-# refresh github pages site
-git clone git@github.com:sasjs/core.github.io.git
-cd core.github.io
-rm -r *
-mv $BUILD_FOLDER/out/doxy/* .
-echo 'core.sasjs.io' > CNAME
-git add *
-git commit -m "build.sh build on $(date +%F:%H:%M:%S)"
-git push
-
-echo "check it out:  https://sasjs.github.io/core.github.io/files.html"

doxy/new_footer.html

@@ -1,22 +0,0 @@
-<!--BEGIN GENERATE_TREEVIEW-->
-    <li class="footer"><b>$generatedby</b>
-    <a href="http://www.doxygen.org/index.html">
-    <img class="footer" src="doxygen.png" alt="doxygen"/></a>
-   <i> For more information visit the </i> <a href="https://github.com/sasjs/core">Macro Core library</a>.</li>
-
-   </ul>
- </div>
-<!--END GENERATE_TREEVIEW-->
-<!--BEGIN !GENERATE_TREEVIEW-->
-<hr class="footer"/>
-<table width="100%"><tbody><tr><td>
-For more information visit the <a href="https://github.com/sasjs/core">Macro Core library</a>.
-</td><td><address class="footer"><small>
-&copy;$year<br/>
-$generatedby <a href="http://www.doxygen.org/index.html">
-<!--<img class="footer" src="$relpath$doxygen.png" alt="doxygen"/>-->doxygen
-</a> $doxygenversion
-</small></address></tr></tbody></table>
-<!--END !GENERATE_TREEVIEW-->
-</body>
-</html>

doxy/new_header.html

@@ -1,72 +0,0 @@
-<!-- HTML header for doxygen 1.8.14-->
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<html xmlns="http://www.w3.org/1999/xhtml">
-<head>
-<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
-<meta http-equiv="X-UA-Compatible" content="IE=9"/>
-<meta name="generator" content="Doxygen $doxygenversion"/>
-<meta name="viewport" content="width=device-width, initial-scale=1"/>
-<!--BEGIN PROJECT_NAME--><title>$projectname: $title</title><!--END PROJECT_NAME-->
-<!--BEGIN !PROJECT_NAME--><title>$title</title><!--END !PROJECT_NAME-->
-<link href="$relpath^tabs.css" rel="stylesheet" type="text/css"/>
-<script type="text/javascript" src="$relpath^jquery.js"></script>
-<script type="text/javascript" src="$relpath^dynsections.js"></script>
-$treeview
-$search
-$mathjax
-<link href="$relpath^$stylesheet" rel="stylesheet" type="text/css" />
-<link REL="icon" HREF="https://sasjs.io/img/runningman.jpg">
-
-$extrastylesheet
-</head>
-<body>
-<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
-
-<!--BEGIN TITLEAREA-->
-<div id="titlearea" style='background-color:white' >
-<table cellspacing="0" cellpadding="0">
- <tbody>
- <tr style="height: 26px;">
-  <!--BEGIN PROJECT_LOGO-->
-  <td id="projectlogo"></td>
-
-  <!--END PROJECT_LOGO-->
-<!--BEGIN PROJECT_NAME-->
-  <td>
-  <div id="projectname">
-    <!--a href=".">
-      <img alt="Macro Core" src="https://macropeople.com/wp-content/uploads/2018/05/macropeople2014retina_V2.png" height=60/>
-     </a-->
-     <a href=".">
-        <img alt="Macro Core" src="./Macro_core_website_1.png" height=60/>
-       </a>
-   </div>
-  </td>
-  <td>
-    <table style="padding-left: 2em;" cellspacing="0" cellpadding="0">
-      <tr><td> Production Ready Macros for SAS Application Developers</td></tr>
-      <tr><td><a href="https://github.com/sasjs/core">
-          https://github.com/sasjs/core
-      </a></td></tr>
-    </table>
-  </td>
-  <!--END PROJECT_NAME-->
-  <!--BEGIN !PROJECT_NAME-->
-   <!--BEGIN PROJECT_BRIEF-->
-   <!--END PROJECT_BRIEF-->
-  <!--END !PROJECT_NAME-->
-  <div class="header">
-  <div class="headertitle">
-
-  <!--BEGIN DISABLE_INDEX-->
-   <!--BEGIN SEARCHENGINE-->
-
-   <td>$searchbox</td>
-   <!--END SEARCHENGINE-->
-  <!--END DISABLE_INDEX-->
- </tr>
- </tbody>
-</table>
-</div>
-<!--END TITLEAREA-->
-<!-- end header part -->

lua/json.lua

@@ -1,5 +1,5 @@
 --
--- json2sas.lua  (modified from json.lua)
+-- json.lua
 --
 -- Copyright (c) 2019 rxi
 --
@@ -22,7 +22,7 @@
 -- SOFTWARE.
 --
 
-local json2sas = { _version = "0.1.2" }
+json = { _version = "0.1.2" }
 
 -------------------------------------------------------------------------------
 -- Encode
@@ -122,7 +122,7 @@ encode = function(val, stack)
   error("unexpected type '" .. t .. "'")
 end
 
-function json2sas.encode(val)
+function json.encode(val)
   return ( encode(val) )
 end
 
@@ -356,7 +356,7 @@ parse = function(str, idx)
   decode_error(str, idx, "unexpected character '" .. chr .. "'")
 end
 
-function json2sas.decode(str)
+function json.decode(str)
   if type(str) ~= "string" then
     error("expected argument of type string, got " .. type(str))
   end
@@ -368,88 +368,4 @@ function json2sas.decode(str)
   return res
 end
 
--- convert macro variable array into one variable and decode
-function json2sas.go(macvar)
-  local x=1
-  local cnt=0
-  local mac=sas.symget(macvar..'0')
-  local newstr=''
-  if mac and mac ~= '' then
-    cnt=mac
-    for x=1,cnt,1 do
-      mac=sas.symget(macvar..x)
-      if mac and mac ~= '' then
-        newstr=newstr..mac
-      else
-        return print(macvar..x..' NOT FOUND!!')
-      end
-    end
-  else
-    return print(macvar..'0 NOT FOUND!!')
-  end
-  -- print('mac:'..mac..'cnt:'..cnt..'newstr'..newstr)
-  local oneVar=json2sas.decode(newstr)
-  local jsdata=oneVar["data"]
-  local meta={}
-  local attrs={}
-  for tablename, data in pairs(jsdata) do -- each table
-    print("Processing table: "..tablename)
-    attrs[tablename]={}
-    for k, v in ipairs(data) do -- each row
-      if(k==1) then  -- column names
-        for a, b in pairs(v) do
-          attrs[tablename][a]={}
-          attrs[tablename][a]["name"]=b
-        end
-      elseif(k==2) then  -- get types
-        for a, b in pairs(v) do
-  	      if type(b)=='number' then
-  	        attrs[tablename][a]["type"]="N"
-  	        attrs[tablename][a]["length"]=8
-  	      else
-            attrs[tablename][a]["type"]="C"
-            attrs[tablename][a]["length"]=string.len(b)
-  	      end
-        end
-      else  --update lengths
-        for a, b in pairs(v) do
-          if (type(b)=='string' and string.len(b)>attrs[tablename][a]["length"])
-          then
-            attrs[tablename][a]["length"]=string.len(b)
-          end
-        end
-  	  end
-  	end
-    print(json2sas.encode(attrs[tablename])) -- show results
-
-    -- Now create the SAS table
-    sas.new_table("work."..tablename,attrs[tablename])
-    local dsid=sas.open("work."..tablename, "u")
-    for k, v in ipairs(data) do
-      if k>1 then
-        sas.append(dsid)
-        for a, b in pairs(v) do
-          sas.put_value(dsid, attrs[tablename][a]["name"], b)
-        end
-        sas.update(dsid)
-      end
-    end
-    sas.close(dsid)
-  end
-  return json2sas.decode(newstr)
-end
-
-
-function quote(str)
-  return sas.quote(str)
-end
-function sasvar(str)
-  print("processing: "..str)
-  print(sas.symexist(str))
-  if sas.symexist(str)==1 then
-    return quote(str)..':'..quote(sas.symget(str))..','
-  end
-  return ''
-end
-
-return json2sas
+return json

lua/ml_json.sas

@@ -1,18 +1,19 @@
 /**
-  @file ml_json2sas.sas
-  @brief Creates the json2sas.lua file
-  @details Writes json2sas.lua to the work directory
+  @file ml_json.sas
+  @brief Compiles the json.lua lua file
+  @details Writes json.lua to the work directory
+  and then includes it.
   Usage:
 
-      %ml_json2sas()
+      %ml_json()
 
 **/
 
-%macro ml_json2sas();
+%macro ml_json();
 data _null_;
-  file "%sysfunc(pathname(work))/json2sas.lua";
+  file "%sysfunc(pathname(work))/ml_json.lua";
   put '-- ';
-  put '-- json2sas.lua  (modified from json.lua) ';
+  put '-- json.lua ';
   put '-- ';
   put '-- Copyright (c) 2019 rxi ';
   put '-- ';
@@ -35,7 +36,7 @@ data _null_;
   put '-- SOFTWARE. ';
   put '-- ';
   put ' ';
-  put 'local json2sas = { _version = "0.1.2" } ';
+  put 'json = { _version = "0.1.2" } ';
   put ' ';
   put '------------------------------------------------------------------------------- ';
   put '-- Encode ';
@@ -135,7 +136,7 @@ data _null_;
   put '  error("unexpected type ''" .. t .. "''") ';
   put 'end ';
   put ' ';
-  put 'function json2sas.encode(val) ';
+  put 'function json.encode(val) ';
   put '  return ( encode(val) ) ';
   put 'end ';
   put ' ';
@@ -369,7 +370,7 @@ data _null_;
   put '  decode_error(str, idx, "unexpected character ''" .. chr .. "''") ';
   put 'end ';
   put ' ';
-  put 'function json2sas.decode(str) ';
+  put 'function json.decode(str) ';
   put '  if type(str) ~= "string" then ';
   put '    error("expected argument of type string, got " .. type(str)) ';
   put '  end ';
@@ -381,90 +382,9 @@ data _null_;
   put '  return res ';
   put 'end ';
   put ' ';
-  put '-- convert macro variable array into one variable and decode ';
-  put 'function json2sas.go(macvar) ';
-  put '  local x=1 ';
-  put '  local cnt=0 ';
-  put '  local mac=sas.symget(macvar..''0'') ';
-  put '  local newstr='''' ';
-  put '  if mac and mac ~= '''' then ';
-  put '    cnt=mac ';
-  put '    for x=1,cnt,1 do ';
-  put '      mac=sas.symget(macvar..x) ';
-  put '      if mac and mac ~= '''' then ';
-  put '        newstr=newstr..mac ';
-  put '      else ';
-  put '        return print(macvar..x..'' NOT FOUND!!'') ';
-  put '      end ';
-  put '    end ';
-  put '  else ';
-  put '    return print(macvar..''0 NOT FOUND!!'') ';
-  put '  end ';
-  put '  -- print(''mac:''..mac..''cnt:''..cnt..''newstr''..newstr) ';
-  put '  local oneVar=json2sas.decode(newstr) ';
-  put '  local jsdata=oneVar["data"] ';
-  put '  local meta={} ';
-  put '  local attrs={} ';
-  put '  for tablename, data in pairs(jsdata) do -- each table ';
-  put '    print("Processing table: "..tablename) ';
-  put '    attrs[tablename]={} ';
-  put '    for k, v in ipairs(data) do -- each row ';
-  put '      if(k==1) then  -- column names ';
-  put '        for a, b in pairs(v) do ';
-  put '          attrs[tablename][a]={} ';
-  put '          attrs[tablename][a]["name"]=b ';
-  put '        end ';
-  put '      elseif(k==2) then  -- get types ';
-  put '        for a, b in pairs(v) do ';
-  put '  	      if type(b)==''number'' then ';
-  put '  	        attrs[tablename][a]["type"]="N" ';
-  put '  	        attrs[tablename][a]["length"]=8 ';
-  put '  	      else ';
-  put '            attrs[tablename][a]["type"]="C" ';
-  put '            attrs[tablename][a]["length"]=string.len(b) ';
-  put '  	      end ';
-  put '        end ';
-  put '      else  --update lengths ';
-  put '        for a, b in pairs(v) do ';
-  put '          if (type(b)==''string'' and string.len(b)>attrs[tablename][a]["length"]) ';
-  put '          then ';
-  put '            attrs[tablename][a]["length"]=string.len(b) ';
-  put '          end ';
-  put '        end ';
-  put '  	  end ';
-  put '  	end ';
-  put '    print(json2sas.encode(attrs[tablename])) -- show results ';
-  put ' ';
-  put '    -- Now create the SAS table ';
-  put '    sas.new_table("work."..tablename,attrs[tablename]) ';
-  put '    local dsid=sas.open("work."..tablename, "u") ';
-  put '    for k, v in ipairs(data) do ';
-  put '      if k>1 then ';
-  put '        sas.append(dsid) ';
-  put '        for a, b in pairs(v) do ';
-  put '          sas.put_value(dsid, attrs[tablename][a]["name"], b) ';
-  put '        end ';
-  put '        sas.update(dsid) ';
-  put '      end ';
-  put '    end ';
-  put '    sas.close(dsid) ';
-  put '  end ';
-  put '  return json2sas.decode(newstr) ';
-  put 'end ';
-  put ' ';
-  put ' ';
-  put 'function quote(str) ';
-  put '  return sas.quote(str) ';
-  put 'end ';
-  put 'function sasvar(str) ';
-  put '  print("processing: "..str) ';
-  put '  print(sas.symexist(str)) ';
-  put '  if sas.symexist(str)==1 then ';
-  put '    return quote(str)..'':''..quote(sas.symget(str))..'','' ';
-  put '  end ';
-  put '  return '''' ';
-  put 'end ';
-  put ' ';
-  put 'return json2sas ';
+  put 'return json ';
 run;
+
+%inc "%sysfunc(pathname(work))/ml_json.lua";
+
 %mend;

main.dox

@@ -13,6 +13,11 @@
         * Not metadata aware
         * No X command
         * Prefixes:  _mf_, _mp_
+
+    Macros starting `mf_` are macro _functions_ and can be used in assignment
+    statements.  Those starting `mp_` are macro _procedures_, which generate
+    SAS statements, and must therefore be applied accordingly.
+
  */
 
 /*! \dir meta
@@ -45,4 +50,15 @@
         * No X command
         * Prefixes: _mv_
 
+ */
+
+ /*! \dir lua
+ *  \brief Lua macros
+ *  \details These macros have the following attributes:
+
+        * OS independent
+        * Work as LUA functions (they are immediately executed/compiled)
+        * Auto-generated from the plain source `.lua` files in the same directory
+        * Prefixes: _ml_
+
  */

meta/mm_assigndirectlib.sas

@@ -14,7 +14,7 @@
       disconnect from MyAlias;
       quit;
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getengine.sas
   @li mp_abort.sas
 

meta/mm_assignlib.sas

@@ -10,7 +10,7 @@
 
       %mm_assignlib(SOMEREF)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
 
   @param libref the libref (not name) of the metadata library
@@ -38,6 +38,7 @@
       rc=metadata_getattr(liburi,"Name",LibName);
       /* now try and assign it */
       if libname("&libref",,'meta',cats('liburi="',liburi,'";')) ne 0 then do;
+        putlog "&libref could not be assigned";
         call symputx('msg',sysmsg(),'l');
         if "&mabort"='HARD' then call symputx('mp_abort',1,'l');
       end;

meta/mm_createapplication.sas

@@ -15,7 +15,7 @@
 
   @warning application components do not get deleted when removing the container folder!  be sure you have the administrative priviliges to remove this kind of metadata from the SMC plugin (or be ready to do to so programmatically).
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
   @li mf_verifymacvars.sas
 

meta/mm_createdataset.sas

@@ -18,7 +18,7 @@
 
     %mm_createdataset(tableuri=G5X8AFW1.BE00015Y)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mm_getlibs.sas
   @li mm_gettables.sas
   @li mm_getcols.sas

meta/mm_createdocument.sas

@@ -12,7 +12,7 @@
       %mm_createdocument(tree=/User Folders/sasdemo
         ,name=MyNote)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
   @li mf_verifymacvars.sas
 

meta/mm_createlibrary.sas

@@ -9,17 +9,17 @@
 
   Usage:
 
-    %mm_createlibrary(
-       libname=My New Library
-      ,libref=mynewlib
-      ,libdesc=Super & <fine>
-      ,engine=BASE
-      ,tree=/User Folders/sasdemo
-      ,servercontext=SASApp
-      ,directory=/tmp/tests
-      ,mDebug=1)
+      %mm_createlibrary(
+        libname=My New Library
+        ,libref=mynewlib
+        ,libdesc=Super & <fine>
+        ,engine=BASE
+        ,tree=/User Folders/sasdemo
+        ,servercontext=SASApp
+        ,directory=/tmp/tests
+        ,mDebug=1)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_verifymacvars.sas
   @li mm_createfolder.sas
 

meta/mm_createstp.sas

@@ -39,7 +39,7 @@
         ,Server=SASApp
         ,stptype=2)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_nobs.sas
   @li mf_verifymacvars.sas
   @li mm_getdirectories.sas

meta/mm_createwebservice.sas

@@ -24,7 +24,7 @@ Usage:
     ;;;;
     %mm_createwebservice(path=/Public/app/common,name=appInit,code=ft15f001,replace=YES)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mm_createstp.sas
   @li mf_getuser.sas
   @li mm_createfolder.sas

meta/mm_deletedocument.sas

@@ -8,7 +8,7 @@
     %mm_createdocument(tree=/User Folders/&sysuserid,name=MyNote)
     %mm_deletedocument(target=/User Folders/&sysuserid/MyNote)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
 
   @param target= full path to the document being deleted
 

meta/mm_deletestp.sas

@@ -7,7 +7,7 @@
 
     %mm_deletestp(target=/some/meta/path/myStoredProcess)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
 
   @param target= full path to the STP being deleted
 

meta/mm_getauthinfo.sas

@@ -7,7 +7,7 @@
 
   @param outds= the ONE LEVEL work dataset to create
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mm_getobjects.sas
   @li mf_getuniquefileref.sas
   @li mm_getdetails.sas

meta/mm_getdocument.sas

@@ -11,7 +11,7 @@
         ,outref=/some/unquoted/filename.ext
       )
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
 
   @param tree= The metadata path of the document
@@ -124,6 +124,7 @@ data _null_;
         when ('&#x0a;') rec='0A'x;
         when ('&#x0d;') rec='0D'x;
         when ('&#36;' ) rec='$'  ;
+        when ('&#x09;') rec='09'x;
         otherwise putlog "WARNING: missing value for " entity=;
       end;
       rc =fput(fileid, substr(rec,1,1));

meta/mm_getfoldermembers.sas

@@ -0,0 +1,95 @@
+/**
+  @file
+  @brief Returns all direct child members of a particular folder
+  @details Displays the children for a particular folder, in a similar fashion
+  to the viya counterpart (mv_getfoldermembers.sas)
+
+  Usage:
+
+      %mm_getfoldermembers(root=/, outds=rootfolders)
+
+      %mm_getfoldermembers(root=/User Folders/&sysuserid, outds=usercontent)
+
+  @param [in] root= the parent folder under which to return all contents
+  @param [out] outds= the dataset to create that contains the list of directories
+  @param [in] mDebug= set to 1 to show debug messages in the log
+
+  <h4> Data Outputs </h4>
+
+  Example for `root=/`:
+
+  |metauri $17|metaname $256|metatype $32|
+  |---|---|---|
+  |A5XLSNXI.AA000001|Products	|Folder|
+  |A5XLSNXI.AA000002|Shared Data	|Folder|
+  |A5XLSNXI.AA000003|User Folders	|Folder|
+  |A5XLSNXI.AA000004|System	|Folder|
+  |A5XLSNXI.AA00003K|30.SASApps	|Folder|
+  |A5XLSNXI.AA00006A|Public|Folder|
+
+  <h4> SAS Macros </h4>
+  @li mm_getfoldertree.sas
+  @li mf_getuniquefileref.sas
+  @li mf_getuniquelibref.sas
+
+  @version 9.4
+  @author Allan Bowe
+
+**/
+%macro mm_getfoldermembers(
+     root=
+    ,outds=work.mm_getfoldertree
+)/*/STORE SOURCE*/;
+
+%if "&root" = "/" %then %do;
+  %local fname1 fname2 fname3;
+  %let fname1=%mf_getuniquefileref();
+  %let fname2=%mf_getuniquefileref();
+  %let fname3=%mf_getuniquefileref();
+  data _null_ ;
+    file &fname1 ;
+    put '<GetMetadataObjects>' ;
+    put '<Reposid>$METAREPOSITORY</Reposid>' ;
+    put '<Type>Tree</Type>' ;
+    put '<NS>SAS</NS>' ;
+    put '<Flags>388</Flags>' ;
+    put '<Options>' ;
+    put '<XMLSelect search="Tree[SoftwareComponents/SoftwareComponent'@;
+    put '[@Name=''BIP Service'']]"/>';
+    put '</Options>' ;
+    put '</GetMetadataObjects>' ;
+  run ;
+  proc metadata in=&fname1 out=&fname2 verbose;run;
+
+  /* create an XML map to read the response */
+  data _null_;
+    file &fname3;
+    put '<SXLEMAP version="1.2" name="SASFolders">';
+    put '<TABLE name="SASFolders">';
+    put '<TABLE-PATH syntax="XPath">//Objects/Tree</TABLE-PATH>';
+    put '<COLUMN name="metauri">><LENGTH>17</LENGTH>';
+    put '<PATH syntax="XPath">//Objects/Tree/@Id</PATH></COLUMN>';
+    put '<COLUMN name="metaname"><LENGTH>256</LENGTH>>';
+    put '<PATH syntax="XPath">//Objects/Tree/@Name</PATH></COLUMN>';
+    put '</TABLE></SXLEMAP>';
+  run;
+  %local libref1;
+  %let libref1=%mf_getuniquelibref();
+  libname &libref1 xml xmlfileref=&fname2 xmlmap=&fname3;
+
+  data &outds;
+    length metatype $32;
+    retain metatype 'Folder';
+    set &libref1..sasfolders;
+  run;
+
+%end;
+%else %do;
+  %mm_getfoldertree(root=&root, outds=&outds,depth=1)
+  data &outds;
+    set &outds(rename=(name=metaname publictype=metatype));
+    keep metaname metauri metatype;
+  run;
+%end;
+
+%mend;

meta/mm_getfoldertree.sas

@@ -1,20 +1,22 @@
 /**
-  @file mm_getfoldertree.sas
+  @file
   @brief Returns all folders / subfolder content for a particular root
   @details Shows all members and SubTrees recursively for a particular root.
   Note - for big sites, this returns a lot of data!  So you may wish to reduce
-  the logging to speed up the process (see example below)
+  the logging to speed up the process (see example below), OR - use mm_tree.sas
+  which uses proc metadata and is far more efficient.
+
   Usage:
 
     options ps=max nonotes nosource;
     %mm_getfoldertree(root=/My/Meta/Path, outds=iwantthisdataset)
     options notes source;
-    
-  @param root= the parent folder under which to return all contents
-  @param outds= the dataset to create that contains the list of directories
-  @param mDebug= set to 1 to show debug messages in the log
 
-  <h4> Dependencies </h4>
+  @param [in] root= the parent folder under which to return all contents
+  @param [out] outds= the dataset to create that contains the list of directories
+  @param [in] mDebug= set to 1 to show debug messages in the log
+
+  <h4> SAS Macros </h4>
 
   @version 9.4
   @author Allan Bowe
@@ -60,7 +62,7 @@ data &outds.TMP/view=&outds.TMP;
     __n1+1;
     /* Walk through all possible associations of this object. */
     __n2=1;
-    if assoctype in ('Members','SubTrees') then 
+    if assoctype in ('Members','SubTrees') then
     do while(metadata_getnasn(pathuri,assoctype,__n2,metauri)>0);
       __n2+1;
       call missing(name,publictype,MetadataUpdated,MetadataCreated);

meta/mm_getgroups.sas

@@ -5,17 +5,15 @@
     blank to return all groups.
   Usage:
 
-    - all groups
-    %mm_getGroups()
+    - all groups: `%mm_getGroups()`
 
-    - all groups for a particular user
-    %mm_getgroups(user=&sysuserid)
+    - all groups for a particular user: `%mm_getgroups(user=&sysuserid)`
 
-  @param user= the metadata user to return groups for.  Leave blank for all
+  @param [in] user= the metadata user to return groups for.  Leave blank for all
     groups.
-  @param outds= the dataset to create that contains the list of groups
-  @param repo= the metadata repository that contains the user/group information
-  @param mDebug= set to 1 to show debug messages in the log
+  @param [in] repo= the metadata repository that contains the user/group information
+  @param [in] mDebug= set to 1 to show debug messages in the log
+  @param [out] outds= the dataset to create that contains the list of groups
 
   @returns outds  dataset containing all groups in a column named "metagroup"
    - groupuri

meta/mm_getroles.sas

@@ -5,9 +5,9 @@
 
   Usage:
 
-    %mm_getroles()
+      %mm_getroles()
 
-  @param outds the dataset to create that contains the list of roles
+  @param [out] outds the dataset to create that contains the list of roles
 
   @returns outds  dataset containing all roles, with the following columns:
     - uri

meta/mm_getservercontexts.sas

@@ -14,7 +14,7 @@
       filename __mc2 clear;
       libname __mc3 clear;
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mm_getrepos.sas
 
   @version 9.3

meta/mm_getstpcode.sas

@@ -123,6 +123,7 @@ data _null_;
         when ('
') rec='0A'x;
         when ('
') rec='0D'x;
         when ('$' ) rec='$'  ;
+        when ('	') rec='09'x;
         otherwise putlog "%str(WARN)ING: missing value for " entity=;
       end;
       rc =fput(fileid, substr(rec,1,1));

meta/mm_getstps.sas

@@ -14,7 +14,7 @@
 
       %mm_getstps(tree=/My Folder/My STPs, name=My STP)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mm_gettree.sas
 
   @param tree= the metadata folder location in which to search.  Leave blank
@@ -23,8 +23,8 @@
     combine with the <code>tree=</code> parameter.
   @param outds= the dataset to create that contains the list of stps.
   @param mDebug= set to 1 to show debug messages in the log
-  @showDesc= provide a non blank value to return stored process descriptions
-  @showUsageVersion= provide a non blank value to return the UsageVersion.  This
+  @param showDesc= provide a non blank value to return stored process descriptions
+  @param showUsageVersion= provide a non blank value to return the UsageVersion.  This
     is either 1000000 (type 1, 9.2) or 2000000 (type2, 9.3 onwards).
 
   @returns outds  dataset containing the following columns

meta/mm_getwebappsrvprops.sas

@@ -1,7 +1,7 @@
 /**
   @file
   @brief Retrieves properties of the SAS web app server
-  @details 
+  @details
   Usage:
 
       %mm_getwebappsrvprops(outds= some_ds)
@@ -21,8 +21,7 @@
       libname __shake clear;
 
   @version 9.4
-  @author Allan Bowe
-  @source https://github.com/sasjs/core
+  @author Allan Bowe https://github.com/sasjs/core
 
 **/
 
@@ -99,6 +98,7 @@ run;
           when ('
') rec='0A'x;
           when ('
') rec='0D'x;
           when ('$' ) rec='$'  ;
+          when ('	') rec='09'x;
           otherwise putlog "WARNING: missing value for " entity=;
         end;
         rc =fput(fileid, substr(rec,1,1));

meta/mm_spkexport.sas

@@ -45,7 +45,7 @@
       ./mmscript.sh "myuser" "mypass"
 
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_loc.sas
   @li mm_tree.sas
   @li mf_getuniquefileref.sas

meta/mm_tree.sas

@@ -46,7 +46,7 @@
             Table
         ,outds=morestuff)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_getquotedstr.sas
   @li mm_getpublictypes.sas
   @li mf_isblank.sas

meta/mm_updatestpservertype.sas

@@ -1,14 +1,13 @@
 /**
-  @file mm_updatestpservertype.sas
+  @file
   @brief Updates a type 2 stored process to run on STP or WKS context
   @details Only works on Type 2 (9.3 compatible) STPs
 
   Usage:
 
-    %mm_updatestpservertype(target=/some/meta/path/myStoredProcess
-      ,type=WKS)
+      %mm_updatestpservertype(target=/some/meta/path/myStoredProcess
+        ,type=WKS)
 
-  <h4> Dependencies </h4>
 
   @param target= full path to the STP being deleted
   @param type= Either WKS or STP depending on whether Workspace or Stored Process

meta/mm_updatestpsourcecode.sas

@@ -6,31 +6,40 @@
 
   Usage:
 
-    %mm_updatestpsourcecode(stp=/my/metadata/path/mystpname
-      ,stpcode="/file/system/source.sas")
+      %mm_updatestpsourcecode(stp=/my/metadata/path/mystpname
+        ,stpcode="/file/system/source.sas")
 
-
-  @param stp= the BIP Tree folder path plus Stored Process Name
-  @param stpcode= the source file (or fileref) containing the SAS code to load
+  @param [in] stp= the BIP Tree folder path plus Stored Process Name
+  @param [in] stpcode= the source file (or fileref) containing the SAS code to load
     into the stp.  For multiple files, they should simply be concatenated first.
-  @param minify= set to YES in order to strip comments, blank lines, and CRLFs.
+  @param [in] minify= set to YES in order to strip comments, blank lines, and CRLFs.
 
-  @param frefin= change default inref if it clashes with an existing one
-  @param frefout= change default outref if it clashes with an existing one
+  @param frefin= deprecated - a unique fileref is now always used
+  @param frefout= deprecated - a unique fileref is now always used
   @param mDebug= set to 1 to show debug messages in the log
 
   @version 9.3
   @author Allan Bowe
 
+  <h4> SAS Macros </h4>
+  @li mf_getuniquefileref.sas
+
 **/
 
 %macro mm_updatestpsourcecode(stp=
   ,stpcode=
   ,minify=NO
+  ,mdebug=0
+  /* deprecated */
   ,frefin=inmeta
   ,frefout=outmeta
-  ,mdebug=0
 );
+
+%if &frefin ne inmeta or &frefout ne outmeta %then %do;
+  %put %str(WARN)ING: the frefin and frefout parameters will be deprecated in
+    an upcoming release.;
+%end;
+
 /* first, check if STP exists */
 %local tsuri;
 %let tsuri=stopifempty ;
@@ -68,7 +77,9 @@ run;
   %return;
 %end;
 
-filename &frefin temp lrecl=32767;
+%local frefin frefout;
+%let frefin=%mf_getuniquefileref();
+%let frefout=%mf_getuniquefileref();
 
 /* write header XML */
 data _null_;
@@ -81,7 +92,7 @@ run;
 /* write contents */
 %if %length(&stpcode)>2 %then %do;
   data _null_;
-    file &frefin mod;
+    file &frefin lrecl=32767 mod;
     infile &stpcode lrecl=32767;
     length outstr $32767;
     input outstr ;
@@ -110,9 +121,6 @@ data _null_;
     </UpdateMetadata>";
 run;
 
-
-filename &frefout temp;
-
 proc metadata in= &frefin out=&frefout;
 run;
 
@@ -124,5 +132,9 @@ run;
     put _infile_;
   run;
 %end;
+%else %do;
+  filename &frefin clear;
+  filename &frefout clear;
+%end;
 
 %mend;

metax/mmx_deletemetafolder.sas

@@ -9,7 +9,7 @@
 
     %mmx_deletemetafolder(loc=/some/meta/folder,user=sasdemo,pass=mars345)
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_loc.sas
 
   @param loc= the metadata folder to delete

metax/mmx_spkexport.sas

@@ -25,7 +25,7 @@ Usage:
         ,outspkpath=%str(/tmp)
     )
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mf_loc.sas
   @li mm_tree.sas
   @li mf_getuniquefileref.sas

package-lock.json

@@ -1,4 +1,13 @@
 {
-  "name": "macrocore",
-  "lockfileVersion": 1
+  "name": "@sasjs/core",
+  "version": "1.0.0",
+  "lockfileVersion": 2,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "@sasjs/core",
+      "version": "1.0.0",
+      "license": "MIT"
+    }
+  }
 }

package.json

@@ -10,13 +10,24 @@
   "author": "Allan Bowe <support@macropeople.com>",
   "repository": {
     "type": "git",
-    "url": "https://github.com/sasjs/core"
+    "url": "git+https://github.com/sasjs/core.git"
   },
   "release": {
-    "branches": ["main"]
+    "branches": [
+      "main"
+    ]
   },
   "publishConfig": {
     "access": "public"
   },
-  "devDependencies": {}
+  "bugs": {
+    "url": "https://github.com/sasjs/core/issues"
+  },
+  "homepage": "https://github.com/sasjs/core#readme",
+  "version": "1.0.0",
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1",
+    "docs": "sasjs doc && ./sasjs/utils/build.sh"
+  }
 }

sasjs/doxy/Doxyfile

@@ -1,5 +1,5 @@
 ALPHABETICAL_INDEX     = NO
-DISABLE_INDEX          = YES
+DISABLE_INDEX          = NO
 ENABLE_PREPROCESSING   = NO
 EXTENSION_MAPPING      = sas=Java ddl=Java
 EXTRACT_LOCAL_CLASSES  = NO
@@ -13,17 +13,21 @@ HIDE_IN_BODY_DOCS      = YES
 HIDE_SCOPE_NAMES       = YES
 HIDE_UNDOC_CLASSES     = YES
 HIDE_UNDOC_MEMBERS     = YES
-HTML_OUTPUT            = doxy
-HTML_HEADER            = ./doxy/new_header.html
-HTML_FOOTER            = ./doxy/new_footer.html
-HTML_EXTRA_STYLESHEET  = ./doxy/new_stylesheet.css
+HTML_OUTPUT            = $(DOXY_HTML_OUTPUT)
+HTML_HEADER            = $(DOXY_CONTENT)new_header.html
+HTML_FOOTER            = $(DOXY_CONTENT)new_footer.html
+HTML_EXTRA_STYLESHEET  = $(DOXY_CONTENT)new_stylesheet.css
 INHERIT_DOCS           = NO
 INLINE_INFO            = NO
-INPUT                  = base meta metax viya
-LAYOUT_FILE            = ./doxy/DoxygenLayout.xml
+INPUT                  = $(DOXY_CONTENT)../../README.md \
+                       = $(DOXY_CONTENT)../../main.dox \
+                         $(DOXY_INPUT)
+USE_MDFILE_AS_MAINPAGE = README.md
+LAYOUT_FILE            = $(DOXY_CONTENT)DoxygenLayout.xml
 MAX_INITIALIZER_LINES  = 0
 PROJECT_NAME           = Macro Core
-PROJECT_LOGO           = doxy/Macro_core_website_1.png
+PROJECT_LOGO           = $(DOXY_CONTENT)Macro_core_website_1.png
+PROJECT_BRIEF          = "Production Ready Macros for SAS Application Developers"
 RECURSIVE              = YES
 REPEAT_BRIEF           = NO
 SHOW_NAMESPACES        = NO

sasjs/doxy/DoxygenLayout.xml

@@ -2,7 +2,7 @@
   <!-- Generated by doxygen 1.8.14 -->
   <!-- Navigation index tabs for HTML output -->
   <navindex>
-    <tab type="mainpage" visible="no" title=""/>
+    <tab type="mainpage" visible="yes" title="Home"/>
     <tab type="pages" visible="no" title="" intro=""/>
     <tab type="modules" visible="no" title="" intro=""/>
     <tab type="namespaces" visible="no" title="">
@@ -16,7 +16,7 @@
       <tab type="classmembers" visible="no" title="" intro=""/>
     </tab>
 
-    <tab type="filelist" visible="yes" title="" intro="List of Files Used in the Macro Core Library"/>
+    <tab type="filelist" visible="yes" title="" intro="List of Files Used in the Macro-Core Library"/>
 
     <tab type="examples" visible="yes" title="" intro=""/>
   </navindex>
@@ -101,11 +101,11 @@
   <!-- Layout definition for a directory page -->
   <directory>
     <briefdescription visible="yes"/>
+    <detaileddescription visible="yes" title=""/>
     <directorygraph visible="yes"/>
     <memberdecl>
       <dirs visible="yes"/>
       <files visible="yes"/>
     </memberdecl>
-    <detaileddescription title=""/>
   </directory>
-</doxygenlayout>
+</doxygenlayout>

sasjs/doxy/new_footer.html

@@ -0,0 +1,32 @@
+<!-- HTML footer for doxygen 1.8.17-->
+<!-- start footer part -->
+<!--BEGIN GENERATE_TREEVIEW-->
+<div id="nav-path" class="navpath">
+  <!-- id is needed for treeview function! -->
+  <ul>
+    $navpath
+    <li class="footer">
+      $generatedby
+      <a href="https://www.doxygen.org/index.html">
+        <img class="footer" src="$relpath^doxygen.png" alt="doxygen"
+      /></a>
+      $doxygenversion
+    </li>
+    <li>
+      <i> For more information visit the </i>
+      <a href="https://github.com/sasjs/core">Macro Core library</a>.
+    </li>
+  </ul>
+</div>
+<!--END GENERATE_TREEVIEW-->
+<!--BEGIN !GENERATE_TREEVIEW-->
+<hr class="footer" />
+<address class="footer">
+  <small>
+    $generatedby &#160;<a href="http://www.doxygen.org/index.html">
+      <img class="footer" src="$relpath^doxygen.png" alt="doxygen" />
+    </a>
+    $doxygenversion
+  </small>
+</address>
+<!--END !GENERATE_TREEVIEW-->

sasjs/doxy/new_header.html

@@ -0,0 +1,95 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<!-- HTML header for doxygen 1.8.17-->
+<html xmlns="https://www.w3.org/1999/xhtml" lang="en">
+  <head>
+    <meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8" />
+    <meta http-equiv="X-UA-Compatible" content="IE=9" />
+    <meta property="og:type" content="website">
+    <meta name="author" content="Allan Bowe">
+    <meta name="generator" content="Doxygen $doxygenversion" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <!--BEGIN PROJECT_NAME-->
+    <title>$projectname: $title</title>
+    <!--END PROJECT_NAME-->
+    <!--BEGIN !PROJECT_NAME-->
+    <title>$title</title>
+    <!--END !PROJECT_NAME-->
+    <link href="$relpath^tabs.css" rel="stylesheet" type="text/css" />
+    <script type="text/javascript" src="$relpath^jquery.js"></script>
+    <script type="text/javascript" src="$relpath^dynsections.js"></script>
+    $treeview $search $mathjax
+    <link href="$relpath^$stylesheet" rel="stylesheet" type="text/css" />
+    <link rel="icon" href="https://sasjs.io/img/runningman.jpg" />
+    $extrastylesheet
+  </head>
+  <body>
+    <div id="top">
+      <!-- do not remove this div, it is closed by doxygen! -->
+
+      <!--BEGIN TITLEAREA-->
+      <div id="titlearea">
+        <table cellspacing="0" cellpadding="0">
+          <tbody>
+            <tr style="height: 56px">
+              <!--BEGIN PROJECT_LOGO-->
+              <td id="projectlogo">
+                <img alt="Logo" src="$relpath^$projectlogo" />
+              </td>
+              <!--END PROJECT_LOGO-->
+              <!--BEGIN PROJECT_NAME-->
+              <td id="projectalign" style="padding-left: 0.5em">
+                <div id="projectname">
+                  <!--BEGIN PROJECT_NUMBER-->&#160;<span id="projectnumber"
+                    >$projectnumber</span
+                  ><!--END PROJECT_NUMBER-->
+                </div>
+                <!--BEGIN PROJECT_BRIEF-->
+                <div id="projectbrief">
+                  Production Ready Macros for SAS Application Developers<br />
+                  <a href="https://github.com/sasjs/core">
+                    https://github.com/sasjs/core
+                  </a>
+                </div>
+                <meta name="Description" content="$projectbrief" />
+                <!--END PROJECT_BRIEF-->
+              </td>
+              <!--END PROJECT_NAME-->
+              <!--BEGIN !PROJECT_NAME-->
+              <!--BEGIN PROJECT_BRIEF-->
+              <td style="padding-left: 0.5em">
+                <div id="projectbrief">$projectbrief</div>
+                <table
+                  style="padding-left: 2em"
+                  cellspacing="0"
+                  cellpadding="0"
+                >
+                  <tr>
+                    <td>
+                      Production Ready Macros for SAS Application Developers
+                    </td>
+                  </tr>
+                  <tr>
+                    <td>
+                      <a href="https://github.com/sasjs/core">
+                        https://github.com/sasjs/core
+                      </a>
+                    </td>
+                  </tr>
+                </table>
+              </td>
+              <!--END PROJECT_BRIEF-->
+              <!--END !PROJECT_NAME-->
+              <!--BEGIN DISABLE_INDEX-->
+              <!--BEGIN SEARCHENGINE-->
+              <td>$searchbox</td>
+              <!--END SEARCHENGINE-->
+              <!--END DISABLE_INDEX-->
+            </tr>
+          </tbody>
+        </table>
+      </div>
+      <!--END TITLEAREA-->
+      <!-- end header part -->
+    </div>
+  </body>
+</html>

sasjs/doxy/new_stylesheet.css

@@ -0,0 +1,5 @@
+#projectlogo img
+{
+	border: 0px none;
+	max-height:70px
+}

sasjs/sasjsconfig.json

@@ -0,0 +1,7 @@
+{
+  "$schema": "https://cli.sasjs.io/sasjsconfig-schema.json",
+  "macroFolders": ["base", "meta", "metax", "viya", "lua"],
+  "docConfig":{
+    "displayMacroCore": false
+  }
+}

sasjs/utils/build.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+####################################################################
+# PROJECT: Macro Core Docs Build                                   #
+# To execute, use the npm command (npm run docs)                   #
+####################################################################
+
+# refresh github pages site
+rm -rf sasjsbuild/docsite
+git clone git@github.com:sasjs/core.github.io.git sasjsbuild/docsite
+rm -rf sasjsbuild/docsite/*
+mv sasjsbuild/docs/* sasjsbuild/docsite/
+cd sasjsbuild/docsite/
+echo 'core.sasjs.io' > CNAME
+git add .
+git commit -m "build.sh build on $(date +%F:%H:%M:%S)"
+git push
+npx sitemap-generator-cli https://core.sasjs.io
+git add .
+git commit -m "adding sitemap"
+git push
+
+echo "check it out:  https://sasjs.github.io/core.github.io/files.html"

viya/mv_createfolder.sas

@@ -4,7 +4,6 @@
   @details Expects oauth token in a global macro variable (default
   ACCESS_TOKEN).
 
-      options mprint;
       %mv_createfolder(path=/Public)
 
 
@@ -15,10 +14,9 @@
 
 
   @version VIYA V.03.04
-  @author Allan Bowe
-  @source https://github.com/sasjs/core
+  @author Allan Bowe, source: https://github.com/sasjs/core
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
   @li mf_getuniquefileref.sas
   @li mf_getuniquelibref.sas
@@ -42,7 +40,7 @@
 %end;
 
 %put &sysmacroname: grant_type=&grant_type;
-%mp_abort(iftrue=(&grant_type ne authorization_code and &grant_type ne password 
+%mp_abort(iftrue=(&grant_type ne authorization_code and &grant_type ne password
     and &grant_type ne sas_services
   )
   ,mac=&sysmacroname
@@ -121,7 +119,7 @@ options noquotelenmax;
         out=&fname2
         &oauth_bearer
         url=%unquote(%superq(href));
-        headers 
+        headers
       %if &grant_type=authorization_code %then %do;
                 "Authorization"="Bearer &&&access_token_var"
       %end;

viya/mv_createjob.sas

@@ -0,0 +1,307 @@
+/**
+  @file
+  @brief Creates a Viya Job
+  @details
+  Code is passed in as one or more filerefs.
+
+      %* Step 1 - compile macros ;
+      filename mc url "https://raw.githubusercontent.com/sasjs/core/main/all.sas";
+      %inc mc;
+
+      %* Step 2 - Create some SAS code and add it to a job;
+      filename ft15f001 temp;
+      parmcards4;
+          data some_code;
+            set sashelp.class;
+          run;
+      ;;;;
+      %mv_createjob(path=/Public/app/sasjstemp/jobs/myjobs,name=myjob)
+
+  The path to the job will then be shown in the log, eg as follows:
+
+  ![viya job location](https://i.imgur.com/XRUDHgA.png)
+
+
+  <h4> SAS Macros </h4>
+  @li mp_abort.sas
+  @li mv_createfolder.sas
+  @li mf_getuniquelibref.sas
+  @li mf_getuniquefileref.sas
+  @li mf_getplatform.sas
+  @li mf_isblank.sas
+  @li mv_deletejes.sas
+
+  @param path= The full path (on SAS Drive) where the job will be created
+  @param name= The name of the job
+  @param desc= The description of the job
+  @param precode= Space separated list of filerefs, pointing to the code that
+    needs to be attached to the beginning of the job
+  @param code= Fileref(s) of the actual code to be added
+  @param access_token_var= The global macro variable to contain the access token
+  @param grant_type= valid values are "password" or "authorization_code" (unquoted).
+    The default is authorization_code.
+  @param replace= select NO to avoid replacing any existing job in that location
+  @param contextname= Choose a specific context on which to run the Job.  Leave
+    blank to use the default context.  From Viya 3.5 it is possible to configure
+    a shared context - see
+    https://go.documentation.sas.com/?docsetId=calcontexts&docsetTarget=n1hjn8eobk5pyhn1wg3ja0drdl6h.htm&docsetVersion=3.5&locale=en
+
+  @version VIYA V.03.04
+  @author [Allan Bowe](https://www.linkedin.com/in/allanbowe)
+
+**/
+
+%macro mv_createjob(path=
+    ,name=
+    ,desc=Created by the mv_createjob.sas macro
+    ,precode=
+    ,code=ft15f001
+    ,access_token_var=ACCESS_TOKEN
+    ,grant_type=sas_services
+    ,replace=YES
+    ,debug=0
+    ,contextname=
+  );
+%local oauth_bearer;
+%if &grant_type=detect %then %do;
+  %if %symexist(&access_token_var) %then %let grant_type=authorization_code;
+  %else %let grant_type=sas_services;
+%end;
+%if &grant_type=sas_services %then %do;
+    %let oauth_bearer=oauth_bearer=sas_services;
+    %let &access_token_var=;
+%end;
+%put &sysmacroname: grant_type=&grant_type;
+
+/* initial validation checking */
+%mp_abort(iftrue=(&grant_type ne authorization_code and &grant_type ne password
+    and &grant_type ne sas_services
+  )
+  ,mac=&sysmacroname
+  ,msg=%str(Invalid value for grant_type: &grant_type)
+)
+%mp_abort(iftrue=(%mf_isblank(&path)=1)
+  ,mac=&sysmacroname
+  ,msg=%str(path value must be provided)
+)
+%mp_abort(iftrue=(%length(&path)=1)
+  ,mac=&sysmacroname
+  ,msg=%str(path value must be provided)
+)
+%mp_abort(iftrue=(%mf_isblank(&name)=1)
+  ,mac=&sysmacroname
+  ,msg=%str(name value must be provided)
+)
+
+options noquotelenmax;
+
+* remove any trailing slash ;
+%if "%substr(&path,%length(&path),1)" = "/" %then
+  %let path=%substr(&path,1,%length(&path)-1);
+
+/* ensure folder exists */
+%put &sysmacroname: Path &path being checked / created;
+%mv_createfolder(path=&path)
+
+%local base_uri; /* location of rest apis */
+%let base_uri=%mf_getplatform(VIYARESTAPI);
+
+/* fetching folder details for provided path */
+%local fname1;
+%let fname1=%mf_getuniquefileref();
+proc http method='GET' out=&fname1 &oauth_bearer
+  url="&base_uri/folders/folders/@item?path=&path";
+%if &grant_type=authorization_code %then %do;
+  headers "Authorization"="Bearer &&&access_token_var";
+%end;
+run;
+%if &debug %then %do;
+  data _null_;
+    infile &fname1;
+    input;
+    putlog _infile_;
+  run;
+%end;
+%mp_abort(iftrue=(&SYS_PROCHTTP_STATUS_CODE ne 200)
+  ,mac=&sysmacroname
+  ,msg=%str(&SYS_PROCHTTP_STATUS_CODE &SYS_PROCHTTP_STATUS_PHRASE)
+)
+
+/* path exists. Grab follow on link to check members */
+%local libref1;
+%let libref1=%mf_getuniquelibref();
+libname &libref1 JSON fileref=&fname1;
+
+data _null_;
+  set &libref1..links;
+  if rel='members' then call symputx('membercheck',quote("&base_uri"!!trim(href)),'l');
+  else if rel='self' then call symputx('parentFolderUri',href,'l');
+run;
+data _null_;
+  set &libref1..root;
+  call symputx('folderid',id,'l');
+run;
+%local fname2;
+%let fname2=%mf_getuniquefileref();
+proc http method='GET'
+    out=&fname2
+    &oauth_bearer
+    url=%unquote(%superq(membercheck));
+    headers
+  %if &grant_type=authorization_code %then %do;
+            "Authorization"="Bearer &&&access_token_var"
+  %end;
+            'Accept'='application/vnd.sas.collection+json'
+            'Accept-Language'='string';
+%if &debug=1 %then %do;
+   debug level = 3;
+%end;
+run;
+/*data _null_;infile &fname2;input;putlog _infile_;run;*/
+%mp_abort(iftrue=(&SYS_PROCHTTP_STATUS_CODE ne 200)
+  ,mac=&sysmacroname
+  ,msg=%str(&SYS_PROCHTTP_STATUS_CODE &SYS_PROCHTTP_STATUS_PHRASE)
+)
+
+%if %upcase(&replace)=YES %then %do;
+  %mv_deletejes(path=&path, name=&name)
+%end;
+%else %do;
+  /* check that job does not already exist in that folder */
+  %local libref2;
+  %let libref2=%mf_getuniquelibref();
+  libname &libref2 JSON fileref=&fname2;
+  %local exists; %let exists=0;
+  data _null_;
+    set &libref2..items;
+    if contenttype='jobDefinition' and upcase(name)="%upcase(&name)" then
+      call symputx('exists',1,'l');
+  run;
+  %mp_abort(iftrue=(&exists=1)
+    ,mac=&sysmacroname
+    ,msg=%str(Job &name already exists in &path)
+  )
+  libname &libref2 clear;
+%end;
+
+/* set up the body of the request to create the service */
+%local fname3;
+%let fname3=%mf_getuniquefileref();
+data _null_;
+  file &fname3 TERMSTR=' ';
+  length string $32767;
+  string=cats('{"version": 0,"name":"'
+  	,"&name"
+  	,'","type":"Compute","parameters":[{"name":"_addjesbeginendmacros"'
+    ,',"type":"CHARACTER","defaultValue":"false"}');
+  context=quote(cats(symget('contextname')));
+  if context ne '""' then do;
+    string=cats(string,',{"version": 1,"name": "_contextName","defaultValue":'
+     ,context,',"type":"CHARACTER","label":"Context Name","required": false}');
+  end;
+  string=cats(string,'],"code":"');
+  put string;
+run;
+
+
+/* insert the code, escaping double quotes and carriage returns */
+%local x fref freflist;
+%let freflist= &precode &code ;
+%do x=1 %to %sysfunc(countw(&freflist));
+  %let fref=%scan(&freflist,&x);
+  %put &sysmacroname: adding &fref;
+  data _null_;
+    length filein 8 fileid 8;
+    filein = fopen("&fref","I",1,"B");
+    fileid = fopen("&fname3","A",1,"B");
+    rec = "20"x;
+    do while(fread(filein)=0);
+      rc = fget(filein,rec,1);
+      if rec='"' then do;  /* DOUBLE QUOTE */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'"');rc =fwrite(fileid);
+      end;
+      else if rec='0A'x then do; /* LF */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'n');rc =fwrite(fileid);
+      end;
+      else if rec='0D'x then do; /* CR */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'r');rc =fwrite(fileid);
+      end;
+      else if rec='09'x then do; /* TAB */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'t');rc =fwrite(fileid);
+      end;
+      else if rec='5C'x then do; /* BACKSLASH */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+      end;
+      else do;
+        rc =fput(fileid,rec);
+        rc =fwrite(fileid);
+      end;
+    end;
+    rc=fclose(filein);
+    rc=fclose(fileid);
+  run;
+%end;
+
+/* finish off the body of the code file loaded to JES */
+data _null_;
+  file &fname3 mod TERMSTR=' ';
+  put '"}';
+run;
+
+/* now we can create the job!! */
+%local fname4;
+%let fname4=%mf_getuniquefileref();
+proc http method='POST'
+    in=&fname3
+    out=&fname4
+    &oauth_bearer
+    url="&base_uri/jobDefinitions/definitions?parentFolderUri=&parentFolderUri";
+    headers 'Content-Type'='application/vnd.sas.job.definition+json'
+  %if &grant_type=authorization_code %then %do;
+            "Authorization"="Bearer &&&access_token_var"
+  %end;
+            "Accept"="application/vnd.sas.job.definition+json";
+%if &debug=1 %then %do;
+   debug level = 3;
+%end;
+run;
+/*data _null_;infile &fname4;input;putlog _infile_;run;*/
+%mp_abort(iftrue=(&SYS_PROCHTTP_STATUS_CODE ne 201)
+  ,mac=&sysmacroname
+  ,msg=%str(&SYS_PROCHTTP_STATUS_CODE &SYS_PROCHTTP_STATUS_PHRASE)
+)
+/* clear refs */
+filename &fname1 clear;
+filename &fname2 clear;
+filename &fname3 clear;
+filename &fname4 clear;
+libname &libref1 clear;
+
+/* get the url so we can give a helpful log message */
+%local url;
+data _null_;
+  if symexist('_baseurl') then do;
+    url=symget('_baseurl');
+    if subpad(url,length(url)-9,9)='SASStudio'
+      then url=substr(url,1,length(url)-11);
+    else url="&systcpiphostname";
+  end;
+  else url="&systcpiphostname";
+  call symputx('url',url);
+run;
+
+
+%put &sysmacroname: Job &name successfully created in &path;
+%put &sysmacroname:;
+%put &sysmacroname: Check it out here:;
+%put &sysmacroname:;%put;
+%put    &url/SASJobExecution?_PROGRAM=&path/&name;%put;
+%put &sysmacroname:;
+%put &sysmacroname:;
+
+%mend;

viya/mv_createwebservice.sas

@@ -1,34 +1,35 @@
 /**
-  @file mv_createwebservice.sas
+  @file
   @brief Creates a JobExecution web service if it doesn't already exist
-  @details  Code is passed in as one or more filerefs.
+  @details
+  Code is passed in as one or more filerefs.
 
-    %* Step 1 - compile macros ;
-    filename mc url "https://raw.githubusercontent.com/sasjs/core/main/all.sas";
-    %inc mc;
+      %* Step 1 - compile macros ;
+      filename mc url "https://raw.githubusercontent.com/sasjs/core/main/all.sas";
+      %inc mc;
 
-    %* Step 2 - Create some code and add it to a web service;
-    filename ft15f001 temp;
-    parmcards4;
-        %webout(FETCH) %* fetch any tables sent from frontend;
-        %* do some sas, any inputs are now already WORK tables;
-        data example1 example2;
-          set sashelp.class;
-        run;
-        %* send data back;
-        %webout(OPEN)
-        %webout(ARR,example1) * Array format, fast, suitable for large tables ;
-        %webout(OBJ,example2) * Object format, easier to work with ;
-        %webout(CLOSE)
-    ;;;;
-    %mv_createwebservice(path=/Public/app/common,name=appinit)
+      %* Step 2 - Create some code and add it to a web service;
+      filename ft15f001 temp;
+      parmcards4;
+          %webout(FETCH) %* fetch any tables sent from frontend;
+          %* do some sas, any inputs are now already WORK tables;
+          data example1 example2;
+            set sashelp.class;
+          run;
+          %* send data back;
+          %webout(OPEN)
+          %webout(ARR,example1) * Array format, fast, suitable for large tables ;
+          %webout(OBJ,example2) * Object format, easier to work with ;
+          %webout(CLOSE)
+      ;;;;
+      %mv_createwebservice(path=/Public/app/common,name=appinit)
 
 
   Notes:
     To minimise postgres requests, output json is stored in a temporary file
     and then sent to _webout in one go at the end.
 
-  <h4> Dependencies </h4>
+  <h4> SAS Macros </h4>
   @li mp_abort.sas
   @li mv_createfolder.sas
   @li mf_getuniquelibref.sas
@@ -54,8 +55,7 @@
     a shared context - see https://go.documentation.sas.com/?docsetId=calcontexts&docsetTarget=n1hjn8eobk5pyhn1wg3ja0drdl6h.htm&docsetVersion=3.5&locale=en
 
   @version VIYA V.03.04
-  @author Allan Bowe
-  @source https://github.com/sasjs/core
+  @author Allan Bowe, source: https://github.com/sasjs/core
 
 **/
 
@@ -566,23 +566,23 @@ run;
     rec = "20"x;
     do while(fread(filein)=0);
       rc = fget(filein,rec,1);
-      if rec='"' then do;
+      if rec='"' then do;  /* DOUBLE QUOTE */
         rc =fput(fileid,'\');rc =fwrite(fileid);
         rc =fput(fileid,'"');rc =fwrite(fileid);
       end;
-      else if rec='0A'x then do;
-        rc =fput(fileid,'\');rc =fwrite(fileid);
-        rc =fput(fileid,'r');rc =fwrite(fileid);
-      end;
-      else if rec='0D'x then do;
+      else if rec='0A'x then do; /* LF */
         rc =fput(fileid,'\');rc =fwrite(fileid);
         rc =fput(fileid,'n');rc =fwrite(fileid);
       end;
-      else if rec='09'x then do;
+      else if rec='0D'x then do; /* CR */
+        rc =fput(fileid,'\');rc =fwrite(fileid);
+        rc =fput(fileid,'r');rc =fwrite(fileid);
+      end;
+      else if rec='09'x then do; /* TAB */
         rc =fput(fileid,'\');rc =fwrite(fileid);
         rc =fput(fileid,'t');rc =fwrite(fileid);
       end;
-      else if rec='5C'x then do;
+      else if rec='5C'x then do; /* BACKSLASH */
         rc =fput(fileid,'\');rc =fwrite(fileid);
         rc =fput(fileid,'\');rc =fwrite(fileid);
       end;