diff --git a/README.md b/README.md index 0d3e84c..5058ef5 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,7 @@ SHA256 digest for macroArray: 53C248E1DE3268946C9CEC7E77BC222F652FBB006D9317BE36 [Documentation for macroArray](https://github.com/yabwon/SAS_PACKAGES/blob/master/packages/macroarray.md "Documentation for macroArray") -- **BasePlus**\[0.62\] adds a bunch of functionalities I am missing in BASE SAS, such as: +- **BasePlus**\[0.7\] adds a bunch of functionalities I am missing in BASE SAS, such as: ``` call arrMissToRight(myArray); call arrFillMiss(17, myArray); @@ -118,8 +118,9 @@ format x bool.; %put %getVars(sashelp.class, pattern = ght$, sep = +, varRange = _numeric_); ``` -SHA256 digest for BasePlus: 278621A6D8BBBB791DEA4C215D4261F2CB8F8B76B1397F7FA9B2E4219E77CB5A +SHA256 digest for BasePlus: 66E966489F4C183CA75FC32D3AF581FEC20FC9C5FF0C36E4DDC5A14BBDA82EAB +[Documentation for BasePlus](https://github.com/yabwon/SAS_PACKAGES/blob/master/packages/baseplus.md "Documentation for BasePlus") - **dynMacroArray**\[0.2\], set of macros (wrappers for a hash table) emulating dynamic array in the data step (macro predecessor of DFA) diff --git a/packages/README.md b/packages/README.md index 5ace517..48f6b7b 100644 --- a/packages/README.md +++ b/packages/README.md @@ -3,6 +3,7 @@ To get started with SAS Packages try this [**`Getting Started with SAS Packages` ## Available packages: Currently the following packages are available: +--- - **SQLinDS**\[2.2\], based on Mike Rhoads' article *Use the Full Power of SAS in Your Function-Style Macros*. The package allows to write SQL queries in the data step, e.g. ``` @@ -19,7 +20,7 @@ run; SHA256 digest for SQLinDS: B280D0B72DB77001ADAAE9C1612B67AD30C2C672371B27F1ACB12016C7A1363D [Documentation for SQLinDS](https://github.com/yabwon/SAS_PACKAGES/blob/master/packages/sqlinds.md "Documentation for SQLinDS") - +--- - **DFA** (Dynamic Function Arrays)\[0.2\], contains set of macros and FCMP functions which implement: a dynamically allocated array, a stack, a fifo queue, an ordered stack, and a priority queue, run `%helpPackage(DFA,createDFArray)` to find examples. ``` @@ -49,7 +50,7 @@ data _null_; run; ``` SHA256 digest for DFA: BB8768E977D62429368CFF2E5338A6553C35C998AEC09AF24088BA713BB54DDA - +--- - **macroArray**\[0.5\], implementation of an array concept in a macro language, e.g. ``` @@ -73,8 +74,9 @@ SHA256 digest for DFA: BB8768E977D62429368CFF2E5338A6553C35C998AEC09AF24088BA713 SHA256 digest for macroArray: 53C248E1DE3268946C9CEC7E77BC222F652FBB006D9317BE36B86410DA31AE35 [Documentation for macroArray](https://github.com/yabwon/SAS_PACKAGES/blob/master/packages/macroarray.md "Documentation for macroArray") +--- -- **BasePlus**\[0.62\] adds a bunch of functionalities I am missing in BASE SAS, such as: +- **BasePlus**\[0.7\] adds a bunch of functionalities I am missing in BASE SAS, such as: ``` call arrMissToRight(myArray); call arrFillMiss(17, myArray); @@ -88,9 +90,11 @@ format x bool.; %put %getVars(sashelp.class, pattern = ght$, sep = +, varRange = _numeric_); ``` -SHA256 digest for BasePlus: 278621A6D8BBBB791DEA4C215D4261F2CB8F8B76B1397F7FA9B2E4219E77CB5A +SHA256 digest for BasePlus: 66E966489F4C183CA75FC32D3AF581FEC20FC9C5FF0C36E4DDC5A14BBDA82EAB +[Documentation for BasePlus](https://github.com/yabwon/SAS_PACKAGES/blob/master/packages/baseplus.md "Documentation for BasePlus") +--- - **dynMacroArray**\[0.2\], set of macros (wrappers for a hash table) emulating dynamic array in the data step (macro predecessor of DFA) SHA256 digest for dynMacroArray: 066186B94B2976167C797C6A6E6217E361E8DEB10F2AB81906E0A325E5243084 - +--- diff --git a/packages/SHA256_for_packages.txt b/packages/SHA256_for_packages.txt index c74278c..2ec79a7 100644 --- a/packages/SHA256_for_packages.txt +++ b/packages/SHA256_for_packages.txt @@ -1,3 +1,6 @@ +/* 20201003 */ +BasePlus: 66E966489F4C183CA75FC32D3AF581FEC20FC9C5FF0C36E4DDC5A14BBDA82EAB + /* 20200914 */ macroArray: 53C248E1DE3268946C9CEC7E77BC222F652FBB006D9317BE36B86410DA31AE35 SQLinDS: B280D0B72DB77001ADAAE9C1612B67AD30C2C672371B27F1ACB12016C7A1363D diff --git a/packages/baseplus.md b/packages/baseplus.md new file mode 100644 index 0000000..fe94646 --- /dev/null +++ b/packages/baseplus.md @@ -0,0 +1,2156 @@ +- [The BasePlus package [ver. 0.7]](#baseplus-package) +- [Content description](#content-description) + * [`%getVars()` macro](#getvars-macro) + * [`%QgetVars()` macro](#qgetvars-macro) + * [`%symdelGlobal()` macro](#symdelglobal-macro) + * [`bool.` format](#bool-format) + * [`boolz.` format](#boolz-format) + * [`ceil.` format](#ceil-format) + * [`floor.` format](#floor-format) + * [`int.` format](#int-format) + * [`arrFill()` subroutine](#arrfill-subroutine) + * [`arrFillC()` subroutine](#arrfillc-subroutine) + * [`arrMissFill()` subroutine](#arrmissfill-subroutine) + * [`arrMissFillC()` subroutine](#arrmissfillc-subroutine) + * [`arrMissToLeft()` subroutine](#arrmisstoleft-subroutine) + * [`arrMissToLeftC()` subroutine](#arrmisstoleftc-subroutine) + * [`arrMissToRight()` subroutine](#arrmisstoright-subroutine) + * [`arrMissToRightC()` subroutine](#arrmisstorightc-subroutine) + * [`catXFc()` function](#catxfc-function) + * [`catXFi()` function](#catxfi-function) + * [`catXFj()` function](#catxfj-function) + * [`catXFn()` function](#catxfn-function) + * [`delDataset()` function](#deldataset-function) + * [`qsortInCbyProcProto()` proto function](#qsortincbyprocproto-proto-function) + * [`fromMissingToNumberBS()` function](#frommissingtonumberbs-function) + * [`fromNumberToMissing()` function](#fromnumbertomissing-function) + * [`quickSort4NotMiss()` subroutine](#quicksort4notmiss-subroutine) + * [`quickSortHash()` subroutine](#quicksorthash-subroutine) + * [`quickSortHashSDDV()` subroutine](#quicksorthashsddv-subroutine) + * [`quickSortLight()` subroutine](#quicksortlight-subroutine) + * [License](#license) + +--- + +# The BasePlus package [ver. 0.7] ############################################### + +The **BasePlus** package implements useful +functions and functionalities I miss in the BASE SAS. + +It is inspired by various people, e.g. +- at the SAS-L discussion list +- at the communities.sas.com (SASware Ballot Ideas) +- etc. + +Kudos to all who inspired me to generate this package: +*Mark Keintz*, +*Paul Dorfman*, +*Richard DeVenezia*, +*Christian Graffeuille*. + +--- + +### BASIC EXAMPLES AND USECASES: #################################################### + +**Example 1**: One-dimensional array functions. + Array parameters to subroutine + calls must be 1-based. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data _null_; + array X[4] _temporary_ (. 1 . 2); + + call arrMissToRight(X); + do i = 1 to 4; + put X[i]= @; + end; + put; + + call arrFillMiss(17, X); + do i = 1 to 4; + put X[i]= @; + end; + put; + + call arrFill(42, X); + do i = 1 to 4; + put X[i]= @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2**: Delete dataset by name. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data toDrop; + x = 17; + run; + data _null_; + p = delDataset("toDrop"); + put p=; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 3**: Strings concatenation with format. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data test; + x = 1 ; y = . ; z = 3 ; + t = "t"; u = " "; v = "v"; + + array a[*] x y z; + array b[*] t u v; + + length s1 s2 s3 s4 $ 17; + s1 = catXFn("z5.", "#", A); + s2 = catXFi("z5.", "#", A); + s3 = catXFc("upcase.", "*", B); + s4 = catXFj("upcase.", "*", B); + + put (_all_) (=); + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Example 4**: Useful formats. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data _null_; + input x @@; + put @1 x= @11 x= bool. @21 x= int. @31 x= ceil. @41 x= floor.; + cards; + . ._ .A -10 -3.14 0 3.14 10 + ; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Example 5**: Getting variables names from datasets. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class + ,pattern = ght$ + ,sep = + + ,varRange = _numeric_)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Example 6**: Quick sort as an alternative to call sortn() +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data _null_; + array test[25000000] _temporary_ ; + + t = time(); + call streaminit(123); + do _N_ = 25000000 to 1 by -1; + test[_N_] = rand("uniform"); + end; + t = time() - t; + put "Array population time: " t; + + t = time(); + call quickSortLight (test); + t = time()-t; + put "Sorting time: " / t=; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +### Content ################################################################### + +--- + +Package contains: +1. macro getvars +2. macro qgetvars +3. macro symdelglobal +4. format bool +5. format boolz +6. format ceil +7. format floor +8. format int +9. function arrfill +10. function arrfillc +11. function arrmissfill +12. function arrmissfillc +13. function arrmisstoleft +14. function arrmisstoleftc +15. function arrmisstoright +16. function arrmisstorightc +17. function catxfc +18. function catxfi +19. function catxfj +20. function catxfn +21. function deldataset +22. proto qsortincbyprocproto +23. function frommissingtonumberbs +24. function fromnumbertomissing +25. function quicksort4notmiss +26. function quicksorthash +27. function quicksorthashsddv +28. function quicksortlight + +*SAS package generated by generatePackage, version 20201001* + +The SHA256 hash digest for package BasePlus: +`66E966489F4C183CA75FC32D3AF581FEC20FC9C5FF0C36E4DDC5A14BBDA82EAB` + +--- +# Content description ############################################################################################ + +## >>> `%getVars()` macro: <<< ####################### + +The getVars() and QgetVars() macro functions +allow to extract variables names form a dataset +according to a given pattern into a list. + +The getVars() returns unquoted value [by %unquote()]. +The QgetVars() returns quoted value [by %superq()]. + +See examples below for the details. + +The `%getVars()` macro executes like a pure macro code. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +%getVars( + ds + <,sep=> + <,pattern=> + <,varRange=> + <,quote=> +) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `ds` - *Required*, the name of the dataset from + which variables are to be taken. + +* `sep = %str( )` - *Optional*, default value `%str( )`, + a variables separator on the created list. + +* `pattern = .*` - *Optional*, default value `.*` (i.e. any text), + a variable name regexp pattern, case INSENSITIVE! + +* `varRange = _all_` - *Optional*, default value `_all_`, + a named range list of variables. + +* `quote =` - *Optional*, default value is blank, a quotation + symbol to be used around values. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** A list of all variables from the + sashelp.class dataset: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 2.** A list of all variables from the + sashelp.class dataset separated + by backslash: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let x = %getVars(sashelp.class, sep=\); + %put &=x; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 3.** Use of regular expressions: + a) A list of variables which name contains "i" or "a" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class, pattern=i|a)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + b) A list of variables which name starts with "w" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class, pattern=^w)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + c) A list of variables which name ends with "ght" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class, pattern=ght$)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 4.** A list of numeric variables which name + starts with "w" or "h" or ends with "x" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class, sep=+, pattern=^(w|h)|x$, varRange=_numeric_)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 5.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data test; + array x[30]; + array y[30] $ ; + array z[30]; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + a) A list of variables separated by a comma: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(test, sep=%str(,))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + b) A list of variables separated by a comma + with suffix 5 or 7: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(test, sep=%str(,), pattern=(5|7)$)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + c) A list of variables separated by a comma + with suffix 5 or 7 from a given variables range: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(test, sep=%str(,), varRange=x10-numeric-z22 y6-y26, pattern=(5|7)$)*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 6.** Case of quotes and special characters + when the quote= parameter is _not_ used: + + a) one single or double qiote: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%bquote(%getVars(sashelp.class, sep=%str(%")))*; + %put *%bquote(%getVars(sashelp.class, sep=%str(%')))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + b) two single or double qiotes: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *"%bquote(%getVars(sashelp.class,sep=""))"*; + %put *%str(%')%bquote(%getVars(sashelp.class,sep=''))%str(%')*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + c) coma separated double quote list: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *"%getVars(sashelp.class,sep=%str(", "))"*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + d) coma separated single quote list: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%str(%')%getVars(sashelp.class,sep=', ')%str(%')*; + %let x = %str(%')%getVars(sashelp.class,sep=', ')%str(%'); + + %put *%str(%')%QgetVars(sashelp.class,sep=', ')%str(%')*; + %let y = %str(%')%QgetVars(sashelp.class,sep=', ')%str(%'); + %let z = %unquote(&y.); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + e) ampersand (&) as a separator [compare behaviour]: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class,sep=&)*; + %let x = %getVars(sashelp.class,sep=&); + + %put *%getVars(sashelp.class,sep=%str( & ))*; + %let x = %getVars(sashelp.class,sep=%str( & )); + + %put *%QgetVars(sashelp.class,sep=&)*; + %let y = %QgetVars(sashelp.class,sep=&); + %let z = %unquote(&y.); + + %put *%QgetVars(sashelp.class,sep=%str( & ))*; + %let y = %QgetVars(sashelp.class,sep=%str( & )); + %let z = %unquote(&y.); + + %put *%getVars(sashelp.class,sep=&)*; + %let x = %getVars(sashelp.class,sep=&); + + %put *%getVars(sashelp.class,sep=%str( & ))*; + %let x = %getVars(sashelp.class,sep=%str( & )); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + f) percent (%) as a separator [compare behaviour]: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%QgetVars(sashelp.class,sep=%)*; + %let y = %QgetVars(sashelp.class,sep=%); + %let z = %unquote(&y.); + + %put *%QgetVars(sashelp.class,sep=%str( % ))*; + %let y = %QgetVars(sashelp.class,sep=%str( % )); + %let z = %unquote(&y.); +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 7.** Case of quotes and special characters + when the quote= parameter is used: + +a) one single or double qiote: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class, quote=%str(%"))*; + %put *%getVars(sashelp.class, quote=%str(%'))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + b) two single or double quotes: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %* this gives an error: ; + %* %put *%getVars(sashelp.class,quote="")*; + %* %put *%getVars(sashelp.class,quote='')*; + + %* this does not give an error: ; + %put *%QgetVars(sashelp.class,quote="")*; + %put *%QgetVars(sashelp.class,quote='')*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + c) coma separated double quote list: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %put *%getVars(sashelp.class,sep=%str(,),quote=%str(%"))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + d) coma separated single quote list: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let x = %getVars(sashelp.class,sep=%str(,),quote=%str(%')); + %put &=x.; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**EXAMPLE 8.** Variables that start with `A` and do not end with `GHT`: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data class; + set sashelp.class; + Aeight = height; +run; + +%put *%getVars(class, pattern = ^A(.*)(?>> `%QgetVars()` macro: <<< ####################### + +The getVars() and QgetVars() macro functions +allow to extract variables names form a dataset +according to a given pattern into a list. + +The getVars() returns unquoted value [by %unquote()]. +The QgetVars() returns quoted value [by %superq()]. + +The `%QgetVars()` macro executes like a pure macro code. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +%QgetVars( + ds + <,sep=> + <,pattern=> + <,varRange=> + <,quote=> +) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `ds` - *Required*, the name of the dataset from + which variables are to be taken. + +* `sep = %str( )` - *Optional*, default value `%str( )`, + a variables separator on the created list. + +* `pattern = .*` - *Optional*, default value `.*` (i.e. any text), + a variable name regexp pattern, case INSENSITIVE! + +* `varRange = _all_` - *Optional*, default value `_all_`, + a named range list of variables. + +* `quote =` - *Optional*, default value is blank, a quotation + symbol to be used around values. + +### EXAMPLES AND USECASES: #################################################### + +See examples in `%getVars()` help for the details. + +--- + +## >>> `%symdelGlobal()` macro: <<< ####################### + +The `%symdelGlobal()` macro deletes all global macrovariables +created by the user. The only exceptions are read only variables +and variables the one which starts with SYS, AF, or FSP. +In that case a warning is printed in the log. + +One temporary global macrovariable `________________98_76_54_32_10_` +and a dataset, in `work` library, named `_%sysfunc(datetime(),hex7.)` +are created and deleted during the process. + +The `%symdelGlobal()` macro executes like a pure macro code. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +%symdelGlobal( + info +) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `info` - *Optional*, default value should be empty, + if set to `NOINFO` or `QUIET` then infos and + warnings about variables deletion are suspended. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** Basic use-case one. + Delete global macrovariables, info notes + and warnings are printed in the log. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let a = 1; + %let b = 2; + %let c = 3; + %let sys_my_var = 11; + %let af_my_var = 22; + %let fsp_my_var = 33; + %global / readonly read_only_x = 1234567890; + + %put _user_; + + %symdelGlobal(); + + %put _user_; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**EXAMPLE 2.** Basic use-case two. + Delete global macrovariables in quite mode + No info notes and warnings are printed in the log. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let a = 1; + %let b = 2; + %let c = 3; + %let sys_my_var = 11; + %let af_my_var = 22; + %let fsp_my_var = 33; + %global / readonly read_only_x = 1234567890; + + %put _user_; + %put *%symdelGlobal(NOINFO)*; + %put _user_; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--- + +## >>> `bool.` format: <<< ####################### + +The **bool** format returns: +*zero* for 0 or missing, +*one* for other values. + +### EXAMPLES AND USECASES: #################################################### + +It allows for a %sysevalf()'ish +conversion-type [i.e. `%sysevalf(1.7 & 4.2, boolean)`] +inside the `%sysfunc()` [e.g. `%sysfunc(aFunction(), bool.)`] + +--- + +## >>> `boolz.` format: <<< ####################### + +The **boolz** format returns: +*zero* for 0 or missing, +*one* for other values. + +*Fuzz* value is 0. + +### EXAMPLES AND USECASES: #################################################### + +It allows for a %sysevalf()'ish +conversion-type [i.e. `%sysevalf(1.7 & 4.2, boolean)`] +inside the `%sysfunc()` [e.g. `%sysfunc(aFunction(), boolz.)`] + +--- + +## >>> `ceil.` format: <<< ####################### + +The **ceil** format is a "wrapper" for the `ceil()` function. + +### EXAMPLES AND USECASES: #################################################### + +It allows for a %sysevalf()'ish +conversion-type [i.e. `%sysevalf(1.7 + 4.2, ceil)`] +inside the `%sysfunc()` [e.g. `%sysfunc(aFunction(), ceil.)`] + +--- + +## >>> `floor.` format: <<< ####################### + +The **floor** format is a "wrapper" for the `floor()` function. + +### EXAMPLES AND USECASES: #################################################### + +It allows for a %sysevalf()'ish +conversion-type [i.e. `%sysevalf(1.7 + 4.2, floor)`] +inside the `%sysfunc()` [e.g. `%sysfunc(aFunction(), floor.)`] + +--- + +## >>> `int.` format: <<< ####################### + +The **int** format is a "wrapper" for the `int()` function. + +### EXAMPLES AND USECASES: #################################################### + +It allows for a %sysevalf()'ish +conversion-type [i.e. `%sysevalf(1.7 + 4.2, integer)`] +inside the `%sysfunc()` [e.g. `%sysfunc(aFunction(), int.)`] + +--- + +## >>> `arrFill()` subroutine: <<< ####################### + +The **arrFill()** subroutine is a wrapper +for the Call Fillmatrix() [a special FCMP subroutine]. + +A numeric array is filled with selected numeric value, e.g. + +for array `A = [. . . .]` the subroutine +`call arrFill(42, A)` returns `A = [42 42 42 42]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrFill(N ,A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `N` - Numeric value. + +2. `A` - Numeric array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + array X[*] a b c; + + put "before: " (_all_) (=); + call arrFill(42, X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrFillC()` subroutine: <<< ####################### + +The **arrFillC()** subroutine fills +a character array with selected character value, e.g. + +for array `A = [" ", " ", " "]` the subroutine +`call arrFillC("B", A)` returns `A = ["B", "B", "B"]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrFillC(C ,A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `C` - Character value. + +2. `A` - Character array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + array X[*] $ a b c; + + put "before: " (_all_) (=); + call arrFillC("ABC", X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissFill()` subroutine: <<< ####################### + +The **arrMissFill()** subroutine fills +all missing values (i.e. less or equal than `.Z`) +of a numeric array with selected numeric value, e.g. + +for array `A = [1 . . 4]` the subroutine +`call arrMissFill(42, A)` returns `A = [1 42 42 4]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissFill(N ,A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `N` - Numeric value. + +2. `A` - Numeric array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + input a b c; +cards4; +1 . 3 +. 2 . +. . 3 +;;;; +run; + +data _null_; + set have ; + array X[*] a b c; + + put "before: " (_all_) (=); + call arrMissFill(42, X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissFillC()` subroutine: <<< ####################### + +The **arrMissFillC()** subroutine fills +all missing values of a character array +with selected character value, e.g. + +for array `A = ["A", " ", "C"]` the subroutine +`call arrMissFillC("B", A)` returns `A = ["A", "B", "C"]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissFillC(C, A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `C` - Character value. + +2. `A` - Character array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + infile cards dsd dlm="," missover; + input (a b c) (: $ 1.); +cards4; +A, ,C + ,B, + , ,C +;;;; +run; + +data _null_; + set have ; + array X[*] $ a b c; + + put "before: " (_all_) (=); + call arrMissFillC("X", X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissToLeft()` subroutine: <<< ####################### + +The **arrMissToLeft()** subroutine shifts +all non-missing (i.e. greater than `.Z`) +numeric elements to the right side of an array +and missing values to the left, e.g. + +for array `A = [1 . 2 . 3]` the subroutine +`call arrMissToLeft(A)` returns `A = [. . 1 2 3]` + +All missing values are replaced with the dot (`.`) + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissToLeft(A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Numeric array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + input a b c; +cards4; +1 . 3 +. 2 . +. . 3 +;;;; +run; + +data _null_; + set have ; + array X[*] a b c; + + put "before: " (_all_) (=); + call arrMissToLeft(X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissToLeftC()` subroutine: <<< ####################### + +The **arrMissToLeftC()** subroutine shifts +all non-missing (i.e. different than empty string) +character elements to the right side of an array +and all missing values to the left, e.g. + +for array `A = ["A", " ", "B", " ", "C"]` the subroutine +`call arrMissToLeftC(A)` returns `A = [" ", " ", "A", "B", "C"]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissToLeftC(A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Character array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + infile cards dsd dlm="," missover; + input (a b c) (: $ 1.); +cards4; +A, ,C + ,B, + , ,C +;;;; +run; + +data _null_; + set have ; + array X[*] $ a b c; + + put "before: " (_all_) (=); + call arrMissToLeftC(X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissToRight()` subroutine: <<< ####################### + +The **arrMissToRight()** subroutine shifts +all non-missing (i.e. greater than `.Z`) +numeric elements to the left side of an array +and missing values to the right, e.g. + +for array `A = [1 . 2 . 3]` the subroutine +`call arrMissToRight(A)` returns `A = [1 2 3 . .]` + +All missing values are replaced with the dot (`.`) + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissToRight(A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Numeric array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + input a b c; +cards4; +1 . 3 +. 2 . +. . 3 +;;;; +run; + +data _null_; + set have ; + array X[*] a b c; + + put "before: " (_all_) (=); + call arrMissToRight(X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `arrMissToRightC()` subroutine: <<< ####################### + +The **arrMissToRightC()** subroutine shifts +all non-missing (i.e. different than empty string) +character elements to the left side of an array +and missing values to the right, e.g. + +for array `A = ["A", " ", "B", " ", "C"]` the subroutine +`call arrMissToRightC(A)` returns `A = ["A", "B", "C", " ", " "]` + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +call arrMissToRightC(A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Character array. + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data have; + infile cards dsd dlm="," missover; + input (a b c) (: $ 1.); +cards4; +A, ,C + ,B, + , ,C +;;;; +run; + +data _null_; + set have ; + array X[*] $ a b c; + + put "before: " (_all_) (=); + call arrMissToRightC(X); + put "after: " (_all_) (=); + +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `catXFc()` function: <<< ####################### + +The **catXFc()** function is a wrapper +of the `catX()` function but with ability +to format character values. + +For array `A = ["a", " ", "c"]` the +`catXFc("upcase.", "*", A)` returns `"A*C"`. + +If format does not handle nulls they are ignored. + +*Caution!* Array parameters to function calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +catXFc(format, delimiter, A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `format` - A name of the *character* format to be used. + +2. `delimiter` - A delimiter string to be used. + +3. `A` - Character array + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + t = "t"; + u = " "; + v = "v"; + + array b[*] t u v; + + length s $ 17; + s = catXFc("upcase.", "*", B); + put (_all_) (=); +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `catXFi()` function: <<< ####################### + +The **catXFi()** function is a wrapper +of the `catX()` function but with ability +to format numeric values but +IGNORES missing values (i.e. `._`, `.`, `.a`, ..., `.z`). + +For array `A = [0, ., 2]` the +`catXFi("date9.", "#", A)` returns +`"01JAN1960#03JAN1960"` + +*Caution!* Array parameters to function calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +catXFi(format, delimiter, A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `format` - A name of the *numeric* format to be used. + +2. `delimiter` - A delimiter string to be used. + +3. `A` - Numeric array + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + x = 1; + y = .; + z = 3; + + array a[*] x y z; + + length s $ 17; + s = catXFi("z5.", "#", A); + put (_all_) (=); +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `catXFj()` function: <<< ####################### + +The **catXFj()** function is a wrapper +of the catX() function but with ability +to format character values. + +For array `A = ["a", " ", "c"]` the +`catXFj("upcase.", "*", A)` returns `"A**C"` + +If format does not handle nulls they are +printed as an empty string. + +*Caution!* Array parameters to function calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +catXFj(format, delimiter, A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `format` - A name of the *character* format to be used. + +2. `delimiter` - A delimiter string to be used. + +3. `A` - Character array + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + t = "t"; + u = " "; + v = "v"; + + array b[*] t u v; + + length s $ 17; + s = catXFj("upcase.", "*", B); + put (_all_) (=); +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `catXFn()` function: <<< ####################### + +The **catXFn()** function is a wrapper +of the `catX()` function but with ability +to format numeric values. + +For array `A = [0, 1, 2]` the +`catXFn("date9.", "#", A)` returns +`"01JAN1960#02JAN1960#03JAN1960"` + +*Caution!* Array parameters to function calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +catXFn(format, delimiter, A) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `format` - A name of the *numeric* format to be used. + +2. `delimiter` - A delimiter string to be used. + +3. `A` - Numeric array + + +### EXAMPLES AND USECASES: #################################################### + +**Example 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +data _null_; + x = 1; + y = .; + z = 3; + + array a[*] x y z; + + length s $ 17; + s = catXFn("z5.", "#", A); + put (_all_) (=); +run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `delDataset()` function: <<< ####################### + +The **delDataset()** function is a "wrapper" +for the `Fdelete()` function. +`delDataset()` function uses a text string with +a dataset name as an argument. + +Function checks for `*.sas7bdat`, `*.sas7bndx`, +and `*.sas7bvew` files and delete them. +Return code of 0 means dataset was deleted. + +For compound library files are +deleted from _ALL_ locations! + + +*Note:* +Currently only the BASE SAS engine datasets/views are deleted. + +Tested on Windows and Linux. Not tested on Z/OS. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +delDataset(lbds_) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `lbds_` - *Required*, character argument containing + name of the dataset/view to be deleted. + The `_last_` special name is honored. + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data TEST1 TEST2(index=(x)); + x = 17; + run; + + data TEST3 / view=TEST3; + set test1; + run; + + data _null_; + p = delDataset("WORK.TEST1"); + put p=; + + p = delDataset("TEST2"); + put p=; + + p = delDataset("WORK.TEST3"); + put p=; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data TEST4; + x=42; + run; + data _null_; + p = delDataset("_LAST_"); + put p=; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 3.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + options dlcreatedir; + libname user "%sysfunc(pathname(work))/user"; + + data TEST5; + x=42; + run; + + data _null_; + p = delDataset("test5"); + put p=; + run; + + libname user clear; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 4.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data TEST6; + x=42; + run; + + %put *%sysfunc(delDataset(test6))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 5.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + options dlcreatedir; + libname L1 "%sysfunc(pathname(work))/L)1"; + libname L2 "%sysfunc(pathname(work))/L(2"; + libname L3 "%sysfunc(pathname(work))/L'3"; + + data L1.TEST7 L2.TEST7 L3.TEST7; + x=42; + run; + + libname L12 ("%sysfunc(pathname(work))/L(1" "%sysfunc(pathname(work))/L)2"); + libname L1L2 (L2 L3); + + %put *%sysfunc(delDataset(L12.test7))*; + %put *%sysfunc(delDataset(L1L2.test7))*; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `qsortInCbyProcProto()` proto function: <<< ####################### + +The **qsortInCbyProcProto()** is external *C* function, +this is the implementation of the *Quick Sort* algorithm. + +The function is used **internally** by +functions in the *BasePlus* package. + +Asumptions: +- smaller subarray is sorted first, +- subarrays of *size < 11* are sorted by *insertion sort*, +- pivot is selected as median of low index value, + high index value, and (low+high)/2 index value. + +`!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!`
+`!CAUTION! Sorted array CANNOT contains SAS missing values !`
+`!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!`
+ +### SYNTAX: ################################################################### + +The basic syntax is the following: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +qsortInCbyProcProto(arr, low, high) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `arr` - An array of double type to be sorted. + +2. `low` - An integer low index of starting position (from which the sorting is done). + +3. `high` - An integer high index of ending position (up to which the sorting is done). + + +### REFERENCES: #################################################### + +*Reference 1.* + +Insertion sort for arrays smaller then 11 elements: + +Based on the code from the following WikiBooks page [2020.08.14]: + +[https://pl.wikibooks.org/wiki/Kody_%C5%BAr%C3%B3d%C5%82owe/Sortowanie_przez_wstawianie](https://pl.wikibooks.org/wiki/Kody_%C5%BAr%C3%B3d%C5%82owe/Sortowanie_przez_wstawianie) + + +*Reference 2.* + +Iterative Quick Sort: + +Based on the code from the following pages [2020.08.14]: + +[https://www.geeksforgeeks.org/iterative-quick-sort/](https://www.geeksforgeeks.org/iterative-quick-sort/) + +[https://www.geeksforgeeks.org/c-program-for-iterative-quick-sort/](https://www.geeksforgeeks.org/c-program-for-iterative-quick-sort/) + +--- + +## >>> `fromMissingToNumberBS()` function: <<< ####################### + +The **fromMissingToNumberBS()** function +gets numeric missing value or a number +as an argument and returns an integer +from 1 to 29. + +For a numeric missing argument +the returned values are: +- 1 for `._` +- 2 for `.` +- 3 for `.a` +- ... +- 28 for `.z` and +- 29 for *all other*. + +The function is used **internally** by +functions in the *BasePlus* package. + +For *missing value arguments* the function +is an inverse of the `fromNumberToMissing()` function. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +fromMissingToNumberBS(x) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `x` - A numeric missing value or a number. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data _null_; + do x = ._, ., .a, .b, .c, 42; + y = fromMissingToNumberBS(x); + put x= y=; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `fromNumberToMissing()` function: <<< ####################### + +The **fromNumberToMissing()** function +gets a number as an argument and returns +a numeric missing value or zero. + +For a numeric argument +the returned values are: +- `._` for 1 +- `.` for 2 +- `.a` for 3 +- ... +- `.z` for 28 and +- `0` for *all other*. + +The function is used **internally** by +functions in the *BasePlus* package. + +For arguments 1,2,3, ..., and 28 the function +is an inverse of the `fromMissingToNumberBS()` function. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~sas +fromNumberToMissing(x) +~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `x` - A numeric value. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + data _null_; + do x = 1 to 29; + y = fromNumberToMissing(x); + put x= y=; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `quickSort4NotMiss()` subroutine: <<< ####################### + +The **quickSort4NotMiss()** subroutine is an alternative to the +`CALL SORTN()` subroutine for 1-based big arrays (i.e. `> 10'000'000` elements) +when memory used by `call sortn()` may be an issue. +For smaller arrays the memory footprint is not significant. + +The subroutine is based on an iterative quick sort algorithm +implemented in the `qsortInCbyProcProto()` *C* prototype function. + + +**Caution 1!** Array _CANNOT_ contains missing values! + +**Caution 2!** Array parameters to subroutine calls must be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +call quickSort4NotMiss(A) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Argument is a 1-based array of NOT missing numeric values. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** For session with 8GB of RAM, + array of size 250'000'000 with values in range + from 0 to 99'999'999 and _NO_ missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + test[_N_] = int(100000000*rand("uniform")); + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSort4NotMiss (test); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2.** Resources comparison for + session with 8GB of RAM. + + Array of size 250'000'000 with random values + from 0 to 999'999'999 and _NO_ missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 8.82s + memory 1'953'470.62k + OS Memory 1'977'436.00k + + Call quickSort4NotMiss: + Sorting time 66.92s + Memory 1'954'683.06k + OS Memory 1'977'436.00k + + Call quickSortLight: + Sorting time 70.98s + Memory 1'955'479.71k + OS Memory 1'977'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `quickSortHash()` subroutine: <<< ####################### + +The **quickSortHash()** subroutine is an alternative to the +`CALL SORTN()` subroutine for 1-based big arrays (i.e. `> 10'000'000` elements) +when memory used by `call sortn()` may be an issue. +For smaller arrays the memory footprint is not significant. + +The subroutine is based on an iterative quick sort algorithm +implemented in the `qsortInCbyProcProto()` *C* prototype function. + +The number of "sparse distinct data values" is set to `100'000` to +use the hash sort instead of the quick sort. + E.g. when number of unique values for sorting is less then + 100'000 then an ordered hash table is used to store the data + and their count and sort them. + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +*Note!* Due to improper memory reporting/releasing for hash + tables in FCMP procedure the reported memory used after running + the function may not be in line with the RAM memory required + for processing. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +call quickSortHash(A) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Argument is a 1-based array of numeric values. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** For session with 8GB of RAM + Array of size 250'000'000 with values in range + from 0 to 99'999'999 and around 10% of various + missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + array m[0:27] _temporary_ + (._ . .A .B .C .D .E .F .G .H .I .J .K .L + .M .N .O .P .Q .R .S .T .U .V .W .X .Y .Z); + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + _I_ + 1; + if rand("uniform") > 0.1 then test[_I_] = int(100000000*rand("uniform")); + else test[_I_] = m[mod(_N_,28)]; + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSortHash (test); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2.** For session with 8GB of RAM + Array of size 250'000'000 with values in range + from 0 to 9'999 and around 10% of various + missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + array m[0:27] _temporary_ + (._ . .A .B .C .D .E .F .G .H .I .J .K .L + .M .N .O .P .Q .R .S .T .U .V .W .X .Y .Z); + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + _I_ + 1; + if rand("uniform") > 0.1 then test[_I_] = int(10000*rand("uniform")); + else test[_I_] = m[mod(_N_,28)]; + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSortHash (test); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 3.** Resources comparison for + session with 8GB of RAM + + A) Array of size 10'000'000 with + random values from 0 to 9'999 range (sparse) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 0.61s + Memory 78'468.50k + OS Memory 101'668.00k + + Call sortn: + Sorting time 0.87s + Memory 1'120'261.53k + OS Memory 1'244'968.00k + + Call quickSortHash: + Sorting time 6.76s + Memory 1'222'242.75k(*) + OS Memory 1'402'920.00k(*) + + Call quickSortLight: + Sorting time 23.45s + Memory 80'527.75k + OS Memory 101'924.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + B) Array of size 10'000'000 with + random values from 0 to 99'999'999 range (dense) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 0.6s + Memory 78'463.65k + OS Memory 101'924.00k + + Call sortn: + Sorting time 1.51s + Memory 1'120'253.53k + OS Memory 1'244'968.00k + + Call quickSortHash: + Sorting time 6.28s + Memory 1'222'241.93k(*) + OS Memory 1'402'920.00k(*) + + Call quickSortLight: + Sorting time 0.78s + Memory 80'669.28k + OS Memory 102'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + C) Array of size 250'000'000 with + random values from 0 to 999'999'999 range (dense) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 15.34s + memory 1'953'471.81k + OS Memory 1'977'436.00k + + Call sortn: + FATAL: Insufficient memory to execute DATA step program. + Aborted during the COMPILATION phase. + ERROR: The SAS System stopped processing this step + because of insufficient memory. + + Call quickSortHash: + Sorting time 124.68s + Memory 7'573'720.34k(*) + OS Memory 8'388'448.00k(*) + + Call quickSortLight: + Sorting time 72.41s + Memory 1'955'520.78k + OS Memory 1'977'180.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + D) Array of size 250'000'000 with + random values from 0 to 99'999 range (sparse) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 16.07 + Memory 1'953'469.78k + OS Memory 1'977'180.00k + + Call sortn: + FATAL: Insufficient memory to execute DATA step program. + Aborted during the COMPILATION phase. + ERROR: The SAS System stopped processing this step + because of insufficient memory. + + Call quickSortHash: + Sorting time 123.5s + Memory 7'573'722.03k + OS Memory 8'388'448.00k + + Call quickSortLight: + Sorting time 1'338.25s + Memory 1'955'529.90k + OS Memory 1'977'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(*) When using hash tables in `Proc FCMP` the RAM + usage is not indicated properly. The memory + allocation is reported up to the session limit + and then reused if needed. The really required + memory is in fact much less then reported. + +--- + +## >>> `quickSortHashSDDV()` subroutine: <<< ####################### + +The **quickSortHashSDDV()** subroutine is an alternative to the +`CALL SORTN()` subroutine for 1-based big arrays (i.e. `> 10'000'000` elements) +when memory used by `call sortn()` may be an issue. +For smaller arrays the memory footprint is not significant. + +The subroutine is based on an iterative quick sort algorithm +implemented in the `qsortInCbyProcProto()` *C* prototype function. + +The number of "sparse distinct data values" (argument `SDDV`) may +be adjusted to use the hash sort instead of the quick sort. + E.g. when number of unique values for sorting is less then + some *N* then an ordered hash table is used to store the data + and their count and sort them. + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +*Note!* Due to improper memory reporting/releasing for hash + tables in FCMP procedure the report memory used after running + the function may not be in line with the RAM memory required + for processing. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +call quickSortHashSDDV(A, SDDV) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Argument is a 1-based array of numeric values. + +2. `SDDV` - A number of distinct data values, e.g. 100'000. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** For session with 8GB of RAM + Array of size 250'000'000 with values in range + from 0 to 99'999'999 and around 10% of various + missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + array m[0:27] _temporary_ + (._ . .A .B .C .D .E .F .G .H .I .J .K .L + .M .N .O .P .Q .R .S .T .U .V .W .X .Y .Z); + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + _I_ + 1; + if rand("uniform") > 0.1 then test[_I_] = int(100000000*rand("uniform")); + else test[_I_] = m[mod(_N_,28)]; + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSortHashSDDV (test, 2e4); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2.** For session with 8GB of RAM + Array of size 250'000'000 with values in range + from 0 to 9'999 and around 10% of various + missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + array m[0:27] _temporary_ + (._ . .A .B .C .D .E .F .G .H .I .J .K .L + .M .N .O .P .Q .R .S .T .U .V .W .X .Y .Z); + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + _I_ + 1; + if rand("uniform") > 0.1 then test[_I_] = int(10000*rand("uniform")); + else test[_I_] = m[mod(_N_,28)]; + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSortHashSDDV (test, 2e4); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +--- + +## >>> `quickSortLight()` subroutine: <<< ####################### + +The **quickSortLight()** subroutine is an alternative to the +`CALL SORTN()` subroutine for 1-based big arrays (i.e. `> 10'000'000` elements) +when memory used by `call sortn()` may be an issue. +For smaller arrays the memory footprint is not significant. + +The subroutine is based on an iterative quick sort algorithm +implemented in the `qsortInCbyProcProto()` *C* prototype function. + +*Caution!* Array parameters to subroutine calls *must* be 1-based. + +### SYNTAX: ################################################################### + +The basic syntax is the following, the `<...>` means optional parameters: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas +call quickSortLight(A) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Arguments description**: + +1. `A` - Argument is a 1-based array of numeric values. + + +### EXAMPLES AND USECASES: #################################################### + +**EXAMPLE 1.** For session with 8GB of RAM + Array of size 250'000'000 with values in range + from 0 to 99'999'999 and around 10% of various + missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + %let size = 250000000; + options fullstimer; + + data _null_; + array test[&size.] _temporary_ ; + + array m[0:27] _temporary_ + (._ . .A .B .C .D .E .F .G .H .I .J .K .L + .M .N .O .P .Q .R .S .T .U .V .W .X .Y .Z); + + t = time(); + call streaminit(123); + do _N_ = &size. to 1 by -1; + _I_ + 1; + if rand("uniform") > 0.1 then test[_I_] = int(100000000*rand("uniform")); + else test[_I_] = m[mod(_N_,28)]; + end; + t = time() - t; + put "Array population time: " t; + + put "First 50 elements before sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + + t = time(); + call quickSortLight (test); + t = time()-t; + put "Sorting time: " / t=; + + put; put "First 50 elements after sorting:"; + do _N_ = 1 to 20; + put test[_N_] = @; + end; + run; +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + +**Example 2.** Resources comparison for + session with 8GB of RAM. + + Array of size 250'000'000 with random values + from 0 to 999'999'999 and _NO_ missing values. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 8.82s + memory 1'953'470.62k + OS Memory 1'977'436.00k + + Call quickSort4NotMiss: + Sorting time 66.92s + Memory 1'954'683.06k + OS Memory 1'977'436.00k + + Call quickSortLight: + Sorting time 70.98s + Memory 1'955'479.71k + OS Memory 1'977'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Example 3.** Resources comparison for + session with 8GB of RAM + + A) Array of size 10'000'000 with + random values from 0 to 9'999 range (sparse) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 0.61s + Memory 78'468.50k + OS Memory 101'668.00k + + Call sortn: + Sorting time 0.87s + Memory 1'120'261.53k + OS Memory 1'244'968.00k + + Call quickSortHash: + Sorting time 6.76s + Memory 1'222'242.75k(*) + OS Memory 1'402'920.00k(*) + + Call quickSortLight: + Sorting time 23.45s + Memory 80'527.75k + OS Memory 101'924.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + B) Array of size 10'000'000 with + random values from 0 to 99'999'999 range (dense) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 0.6s + Memory 78'463.65k + OS Memory 101'924.00k + + Call sortn: + Sorting time 1.51s + Memory 1'120'253.53k + OS Memory 1'244'968.00k + + Call quickSortHash: + Sorting time 6.28s + Memory 1'222'241.93k(*) + OS Memory 1'402'920.00k(*) + + Call quickSortLight: + Sorting time 0.78s + Memory 80'669.28k + OS Memory 102'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + C) Array of size 250'000'000 with + random values from 0 to 999'999'999 range (dense) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 15.34s + memory 1'953'471.81k + OS Memory 1'977'436.00k + + Call sortn: + FATAL: Insufficient memory to execute DATA step program. + Aborted during the COMPILATION phase. + ERROR: The SAS System stopped processing this step + because of insufficient memory. + + Call quickSortHash: + Sorting time 124.68s + Memory 7'573'720.34k(*) + OS Memory 8'388'448.00k(*) + + Call quickSortLight: + Sorting time 72.41s + Memory 1'955'520.78k + OS Memory 1'977'180.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + D) Array of size 250'000'000 with + random values from 0 to 99'999 range (sparse) + and around 10% of missing data. +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas + Array: + Population time 16.07 + Memory 1'953'469.78k + OS Memory 1'977'180.00k + + Call sortn: + FATAL: Insufficient memory to execute DATA step program. + Aborted during the COMPILATION phase. + ERROR: The SAS System stopped processing this step + because of insufficient memory. + + Call quickSortHash: + Sorting time 123.5s + Memory 7'573'722.03k + OS Memory 8'388'448.00k + + Call quickSortLight: + Sorting time 1'338.25s + Memory 1'955'529.90k + OS Memory 1'977'436.00k +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(*) When using hash tables in `Proc FCMP` the RAM + usage is not indicated properly. The memory + allocation is reported up to the session limit + and then reused if needed. The really required + memory is in fact much less then reported. + +--- + +## License #################################################################### + +Copyright (c) 2020 Bartosz Jablonski + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- diff --git a/packages/baseplus.zip b/packages/baseplus.zip index 17b771b..5c7ba5e 100644 Binary files a/packages/baseplus.zip and b/packages/baseplus.zip differ