Not all tests pass using non-English culture (e.g. Turkish, Korean, German)

[bergmeister]

Steps to reproduce

[System.Threading.Thread]::CurrentThread.CurrentCulture = [cultureinfo]::CreateSpecificCulture('tr-TR')
# Build and install latest version of PSSA after the fix of PR 1097
Invoke-Pester -Script ".\Tests\Engine",".\Tests\Rules",".\Tests\Documentation"

Expected behavior

All Tests pass

Actual behavior

21 tests failed, some of them clearly related to the special i character in Turkish.
The same number of failures also happen using Korean culture ko-KR

Environment data

This is as of the latest state of development after the first fix of PR 1097 using PowerShell 5.1

[bergmeister]

cc @alatas that I ran all tests using Turkish culture to possibly find more issues. Mostly it seems that PSSA behaves reasonably OK-ish using Turkish culture but there are some test cases that fail that we can look into

[alatas]

@bergmeister I run all tests in a computer that Turkish culture is set, and 67 of tests are failed. Then, I built the source from ground up and get the same result (Win10 with PS 5.1)

I'd happy to help if you need

[bergmeister]

@alatas So far I could identify and fix 2 problems and locally all tests are working fine now on Windows PowerShell when I change both the CurrentCulture and CurrentUICulture to Turkish but somehow not in the CI system. You can see those additional improvements in PR #1099. Locally, I see that the locale even matters building and I get a similar failure count to you when I change my culture to Turkish as well, therefore I suspect there are also some .Net Core and PSCore bugs, I will try to isolate now build from runtime problems.
On PowerShell Core I could not fix it yet locally, there is issue #944 where the tests do not pass locally on my machine using PSCore and to me it seems that is because my UI culture is usually en-GB. To me it seems that on PowerShell Core there is a general Problem when the UI culture is not en-usand on Windows PowerShell only if the CurrentCulture and UICulture are not English. The PowerShell Core issue aside, I'll try to create a Turkish VM and take it from there. Thanks for your help, it would be great if you could run the tests again on your machine by installing my latest locally built module here by just copying into C:\Program Files\WindowsPowerShell\Modules
PSScriptAnalyzer.zip

[alatas]

@bergmeister I use the following commands to test.

.\build.ps1 -Clean
.\build.ps1 -Framework full -PSVersion 5 -Configuration Release
Import-Module .\out\PSScriptAnalyzer\PSScriptAnalyzer.psd1
.\build -Test
Import-Module ./build.psm1
Get-TestFailures

After the commit 227b64f, 67 failed test drops to 46 in Win10 Powershell 5.1.

I downloaded the compiled file and extract it into modules folder, deleted TestResults.xml, Then run the tests again without importing the locally compiled files. But, this approach hasn't changed the results. I changed the failed test resulting scripts a little bit to summarise the results as a csv file. It outputs the failed tests as follows

"Order","CaseName","TestFile","Time"
"1","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).Test parameter help for Get-ScriptAnalyzerRule.gets help for parameter: CustomRulePath : in Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0845"
"2","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).Test parameter help for Get-ScriptAnalyzerRule.gets help for parameter: Name : in Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"3","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).Test parameter help for Get-ScriptAnalyzerRule.gets help for parameter: RecurseCustomRulePath : in Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0076"
"4","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).Test parameter help for Get-ScriptAnalyzerRule.help for Get-ScriptAnalyzerRule has correct parameter type for RecurseCustomRulePath","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0206"
"5","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).Test parameter help for Get-ScriptAnalyzerRule.gets help for parameter: Severity : in Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.008"
"6","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).should not be auto-generated","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.3422"
"7","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).gets description for Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0217"
"8","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).gets example code from Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0181"
"9","Test help for Get-ScriptAnalyzerRule in PSScriptAnalyzer (1.17.1).gets example help from Get-ScriptAnalyzerRule","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0165"
"10","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-Formatter.gets help for parameter: Range : in Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0257"
"11","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-Formatter.help for Invoke-Formatter has correct parameter type for Range","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0679"
"12","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-Formatter.gets help for parameter: ScriptDefinition : in Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0078"
"13","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-Formatter.gets help for parameter: Settings : in Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"14","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).should not be auto-generated","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0554"
"15","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).gets description for Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0082"
"16","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).gets example code from Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0072"
"17","Test help for Invoke-Formatter in PSScriptAnalyzer (1.17.1).gets example help from Invoke-Formatter","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"18","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: CustomRulePath : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0416"
"19","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: EnableExit : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0081"
"20","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for EnableExit","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0091"
"21","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: ExcludeRule : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0078"
"22","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: Fix : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0078"
"23","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for Fix","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0095"
"24","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: IncludeDefaultRules : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"25","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for IncludeDefaultRules","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0091"
"26","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: IncludeRule : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0081"
"27","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: Path : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"28","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: Recurse : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0076"
"29","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for Recurse","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0291"
"30","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: RecurseCustomRulePath : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0102"
"31","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for RecurseCustomRulePath","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0088"
"32","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: ReportSummary : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0079"
"33","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for ReportSummary","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0084"
"34","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: SaveDscDependency : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0075"
"35","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for SaveDscDependency","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0086"
"36","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: ScriptDefinition : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0099"
"37","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: Settings : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0076"
"38","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: Severity : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0074"
"39","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.gets help for parameter: SuppressedOnly : in Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0076"
"40","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.help for Invoke-ScriptAnalyzer has correct parameter type for SuppressedOnly","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0089"
"41","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.finds help parameter in code: Confirm","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0075"
"42","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).Test parameter help for Invoke-ScriptAnalyzer.finds help parameter in code: WhatIf","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0075"
"43","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).should not be auto-generated","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0534"
"44","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).gets description for Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0081"
"45","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).gets example code from Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0075"
"46","Test help for Invoke-ScriptAnalyzer in PSScriptAnalyzer (1.17.1).gets example help from Invoke-ScriptAnalyzer","C:\Kod\PSScriptAnalyzer\Tests\Engine\ModuleHelp.Tests.ps1","0.0078"

[bergmeister]

@alatas Thanks, that looks all good. The test failures are probably because you did not build the documentation (i.e. the cmdlet help), hence why the help is missing. You can build the documentation using .\build.ps1 -Documentation (you would do this before you import the module in line3 of your example). I am currently working on the build script in another PR to improve it and make it easier to use in the future

[alatas]

@bergmeister yes, you are right! I built the documentation, and all the tests are succeed 👍

.\build.ps1 -Clean
.\build.ps1 -Framework full -PSVersion 5 -Configuration Release
.\build.ps1 -Documentation
Import-Module .\out\PSScriptAnalyzer\PSScriptAnalyzer.psd1
Remove-Item TestResults.xml -ErrorAction SilentlyContinue
.\build -Test
Import-Module ./build.psm1
Get-TestFailures