Mark Minasi's Tech Forum
Sign up Calendar Latest Topics
 
 
 


Reply
  Author   Comment  
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #1 
RVTOOLS is a windows .NET 2.0 application which uses the VI SDK to display information about your virtual machines and ESX hosts. 

http://www.robware.net/

You can find documentation and a BAT file here - 

http://www.robware.net/download/RVTools.pdf

This is the preferred Application and Reporting method for this client,,, so using vCheck or PowerCLI is NOT an option at this time...  (I am working on that)
 

This tool is being used to scan our ESX once a week and the output for each ESX host is saved to an XLS file, which is then copied to a Network Share where the Data is made available to the respective parties...  Currently this is a manual process where a user that has Read Access to all of the ESX Hosts listed (DomainA.txt & DomainB.txt) logs into TWO Separate WIN Servers where he runs a portion of the Server List on each Server.

Even with two separate ESX lists on two separate servers it still takes up to 5 to 6 hours to complete, then the BAT file copies the xls files from each server to the single network share.


I created a similar method using PowerShell where I am able to run 5 instances of the App at a time.  5 seems to be the sweet spot for this app since it is very CPU intensive anything over that at a given time will drag the CPU to its knees and I start getting .NET Errors and ALL of the processes get hung and fail...

If there is an error it will be displayed in the Process Property "MainWindowTitle"

Examples
  • "RVTools: SSPILogin failed"
  • "RVTools: Retrieve Service Content"
  • "RVTools: Retrieve Service Content"

RVTOOLS00.gif 


Which also creates a GUI Alert which is waiting for someone to click OK.

RVTOOLS01.gif 

RVTOOLS02.gif 





To check for any of these processes that get hung due to an error and to keep the number to 5 I used a Do{} While{} loop in my Foreach to only allow 5 at a given time.



Function WORKING_06{
param (
$Date = (Get-Date -f "yyyyMMdd"),
$Servers = @(Get-Content "D:\Temp\RVTOOLS\ESX_All.txt"),
$ExportPath = "D:\Temp\RVTOOLS\$Date",
$MaxNumberOfThings = 5,
$MinNumberOfThings = 0
)
# Creates the Folder & Path with current date
New-Item -Path "$ExportPath" -ItemType Directory -ErrorAction SilentlyContinue

Foreach ($Server in $Servers){
Write-Output "Working with Server: $($Server).Name"
# This is executing the RVTOOL App against Each Server - NOTE THIS IS USING -PASSTHROUGHAUTH ..!
. "C:\Program Files (x86)\RobWare\RVTools\RVTools.exe" -passthroughAuth -s "$Server" -c ExportAll2xls -d "$ExportPath" -f "$Server.xls"
Do {
# Look for any processes that have a GUI waiting to click OK
$StopProc01 = Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | Where{$_.MainWindowTitle -match "RVTools: SSPILogin failed" -or $_.MainWindowTitle -match "RVTools: Retrieve Service Content" -or $_.MainWindowTitle -match "RVTools: Retrieve Service Content"} | Select ID,Name,ProcessName,MainWindowTitle
# This txt file should be empty If anything is in here there is a problem with the Account, Server or Network access
$StopProc01 | Out-File "$ExportPath\$Date Script_Error.txt" -Append
# If Process is found with Error Message listed it will stop that process (Error MSG is a GUI Alert)
$StopProc01 | Stop-Process
# Gets the Count Of Running RVTOOLS Processes - Limits it to 5 at a time
$Count01 = (Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | measure).count
if($Count01 -gt $MaxNumberOfThings) {
Start-Sleep -Seconds 30
}
# This keeps runninng while there are processes running equal to or greater than 5
} While( $Count01 -ge $MaxNumberOfThings )

} # End Foreach Server





Once it has passed all of the ESX Servers in the ESX_ALL.txt then it will drop into the next D{} While {} Loop where it will again watch for any errors, which are listed in the running Processes and will wait until they have all completed or have been Stopped before it moves onto the next step of using RoboCopy to copy the XLS files to the network share.



Write-Output "Starting to check for RVTOOLS Processes that are still running" 
Start-Sleep -Seconds 5
                # This is watching for RVTOOLS Processes that are still running & any SSPILogin failed RVTools: Retrieve Service Content Errors that need to be closed.
                Do {
                    # Look for any processes that have a GUI waiting to click OK 
                    $StopProc02 = Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | Where{$_.MainWindowTitle -match "RVTools: SSPILogin failed" -or $_.MainWindowTitle -match "RVTools: Retrieve Service Content" -or $_.MainWindowTitle -match "RVTools: Retrieve Service Content"} | Select ID,Name,ProcessName,MainWindowTitle
                    Write-Output $StopProc02
                    $StopProc02 | Out-File "$ExportPath\$Date Script_Error.txt" -Append
                    $StopProc02 | Stop-Process
                    #Get Count Of Running Processes - When is $Null Move On To RoboCopy
                    $Count02 = (Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | measure).count
                    Write-Output $StopProc02
                    if($Count02 -gt $MinNumberOfThings) {
                      Start-Sleep -Seconds 5
                    }
                  } While( $Count02 -ne $MinNumberOfThings )
Start-Sleep -Seconds 15
$DoneCount = (Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | measure).count
Write-Output "Number Of RVTOOLS Running : $($DoneCount)"
#
Start-Sleep -Seconds 15
$FinishRVTOOLS = Get-Date
Write-Output "RVTOOLS Has Completed at: $($FinishRVTOOLS)"
Start-Sleep 15
#
$StartRobo = Get-Date
Write-Output "Starting RoboCopy Of All Files at:  $($StartRobo)"
#
#RoboTools
# Sets Date For Folder Date Stamp EX: 20160205 
# Source & Destination & LOG File Paths
$SrcRVTOOLS = $ExportPath
$DestRVTOOLS = "\\BigDog01\Capacity_Planning\RVTOOLS\$Date"
$RVTOOLSRoboCopyLogfile = "$ExportPath\$Date RVTOOLS_RoboCopy.log"
$RVTOOLSScriptLogfile = "$ExportPath\$Date Script_Error.txt"
# Copy Folder DomainA & DomainB with Robocopy
Robocopy $SrcRVTOOLS $DestRVTOOLS /E /SEC /ETA /LOG:$RVTOOLSRoboCopyLogfile
# Email Notice of Reports with attachments
$Subject = "Robocopy of RVTOOLS EXPORT for DomainA & DomainB Summary $Date"
$TO = "timbolton@BigDog.com"
$Attachments = $RVTOOLSRoboCopyLogfile,$RVTOOLSScriptLogfile 
$SMTPServer = "BigDog.com"
$SMTPPort = "25"
$Body = @"
Hello,
Robocopy of RVTOOLS EXPORT for DomainA & DomainB Summary $Date has completed.  
See attached log file $Date RVTOOLS.log for details and to check for any Errors.

DomainA & DomainB: $DestRVTOOLS

 
-----------------------------------------------------------------------------
If you have any questions please contact Infrastructure Tools & Automation at Email:  Engineering@BigDog.com
PLEASE DO NOT REPLY TO THIS EMAIL. REPLIES TO THIS ADDRESS ARE ROUTED TO AN UNMONITORED MAILBOX.
-----------------------------------------------------------------------------
"@
$Email = @{
From = "RVTOOLS_EXPORT@BigDog.com"
To = $TO
#Cc = 
#Bcc = "$MyEmail"
Subject = "$Subject"
Body = "$Body"
SMTPServer = "BigDog.com"
Attachments = $Attachments
}
Send-MailMessage @Email

$FinishRobo = Get-Date
Write-Output 'RoboCopy Completed ' $($FinishRobo)
} # End Function Working_06




This works fine if I am logged into the Server that the RVTOOLS App is running on.  I can even pass Different Credentials to run the function with a Domain Account that has more access to more ESX Host than my account.  The Get-Process check for the hung processes still works...

However!!  If I set this up as a Scheduled Task or run this from BladeLogic and pass the credentials then the Get-Process check for the hung processes DOES NOT WORK!

When I look at the running processes (Logged in with my account) I can see the running RVTOOLS Processes but the values that I am keying off of are all blank.  Even the StartTime is blank so I cannot even use that anymore.

When I open a PowerShell console As Administrator I can see the StartTime but the "CPU" and "MainWindowTitle" fields are all blank even when I know for a fact that the process has hung...


NOTE!!!  
  • I am looking to see if I can get any information from Win32_Process that I can key off of so that when a Process is hung I can get its PID or ID and stop it then allow the next one in the list to process...
  • Due to the number of ESX Hosts that we need scanned the BAT file, running one at a time, is not a good option.  When I am running this with PowerShell / RoboCopy I have been able to scan 80% of the list in under 45 mins.  That includes coping the files to the network share...
  • I tried using Start-Job but when doing so it started the application for everyone of the listed servers at one time since as soon as the App Started, the Job was set to Completed even though the App was still running using up CPU cycles...
  • If we only had 10 or 20 ESX Hosts this none of this would not be an issue...

Any help or suggestions would be greatly appreciated!







__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
Pieter

Avatar / Picture

Senior Member
Registered:
Posts: 301
Reply with quote  #2 
Sorry for asking the obvious, but the credentials used for the Scheduled task are the same as for the local logon ?



__________________
Pieter Demeulemeester
0
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #3 
Quote:
Originally Posted by Pieter
Sorry for asking the obvious, but the credentials used for the Scheduled task are the same as for the local logon ?




Yes, it was...

Well let me rephrase.  There "was" a profile for the account that has greater access into the ESX Hosts which the Scheduled Task was running as with "Run With Highest Privileges" set and "Run whether user is logged on or not"  This account "was" also part of the Local Admin Group of that Server.

I say "was" due to the fact that option is no longer available due to new security restrictions preventing setting a Sched Task to "Run whether user is logged on or not" 

That is another topic for another day...  : )




__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
wobble_wobble

Avatar / Picture

Associate Troublemaker Apprentice
Registered:
Posts: 937
Reply with quote  #4 
Tim,

I'm not as advanced as that, still using a batch file...

The RVTools instance is using the passthru credentials, so if the account calling the scheduled task is not permitted against the vCenter/ ESX instances, it will fail.
Might this help - http://blog.coretech.dk/rja/store-encrypted-password-in-a-powershell-script/

__________________
Have you tried turning it off and walking away? The next person can fix it!

New to the forum? Read this
0
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #5 
Quote:
Originally Posted by wobble_wobble
Tim,

I'm not as advanced as that, still using a batch file...

The RVTools instance is using the passthru credentials, so if the account calling the scheduled task is not permitted against the vCenter/ ESX instances, it will fail.
Might this help - http://blog.coretech.dk/rja/store-encrypted-password-in-a-powershell-script/


REAL CLOSE!!!!

Working off of 

Get-WmiObject Win32_PerfRawData_PerfProc_Process


Just need to tweak the timing a bit so it doesnt kill a process that is pausing while the XLS file is created & I need to pass the Credential to RoboCopy to create the Destination Location since the script is running as a Local Admin Account not a Domain Account.

SO,,,,  I need to pass the same Credentials that is running RVTOOLS which has full access to the Network Share location.



__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
wobble_wobble

Avatar / Picture

Associate Troublemaker Apprentice
Registered:
Posts: 937
Reply with quote  #6 
No, the account running the scheduled task needs to be the same one as the ESXi/ vCenter account.

Not sure about your file share...i just add in the account and permissions


__________________
Have you tried turning it off and walking away? The next person can fix it!

New to the forum? Read this
0
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #7 
Quote:
Originally Posted by wobble_wobble
No, the account running the scheduled task needs to be the same one as the ESXi/ vCenter account.

Not sure about your file share...i just add in the account and permissions



Joe,

If your environment is smaller and your account has access then yours will be easy.

For mine I have to deal with a lot of unknowns and "maybe" 's  

For my issue I am going to add a -Timeout parameter to this existing bit of code.

I have found that with our environment, it "should" finish within 12 minutes so I am setting this to 15 minutes -


                   

$StopProc01 = Get-WmiObject Win32_Process | Where{$_.Name -Like "RVTOOLS*" -and $_.ConvertToDateTime( $_.CreationDate ).Addminutes(15) -lt (Get-Date) } | Select Handle
        If($StopProc01.Handle){
       Stop-Process -ID $StopProc01.Handle -Force -ErrorAction SilentlyContinue
       $StopProc01 | Out-File "$ExportPath\$Date Script_Error.txt" -Append
       }
      Else{}
# Get Count Of Running Processes - Limit to 5 at a time
$Count01 = (Get-process -Name "RVTOOLS" -ErrorAction SilentlyContinue | measure).count
      If($Count01 -gt $MaxNumberOfThings) {
Start-Sleep -Seconds 5
      }
# This keeps runninng processes to no more than 5 at a time.
} While( $Count01 -ge $MaxNumberOfThings )





As soon as I get done with this meeting I will post it here or will shoot you a Word Doc with info on how it works.




__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #8 
Making updates to both of these methods today.  Will post when available...

I was suppressing simple errors in PowerShell but the tool we were using for automation is catching everything and crashing the job via filling up its log files...

__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
Kb0ykg

Still Checking the Forum Out
Registered:
Posts: 3
Reply with quote  #9 
Good Evening,

Was wondering if you were able to get powershell, encrypted password to work with RVtools as a scheduled task?
0
cj_berlin

Avatar / Picture

Senior Member
Registered:
Posts: 420
Reply with quote  #10 
Quote:
Originally Posted by Kb0ykg
Good Evening, Was wondering if you were able to get powershell, encrypted password to work with RVtools as a scheduled task?


If by "powershell encrypted password" you mean one stored in a SecureString, then you need to understand where the encryption key is stored. If you encrypt a SecureString, a new key is generated for that. It is then stored in C:\Users\<USERNAME>\AppData\Roaming\Microsoft\Crypto\RSA\<USERSID> as a file. In this file, the key is encrypted using your password. Now when you decrypt the string and run the shell as the same user, all is fine. If you run it as another user, you can't decrypt because that other user context doesn't have a key for this SecureString. You could of course specify the key but then you would have crypted text and the key in one place ;-)

__________________
Evgenij Smirnov

My personal blog (German): http://www.it-pro-berlin.de/
My stuff on PSGallery: https://www.powershellgallery.com/profiles/it-pro-berlin.de/
0
Kb0ykg

Still Checking the Forum Out
Registered:
Posts: 3
Reply with quote  #11 
Thanks for the quick reply. Sorry, yes the secureString. We created the SecureString by running the powershell application as the user that we needed the SecureString created for. We then have a powershell script(again ran as the same account that created the SecureString) that has the required ps code to decrypt the SecureString and we attempt to pass it rvtools -u username -p $cred -s vcenter.local.com but it not having any luck with it accepting the SecureString.

Any suggestions?

Quote:
Originally Posted by cj_berlin


If by "powershell encrypted password" you mean one stored in a SecureString, then you need to understand where the encryption key is stored. If you encrypt a SecureString, a new key is generated for that. It is then stored in C:\Users\<USERNAME>\AppData\Roaming\Microsoft\Crypto\RSA\<USERSID> as a file. In this file, the key is encrypted using your password. Now when you decrypt the string and run the shell as the same user, all is fine. If you run it as another user, you can't decrypt because that other user context doesn't have a key for this SecureString. You could of course specify the key but then you would have crypted text and the key in one place ;-)
0
cj_berlin

Avatar / Picture

Senior Member
Registered:
Posts: 420
Reply with quote  #12 
You don't pass a SecureString to RVTOOLS but rather plaintext credentials. Can you verify that your script actually does decrypt the password to the correct value?
__________________
Evgenij Smirnov

My personal blog (German): http://www.it-pro-berlin.de/
My stuff on PSGallery: https://www.powershellgallery.com/profiles/it-pro-berlin.de/
0
Kb0ykg

Still Checking the Forum Out
Registered:
Posts: 3
Reply with quote  #13 
Quote:
Originally Posted by cj_berlin
You don't pass a SecureString to RVTOOLS but rather palintext credentials. Can you verify that your script actually does decrypt the password to the correct value?


I will it another try tomorrow again to verify. For the password itself we are passing the $cred, that was created during our SecureString process. During our testing of our SecureString decrypt process, it wrote back on the screen the correct value that we secured with SecureString.
0
jsclmedave

Administrator
Registered:
Posts: 501
Reply with quote  #14 
Again Evgenij is Spot On!

Our 3rd party application (BladeLogic) is handling the PW encryption/decryption piece.

The account being used also has rights to the vMware targets (READ), the server where it is being ran from and for my script the Network Share where the results are being copied too...

I would also check out vCheck http://www.virtu-al.net/vcheck-pluginsheaders/vcheck/ which is AWESOME!   This may provide you with more info and in my opinion is easier to execute.


 

__________________
Tim Bolton @jsclmedave
Email: [string](0..20|%{[char][int](32+('527377347976847978324785847679797514357977').substring(($_*2),2))}) -replace ' '  

New to the forum? Please Read this
0
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.