Well, let's follow all the steps of a child console process'es execution.
(1) In the beginning, there was an NppExec's script. The script's commands (which will be executed) are located inside `g_nppExec.m_ScriptCmdList` - bidirectional list of strings. Each string is a separate command.
(2) The CmdList (list of commands) is passed to the CNppExecCommandExecutor:
... CNppExecCommandExecutor::ScriptableCommand * pCommand = new CNppExecCommandExecutor::DoRunScriptCommand(tstr(), CmdList, nRunFlags); GetCommandExecutor().ExecuteCommand(pCommand); ...
and the method ExecuteCommand() simply adds the given Command to the queue:
m_ExecuteQueue.push_back( std::shared_ptr<Command>(cmd) ); ... m_ExecuteCmdEvent.Set();
(3) The thread BackgroundExecuteThreadFunc monitors the m_ExecuteQueue and executes the given command in this thread (thus, not in the main thread):
...
auto& ExecuteQueue = pCommandExecutor->m_ExecuteQueue;
isExecuteQueueEmpty = ExecuteQueue.empty();
if ( !isExecuteQueueEmpty )
pCommand = ExecuteQueue.front();
...
if ( !isExecuteQueueEmpty )
{
if ( pCommand )
{
if ( !pCommand->IsExpired() )
pCommand->Execute(); // pCommand contains the CmdList
}
...
}
(4) The pCommand->Execute() eventually calls CScriptEngine::Run() where all the commands from the CmdList are finally executed. You can ensure it by looking at the brief code of this method:
...
CListItemT<tstr>* p = m_CmdList.GetFirst(); // first list item
while ( p && ContinueExecution() )
{
...
S = p->GetItem(); // extract a string
if (S.length() > 0)
{
...
nCmdType = ModifyCommandLine(this, S, ifState);
...
}
p = (m_execState.pScriptLineNext == INVALID_TSTR_LIST_ITEM) ?
p->GetNext() : m_execState.pScriptLineNext; // next list item
}
...
The following line:
nCmdType = ModifyCommandLine(this, S, ifState);
returns an identifier of an internal command such as NPP_EXEC, NPP_RUN and so on. Also, it expands (substitutes the corresponding values) the variables such as $(FILE_NAME), $(NPP_DIRECTORY) and so on. And here is how the command `S` is executed:
m_nCmdType = nCmdType; ... m_sCmdParams = S; EXECFUNC pCmdExecFunc = m_CommandRegistry.GetCmdExecFunc(m_nCmdType); nCmdResult = pCmdExecFunc(this, S);
(5) When the `nCmdType` is 0 (i.e. CMDTYPE_UNKNOWN), it means the following: the current string does not contain any internal NppExec's command, so it is interpreted as an external command - i.e. as a path to some executable file. Let's look at the corresponding method to which pCmdExecFunc points to in this case:
CScriptEngine::eCmdResult CScriptEngine::Do(const tstr& params)
{
...
std::shared_ptr<CChildProcess> proc(new CChildProcess(this));
...
proc->Create(m_pNppExec->GetConsole().GetDialogWnd(), params.c_str())
...
}
As you can see, it calls the method CChildProcess::Create, where the `params` parameter is the passed value of `S` mentioned above. So, let's refer to CChildProcess::Create.
(6) The first intelligible thing inside CChildProcess::Create is a creation of the input/output pipes (m_hStd Input/Output Read/Write Pipe). These pipes are "information channels" between the NppExec plugin and a child process which will be created soon. Don't forget about the pipes - we'll return to them later. The second intelligible thing is actually the creation of a child process. Here is the corresponding code:
// initialize STARTUPINFO struct
ZeroMemory(&si, sizeof(STARTUPINFO));
si.cb = sizeof(STARTUPINFO);
si.dwFlags = STARTF_USESHOWWINDOW | STARTF_USESTDHANDLES;
si.wShowWindow = SW_HIDE;
si.hStdInput = m_hStdInReadPipe; // pipe for user's input (stdin)
si.hStdOutput = m_hStdOutWritePipe; // pipe for process'es stdout
si.hStdError = m_hStdOutWritePipe; // pipe for process'es stderr
...
tstr sCmdLine = cszCommandLine;
...
if ( ::CreateProcess(NULL, sCmdLine.c_str(), ...) )
{
...
}
The second parameter of CChildProcess::Create (this parameter corresponds to `params` and `S` mentioned above) is passed to the system's function CreateProcess. This parameter is a path to an executable which will be run as a child process of the plugin.
(7) Now the child process is created. Let's assume it is a console process. As you remember, the whole CScriptEngine::Run() method is running in another thread, so this child process is created and handled in that thread also. The NppExec Console serves as the child console process'es output and input while this process is running. I.e. when you type some string in the ConsoleDlg's RichEdit control and press Enter, this string will be passed to the child process. You can find the corresponding code inside the "ConsoleDlg::OnNotify" function inside "DlgConsole.cpp". Search for "GetCommandExecutor().ExecuteChildProcessCommand". But let's return to the capturing of the child process'es output.
(8) This is the code of the capturing:
CStrT<char> bufLine; // a buffer for child process'es output
bool bPrevLineEmpty = false;
bool bDoOutputNext = true;
int nPrevState = 0;
DWORD dwRead = 0;
...
do
{
dwRead = readPipesAndOutput(bufLine, bPrevLineEmpty, nPrevState, false, bDoOutputNext);
...
}
while ( (isConsoleProcessRunning = (::WaitForSingleObject(m_ProcessInfo.hProcess, dwCycleTimeOut) == WAIT_TIMEOUT))
&& m_pScriptEngine->ContinueExecution() && !isBreaking() );
// NOTE: time-out inside WaitForSingleObject() prevents from 100% CPU usage!
if ( m_pScriptEngine->ContinueExecution() && (!isBreaking()) && !m_pScriptEngine->GetTriedExitCmd() )
{
// maybe the child process is exited but not all its data is read
readPipesAndOutput(bufLine, bPrevLineEmpty, nPrevState, true, bDoOutputNext);
}
We try to read data from the console process (by means of pipes) while the process is not finished and while the plugin's ConsoleDlg is visible. The "readPipesAndOutput" method actually does all the work about the capturing of the child console process'es output and showing this output in the ConsoleDlg.
(9) The whole "readPipesAndOutput" method can drive you crazy (I wrote this method by parts ;-)) but, in brief, it's very simple:
...
do
{
Sleep(10); // it prevents from 100% CPU usage while reading!
dwBytesRead = 0;
if ( !::PeekNamedPipe(m_hStdOutReadPipe, NULL, 0, NULL, &dwBytesRead, NULL) )
{
dwBytesRead = 0;
}
...
if ( (dwBytesRead > 0) || bOutputAll )
{
// some data is in the Pipe or bOutputAll==true
bool bContainsData = (dwBytesRead > 0) ? true : false;
// without bContainsData==true the ReadFile operation will never return
if ( bContainsData )
::ZeroMemory(Buf, CONSOLEPIPE_BUFSIZE);
dwBytesRead = 0;
if ( (bContainsData
&& ::ReadFile(m_hStdOutReadPipe, Buf, (CONSOLEPIPE_BUFSIZE-1)*sizeof(char), &dwBytesRead, NULL)
&& (dwBytesRead > 0)) || bOutputAll )
{
// some data has been read from the Pipe or bOutputAll==true
...
}
}
}
while ( (dwBytesRead > 0) && m_pScriptEngine->ContinueExecution() && !isBreaking() );
As you can see, at first we verify if there is something in the pipe:
::PeekNamedPipe(..., &dwBytesRead, ...) ... bool bContainsData = (dwBytesRead > 0) ? true : false;
then we read the data:
ReadFile(...)
And here is the problem. Sometimes PeekNamedPipe returns FALSE or returns dwBytesRead == 0 though some data must be in the output pipe already.
As of July 2023, this issue with PeekNamedPipe is still relevant. PeekNamedPipe still sometimes returns FALSE or returns dwBytesRead == 0 when some data has already been written by the console child process to its output - and thus this data is expected to be present in the output pipe. But no, PeekNamedPipe returns FALSE or dwBytesRead == 0. And if we call ReadFile at that point, it just does not return until the pipe internally "decides" to give its data away. (And usually it happens when the child process either exits or calls fflush() internally. Unfortunately, NppExec can't control that from its side of the pipe).
This is relevant for Windows 2000, XP, Vista, 7, 8, 8.1, 10 and 11. Microsoft even introduced so-called PseudoConsole in Windows 10, but still has not fixed this buggy implementation of PeekNamedPipe.
This is the only reason of why sometimes you don't see any output in NppExec's Console while some output is expected. The data is inside the pipe's internal buffer and there is no way to get it from there. Sad but true.