fix: stop heredoc delimiter from absorbing trailing control operators #1117
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -142,6 +142,27 @@ interface ScanResult { | |
| * Returns the bare delimiter word (for terminator line matching) and the index | ||
| * of the first character after the delimiter token. | ||
| */ | ||
| // POSIX control operators that terminate a word. A heredoc delimiter is a | ||
| // single word, so it must stop at these just like the shell's tokenizer does. | ||
| // This prevents a trailing control operator (e.g. `;`, `|`, `&&`) from being | ||
| // absorbed into the delimiter (see the "match POSIX shell tokenization" rule | ||
| // in scanTopLevelQuotes). | ||
| function isHeredocDelimiterTerminator(char: string): boolean { | ||
| return ( | ||
| char === "\n" || | ||
| char === "\r" || | ||
| char === " " || | ||
| char === "\t" || | ||
| char === ";" || | ||
| char === "|" || | ||
| char === "&" || | ||
| char === ">" || | ||
| char === "<" || | ||
| char === "(" || | ||
| char === ")" | ||
| ) | ||
| } | ||
|
|
||
| function parseHeredocDelimiter(command: string, start: number): { delimiter: string; endIndex: number } { | ||
| let i = start | ||
| let delimiter = "" | ||
|
|
@@ -160,11 +181,11 @@ function parseHeredocDelimiter(command: string, start: number): { delimiter: str | |
| if (command[i] === '"') i++ // consume closing " | ||
| } else if (command[i] === "\\") { | ||
| i++ // skip backslash | ||
| while (i < command.length && !isHeredocDelimiterTerminator(command[i])) { | ||
| delimiter += command[i++] | ||
| } | ||
| } else { | ||
| while (i < command.length && !isHeredocDelimiterTerminator(command[i])) { | ||
| delimiter += command[i++] | ||
| } | ||
|
Comment on lines
182
to
190
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== file outline =="
ast-grep outline src/shared/parse-command.ts --view expanded || true
echo "== relevant lines =="
nl -ba src/shared/parse-command.ts | sed -n '1,260p'
echo "== usages/tests =="
rg -n "parseCommand|heredoc|EOF|\\\\;|delimiter" -S src || trueRepository: Zoo-Code-Org/Zoo-Code Length of output: 1545 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== parse-command relevant lines using sed =="
sed -n '30,245p' src/shared/parse-command.ts
echo "== read-only behavioral probe =="
python3 - <<'PY'
from pathlib import Path
import ast
source = Path("src/shared/parse-command.ts").read_text()
# Identify the JS parser-like constants/functions that the file uses.
print("source includes isHeredocDelimiterTerminator:", "isHeredocDelimiterTerminator" in source)
print("source includes parseHeredocDelimiter:", "parseHeredocDelimiter" in source)
# Extract and evaluate the exact isHeredocDelimiterTerminator function from source so
# the probe uses the same predicate as the file.
start = source.index("function isHeredocDelimiterTerminator")
end = source.index("function parseHeredocDelimiter", start)
func = source[start:end]
print("== extracted isHeredocDelimiterTerminator ==")
print(func)
# Model the parser-controlled branch structure around the heredoc delimiter scan in parseHeredocDelimiter.
# This does not compile/execute repository code.
def is_delimiter_term(char):
return char in " \t\r\n;"
def current_scan(command, start):
i = start + 1
delimiter = ""
while i < len(command):
if command[i] == "\\" and i + 1 < len(command):
i += 2
continue
if not is_delimiter_term(command[i]):
j = i
while j < len(command) and not is_delimiter_term(command[j]):
j += 1
return command[start + 1:j], j
break
return "", i
# Case from the review: escaped semicolon outside quotes should produce delimiter "EOF;" with current behavior.
print("current_scan result for 'cat <<EOF\\\\;' command:", current_scan("cat <<EOF\\;", 4))
PYRepository: Zoo-Code-Org/Zoo-Code Length of output: 9317 🌐 Web query:
💡 Result: In POSIX shell, the handling of heredocs (here-documents) depends on whether the heredoc delimiter word is quoted [1][2]. Quote Removal and Delimiters If any part of the delimiter word is quoted, the shell performs quote removal on the word to determine the final delimiter [1][3]. Crucially, if the word is quoted, the lines within the heredoc are not expanded (meaning no parameter expansion, command substitution, or arithmetic expansion is performed) [1][2]. The shell treats the heredoc content effectively as a literal, single-quoted string [4]. Backslash Escapes The behavior of backslashes inside a heredoc differs based on the quoting of the delimiter: 1. Unquoted Delimiter: If the delimiter word is unquoted, the shell performs expansions (parameter, command, and arithmetic) on the heredoc content [2]. Within this content, the backslash character is used to escape the special characters '$', '`', and '' [2]. Additionally, a backslash followed immediately by a newline is interpreted as line continuation and is removed [1][4]. 2. Quoted Delimiter: If the delimiter word is quoted, the heredoc content is not expanded [1][4]. Because the content is treated as a literal string, backslashes lose their special status as escape characters and are preserved as literal characters, including backslash-newline sequences [4]. Control Operators Heredocs are a form of redirection and are associated with a command [5]. They are independent of control operators like '|', ';', '&', '&&', or '||' [5]. A heredoc remains attached to the command it follows, regardless of whether that command is part of a pipeline or followed by a control operator [6]. The shell identifies the heredoc content by reading from the current input source until it encounters a line containing only the delimiter word (after any necessary quote removal) and a newline character [6][2]. If the operator '<<-' is used, the shell also strips leading tab characters from the heredoc lines and the line containing the delimiter [1][2]. Top Results: [1] https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/utilities/V3_chap02.html [4] https://unix.stackexchange.com/questions/340923/what-does-posix-require-for-quoted-here-documents-inside-command-substitution [5] https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/utilities/V3_chap02.html [2] https://stackoverflow.com/questions/75535047/in-heredoc-what-is-the-delimiter-for-a-single-quote Citations:
🌐 Web query:
💡 Result: In shell scripting, the behavior of a heredoc delimiter regarding quoting and expansion is defined by the POSIX standard [1]. When you define a heredoc using a redirection operator like <<WORD, the shell treats WORD as the delimiter. If any part of this delimiter word is quoted (for example, <<'EOF', <<"EOF", or <<\EOF), the shell performs quote removal on the word to determine the final delimiter [1]. Crucially, if the delimiter is quoted in this way, the contents of the heredoc are treated as a literal string and are not subject to parameter expansion, command substitution, or arithmetic expansion [1]. Conversely, if the delimiter word is unquoted, the delimiter is the word itself, and the contents of the heredoc are fully expanded by the shell (performing parameter expansion, command substitution, and arithmetic expansion) after the trailing delimiter is located [1]. Additionally, if the redirection operator is <<- (rather than <<), the shell strips leading tab characters from each line of the heredoc as it is read, which is commonly used to allow for proper indentation of heredocs within code blocks while maintaining the intended formatting of the resulting document [1]. Citations: 🌐 Web query:
💡 Result: According to the POSIX Shell Command Language specification (specifically section 2.7.4, Here-Document), the handling of the heredoc delimiter word and the subsequent expansion of the heredoc body are determined by whether the word is quoted [1][2][3]. If any part of the heredoc delimiter word is quoted, the following rules apply: 1. The delimiter used to terminate the heredoc is formed by performing quote removal on the original word [1][2][3]. 2. The lines within the heredoc body are not subject to any expansions (such as parameter expansion, command substitution, or arithmetic expansion) [1][2][3]. If no part of the heredoc delimiter word is quoted: 1. The delimiter is the word itself [1][2][3]. 2. All lines of the heredoc are expanded for parameter expansion, command substitution, and arithmetic expansion [1][2][3]. In this mode, the backslash character behaves as it does inside double-quotes, meaning it can be used to escape specific characters like the backslash itself, the dollar sign, and backticks [1][4]. Quote removal itself is defined in section 2.6.7 of the POSIX specification, which mandates that the quote characters (backslash, single-quote, and double-quote) present in the original word are removed unless they have themselves been quoted [1][2]. Quote removal is always the final step in the word expansion process [1][2]. Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== scanTopLevelQuotes heredoc branch =="
sed -n '215,295p' src/shared/parse-command.ts
echo "== parseCommand parseHeredocDelimiter call sites =="
rg -n "parseHeredocDelimiter|heredoc|delimiter" src/shared/parse-command.ts
echo "== behavioral comparison for unquoted delimiter with escaped control operator =="
python3 - <<'PY'
def is_delimiter_term(char: str) -> bool:
return char in "\n \t;|&><()"
def current_parse_heredoc_delimiter(command: str, start: int):
i = start
delimiter = ""
if command[i] == "'":
i += 1
while i < len(command) and command[i] != "'" and command[i] != "\n":
delimiter += command[i]
i += 1
if i < len(command) and command[i] == "'":
i += 1
elif command[i] == '"':
i += 1
while i < len(command) and command[i] != '"' and command[i] != "\n":
delimiter += command[i]
i += 1
if i < len(command) and command[i] == '"':
i += 1
elif command[i] == "\\":
i += 1
while i < len(command) and not is_delimiter_term(command[i]):
delimiter += command[i]
i += 1
else:
while i < len(command) and not is_delimiter_term(command[i]):
delimiter += command[i]
i += 1
return delimiter, i
def fixed_unquoted_parse_heredoc_delimiter(command: str, start: int):
i = start
delimiter = ""
while i < len(command):
if command[i] == "\\":
if i + 1 < len(command):
delimiter += command[i + 1]
i += 2
else:
# Trailing unquoted escape: treat backslash literally, then stop.
delimiter += "\\"
i += 1
continue
if is_delimiter_term(command[i]):
break
delimiter += command[i]
i += 1
return delimiter, i
for command in ["cat <<\\;", "cat <<EOF\\;", "cat <<EOF\\n", "cat <<EOF\\|"]:
print(f"{command!r}: current={current_parse_heredoc_delimiter(command, 4)}, fixed={fixed_unquoted_parse_heredoc_delimiter(command, 4)}")
PYRepository: Zoo-Code-Org/Zoo-Code Length of output: 5706 🌐 Web query:
💡 Result: According to the POSIX Shell Command Language standard, the behavior of a backslash within double quotes is as follows: 1. Special Meaning: Inside double quotes, a backslash retains its special meaning as an escape character only when it is immediately followed by one of the following "special" characters: $, `, ", , or a [1][2][3]. 2. Processing: When a backslash is followed by one of these special characters, the backslash is deleted (removed), and the subsequent character is preserved literally [1][4][3]. 3. Ordinary Characters: If a backslash is followed by a character that is not one of the special characters listed above, both the backslash and the character following it are left unmodified [4][3]. The backslash is not removed in this case [5]. For example, the string "\a" inside double quotes remains "\a" because 'a' is not a special character that a backslash escapes within double quotes [3]. Conversely, "\" results in a single literal backslash because the first backslash escapes the second, causing the first to be removed during quote removal [3]. Quote removal itself is a distinct phase performed last in the shell's processing of a word [1][2]. During this phase, the quote characters (including double quotes and backslashes used for escaping special characters) that were present in the original word are removed, unless they were themselves quoted [6][1][7]. Top Results: [6][1][4][3] Citations:
🌐 Web query:
💡 Result: According to the POSIX Shell Command Language specification (specifically section 2.7.4, Here-Document), the handling of a here-document depends on whether the delimiter word is quoted [1][2][3]. If any part of the delimiter word is quoted, the shell performs quote removal on the word to determine the final delimiter [1][2]. Crucially, when the delimiter is quoted, the lines of the here-document are not expanded; they are treated literally [1][2][4]. Because the here-document content is not expanded, the backslash character does not act as an escape character (and backslash-newline line continuation is not performed) within the here-document body [5]. In contrast, if no part of the delimiter word is quoted, the shell does not perform quote removal on the delimiter, and the content of the here-document is expanded for parameter expansion, command substitution, and arithmetic expansion [1][2][6]. In this unquoted mode, the backslash character inside the here-document behaves similarly to how it behaves inside double-quotes, meaning it may be used to escape specific characters [1][2][6]. The phrase "If any part of the word is quoted" refers to the presence of quoting characters (like backslashes, single-quotes, or double-quotes) in the delimiter word provided by the user [1][7]. When these are present, they are removed to form the actual delimiter, and the resulting here-document body remains entirely uninterpreted [1][2][5]. Citations:
🌐 Web query:
💡 Result: Under the POSIX Shell Command Language specification, particularly section 2.7.4 (Here-Document), the use of a quoted delimiter—such as in the form <<'word' or <<\word—activates a specific behavior for the here-document [1][2]. When any part of the delimiter word is quoted, the shell performs the following steps: 1. Quote Removal: The shell performs quote removal on the delimiter word to determine the actual string that will serve as the terminating delimiter [1][3]. 2. Inhibition of Expansion: Because the delimiter is quoted, the here-document lines are not subject to parameter expansion, command substitution, or arithmetic expansion [1][2][3]. 3. Literal Interpretation: The contents of the here-document are treated as literal text, effectively as if the entire block were single-quoted [4]. This means characters like backslashes are not treated as escape characters, and the special handling of backslash-newline for line continuation does not occur within the body of the here-document [4][5]. The delimiter itself is identified by taking the word provided in the redirection operator and applying quote removal to it [1][2]. If the input word was <<\word, the backslash is removed during the quote removal phase, resulting in the delimiter "word" [1][5]. Consequently, the shell will continue to read lines from the input until it encounters a line that consists solely of "word" (after any necessary stripping if the <<- operator is used) [1][2][3]. In summary, quoting the delimiter ensures that the here-document remains entirely uninterpreted, preserving the input exactly as written [5][6]. Citations:
🌐 Web query:
💡 Result: In the context of the POSIX Shell Command Language, the rule regarding quoted here-document delimiters is found within the specifications for Here-Documents [1][2]. If any part of the word (the delimiter) is quoted, the following rules apply: 1. Quote removal is performed on the word to determine the actual delimiter string [1][2]. 2. The lines of the here-document are not expanded; parameter expansion, command substitution, and arithmetic expansion are inhibited [1][3]. 3. The content of the here-document is treated effectively as if it were inside single quotes [4]. If no part of the word is quoted, the delimiter is the word itself (without modification), and the contents of the here-document are subject to standard expansion (parameter expansion, command substitution, and arithmetic expansion), behaving similar to text within double quotes (with specific exceptions for the double-quote character itself) [2][5]. The phrase "If any part of the word is quoted" refers to the shell's mechanism for handling quoting characters (backslash, single-quote, or double-quote) present in the delimiter word [1][2]. Quote removal, which occurs last, is the process of removing these special quote characters from the word to determine the final, literal string used to terminate the here-document [2][5]. Citations:
Skip backslash escapes before delimiter-word tokenization.
🤖 Prompt for AI Agents |
||
| } | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.